I usually connect with my server via ssh in a terminal and run basic commands. What’s a better, more efficient and modern way of doing that? Especially considering ai and documentation along the way? I wonder if there’s a better approach than “connect from remote and act local”. Is there a method to “code local and push to remote”?
I use a fedora server with podman, caddyfile and vi.
If you’re connecting from a system that’s a long distance away from your server in terms of latency, mosh can be more comfortable to use than ssh.
Maybe a bit more on the overkill side, but I use Talos Linux (declarative K8s distribution), then Terraform for initial setup and FluxCD for everything else (including VMs via Kubevirt). It’s a hell of an initial learning curve, but afterwards I just do my talosctl upgrade and upgrade-k8s from time to time, and don’t have to worry about anything else. Upside is, Kubernetes provides a unified, extensible API for everything, for example:
- reverse proxy via Ingress or Gateway-API
- firewall via NetworkPolicy
- even databases like Postgres through an Kubernetes Operator like CNPG, with a similar simplicity as with the big cloud providers (just a single yaml file with the specs like storage capacity, CPU and memory limits, backup target and schedule etc.)
- it is very scriptable (everything is managed via the API)
- automated image updates via either FluxCD itself or just dependabot/RenovateBot creating PRs to your GitOps repo
Downsides:
- you have to figure out persistent storage (CSI), which is slightly more complicated than just a single filesystem and manually specifying volume mounts like with docker, but if done right, you have a simple interface with powerful capabilities via the Kubernetes API
- there are simple CSI implementations that just expose node local disks via LVM or ZFS, so if you just have a single node, or don’t care about replication, it’s fairly easy to get started
- initial learning curve is quite high, especially if you have no prior experience with container orchestration in general
- definitely overkill if you just want something simple that “just works”
I assume this is on your own physical hardware? What parts do you set up with terraform?
Yes, my own hardware. But before I decided to build my own server last September, I was on a similar setup on a dedicated server at Hetzner, just less powerful (I’m so goddamn glad I bought the memory and storage back then, I probably wouldn’t even get the 192GB DDR5 right now for the price I paid for all memory and storage combined).
Terraform starts at the Talos initial machine config and cluster bootstrap including CNI setup and routing configuration (I have an opnsense in front that handles ingress routing) until FluxCD is fully set up and takes over.
Ingress traffic works via a small Hetzner VPS, and BGP through Wireguard, so there is no DNAT or SNAT between the open Internet and my Kubernetes load balancers and all services see the actual client IP. This took a shit load of time to get right, because wireguard only has an mtu of 1420, but the regular uplink is usually 1500, so pmtu discovery in my load balancers implementation (Cilium) and mss clamping in opnsense had to all be configured
Poorly.
Hear hear
Hell yeah
I wait for something to break. Then I yell “God fucking dammit, I don’t have time for this right now!” and spend an hour triaging things before I just try docker down, pull, up and everything works.
same. or i just df and see / 100%
Early on I wrote a script for my user account to run df and save the output to a file, then a second script to read that file and if the drive was full send me an email.
First, it failed because it couldn’t send the email because the email service failed under full disk conditions.
Second, it failed because it couldn’t write the file to disk because the disk was full.
Third, it failed because outbound SMTP was blocked by my ISP.
I learned a lot about how well you can fail if you really put your mind to it. Now I have Home Assistant grabbing the disk stats for my machines and flagging anything over 90%.
Local ansible playbook, easily replayable and documented. This in a git repo and you’re fine.
I ssh into it twice a year to perform updates but otherwise i leave it alone. Server’s been chugging along for like a decade now.
Similar, but you might want to update more frequently with the huge number of critical security bugs being discovered by AI recently
I have unattended upgrades configured
Hmm I need to figure out how to do that
If you are using aptitude (apt) it is quite simple: https://ubuntu.com/server/docs/how-to/software/automatic-updates/
If you just want security updates and are fine with the default configs:
# apt install unattended-upgrades
Twice a year is crazy. I suppose most of my maintainence is adding drives (I’m a creator and I definitely totally need to archive all of my footage), but I also run a fair few apps that have needed manual update steps, so I do all my updates manually so I can fix stuff if it breaks.
Tailscale service configs for HTTPS ingress and DNS resolution. Podman quadlets / podlets for everything except backup software and tailscale itself. I use the open version of VS code to edit config files in place, but I push them to a private git as well as having them backed up. Storage for app data, media, and configs (everything but OS) is via NFS shares from my gaming PC’s RAID volume. I can run a plenty of apps on an old thinkpad this way. After you configure the first few quadlets with tailscale, it gets real straightforward, but there is a learning curve. Yes, I use SSH, but it is the remote conmection and terminal in Code.
Oh yeah, podman let’s me run all the containers as user-space systemd services.
I just use ssh and manually type commands. Keep doing it and you will get good and it will become second nature.
No need to burn tokens on basic tasks.
Master your tools. Learn awk, sed, just, etc.
Some shell customisation can help. For example I’m fond of zsh-auto-suggestions and skim. Makes me quicker.
Ansible.
That, and there is no change that isn’t done on ansible, or, when it required trying and research gets backported into a bit of ansible.
I write my NixOS configs in my PC, commit to the repo, then push it to the server. Then SSH in, and apply it. Or more likely, MOSH in.
Ansible is one way to “code local, push to remote(s)”. Can define so-called playbooks, which is basically just a script, to do reoccuring tasks like e.g. updates.
Ansible is great, I just wish it would automatically cleanup stuff on the server when I remove parts of the config. I know it’s not the goal, but I dream about a tool that works this way. I get many leftover config over years
That sounds like NixOS to me.
I have yet to try it but apparently
We fixed that with roles that manage both install and uninstall
That works but that’s only useful if you have many machines. I have a single server so it’s the same burden as doing it manually. If only the uninstall step could be completely automated by just reverting the install step
I generally find writing a good role takes longer than doing it manually. Especially as you have to keep testing failure conditions as you think of them.
However, the point is, that once you’re finished with the role, you never have to do that process again.
Not if you want to rebuild Not if you want to migrate to a new os Not if you build a second server Not if you move to a newcloud hosting provider
Porting to a new OS or after an update that changes how the software works is about as fast as doing it manually
Maybe it’s only one server today, but how many times would you need to rebuild it over the running lifetime?
The killer feature is that having something like ansible lets you kill your darlings, and keep servers aa cattle, not pets.
Manual setup chops isn’t a different skill here. I view ansible as a combination setup and documentation. If you’re going to figure out how, why not write down how you did it? If you’re going to write it down complete with code lines to run, why not script it?
Not quite, if you write the role to manage the life cycle of a bit of software then uninstall is part of that life cycle.
But I do see that investment in time looks more then just do it by hand
To be fair to Ansible, that’s probably a packaging issue.
If the 1st run of an application creates a bunch of files and folders, then the packager won’t know about them and Ansible’s just relying on that.
But yeah, for 1 machine I’d find Ansible overkill (not a problem per se), but really helps when you have 5 Rasperry Pi Zeros scattered around the house 😉
It’s good to get comfortable in the command line, including over ssh.
Especially considering ai and documentation along the way?
If I am going to ask AI or Google about something, I just do that to help me find the answer and then apply the answer myself. That way I learn, and can double check the AI isn’t hallucinating, at least on in obvious ways.
As for documentation, I keep a notes folder with detailed notes on manual configuration I’ve done, how and why, and the things I’ve learned along the way. I’ve found it’s both useful to help remember the things I’ve learned, and it is useful to go back to refer to.
A mix of prayer and bash scripts.
Remove bash scripts and add poorly configured logging and that’s me!
At least you have logging!
I use ssh with tmux
This is the way
<insert Mandalorian gif here>
I spent a while getting Ansible to be able to setup and maintain my server(s). And ended up not using it as much as I should, mostly because the computer I used to run Ansible from became a “Steam Machine” so I’m rarely sitting on it with a keyboard now and I don’t want to have personal info and keys on my work computer.
But it was a good learning, and useful while I used it. Now I just use ssh and compose files directly.
You upgraded from ansible to ssh. As someone forced to write yaml for ansible professionally and has used many other config management tools, I can only commend your success at escaping ansible.
Try mgmtconfig.
