Don’t worry, you’re one Docker pull away from having to look up how to manually migrate Postgres databases within running containers!
(Looks at my PaperlessNGX container still down. Still irritated.)
https://github.com/pgautoupgrade/docker-pgautoupgrade
Or if you are on k8s, you can use cloudnativepg.
I’m just using Docker on Proxmox, buuuut… I’m gonna look into this project. It looks like a LIFESAVER. Thank you for sharing this. You’re awesome! :D
I feel your pain. Had to fix my immich, NC and Joplin postgresdb. Turned out, DB via NFS is a risky life. ;D
Backups. You’re forgetting them.
Pro tip: If you’re using openwrt or other managed network components don’t forget to automatically back those up too. I almost had to reset my openwrt router and having to reconfigure that from scratch sucks.
I should do some breaking network changes… While tunneled in.
“Yes, while connected to my wireguard server through port 123 here from my Chinese office, I should probably try to upgrade the wireguard server. That’s a great idea!”
Ask me how I know.
I stopped the tailscale service…
… while ssh’d through the tailscale interface.
Luckily, it was my home server and I had to drive there anyway.
I used to make nginx changes while vpn’d into my network and utilizing guacamole (served via said nginx). I’m not a smart man.
Off topic, warning: this comment section is making me want to learn things
It’s been 2 days off reddit and my brain has opinions other than “aaaargh” or “meh”.
Proceed with caution
No upstream bugs to fix?
Yes that does seem to describe modern computing, indeed, consumer electronics in general.
It’s no longer about solving actual problems, it IS the problem.
How is the kubernetes (k3s/rke2) migration coming along?
One word: chaos engineering!
It makes me start looking for the next thing. Got my jellyfin, got my pi hole, my retro console and just recently home assistant set up. (Just a few more buts to add to that). Next i think i am going to look into self hosting a cloud storage solution. Like google drive/photos etc. Would be nice to make my own backups and have them offline
Let’s tinker around and accidentally break something.
My
manperson!and debug it until you have to reinstall your entire stack from scarch
GET OUT OF MY HOUSE!
Are you implying it’s possible to debug without having to reinstall from scratch? Preposterous! 😂
Scarched arth
Guess this is a good time to test my infrastructure automation.
“Damn, I’ve got this Debian server shit down. I wonder how an opensuse server would work out”
*installs tumbleweed*
True story
Have you tried introducing unnecessary complexity?
If you know how your setup works, then that’s a great time for another project that breaks everything.
Saturday morning: “Incus and podman seem interesting. I bet I could swap everything over while the family is out this afternoon”
Sunday evening: “Dad, when will the lights work again?”
“Dad, when will the lights work again?
As soon as selinux decides I have permission.
The old lighting wasn’t that great anyway. If I were to just put lighting on a DMX512-controlled network, then all of it could be synchronized to whole-house audio…
Don’t forget to integrate it into Home Assistant so you can alert the ISS when the mail man is on the porch.
Infrastructure diagram? No! In this homelab we refer to the infrastructure hyperdodecahedron.
It seems like a good time to learn graphviz’s dot format for the network layout diagrams, with automated layout.
https://mamchenkov.net/wordpress/2015/08/20/graphviz-dot-erds-network-diagrams/
TIL. Thank you!
Sure. What that guy is using is actually not the most-interesting diagram style, IMHO, for automatic layout of network maps, if you want large-scale stuff, which is where the automatic layout gets more interesting. I have some scripts floating around somewhere that will generate very large network maps — run a bunch of traceroutes, geolocate IPs, dump the results into an sqlite database, and then generate an automatically laid-out Internet network map. I don’t want to go to the trouble of anonymizing the addresses and locations right now, but if you have a graphviz graph and want to try playing with it, I used:
goes looking
Ugh, it’s Python 2, a decade-and-a-half old, and never got ported to Python 3. Lemme gin up an example for the non-hierarchical graphviz stuff:
graph.dot:
graph foo { a--b a--d b--c d--e c--e e--f b--d }Processed with:
$ sfdp -Goverlap=prism -Gsep=+5 -Gesep=+4 -Gremincross -Gpack -Gsplines=true -Tpdf -o graph.pdf graph.dotGenerates something like this:

That’ll take a ton of graphviz edges and nicely lay them out while trying to avoid crossing edges and stuff, in a non-hierarchical map. Get more complicated maps that it can’t use direct lines on, it’ll use splines to curve lines around nodes. You can create massive network maps like this. Note that I was last looking at graphviz’s automated layout stuff about 15 years ago, so it’s possible that they have better layout algorithms now, but this can deal with enormous numbers of nodes and will do reasonable things with them.
I just grabbed his example because it was the first graphviz network map example that came up on a Web search.
Haha too right mate
unnecessary complexity?
I can help with that. It’s a skill I have. LOL
This is just as true in my non-computer hobbies that involve physical systems instead of code and configs!
If I had to just barely meet the requirements using as little budget as possible while making it easy for other people to work on, that would be called “work.” My brain needs to indulge in some over-engineering and “I need to see it for myself” kind of design decisions.
If it’s stable, it’s not a lab.
That’s infrastructure.
I’ve moved my homelab twice because it became stable, I really liked the services it was running, and I didn’t want to disturb the last lab**cough**prod server.
My current homelab will be moar containers. I’m sure I’ll push it to prod instead of changing the IP address and swapping name tags this time.
When’s the last time you checked if your backup solution works?
But if my backups actually work then I miss out on the joy of rebuilding everything from scratch and explaining to my wife why non of the lights in the house work anymore.
Carry around a candle in one of those old timey holders like Scrooge Mcduck
Yesterday! Switched my media server from freebsd to alpine and got the arr stack all set up using the backup zip files
Backup? Psh… That’s what the lab is for.
What’s a backup solution…? (I’m only being half sarcastic, I really need to set one up, but it’s not as “fun” as the rest of my homelab, open to suggestions)
No mercy for you, then. ;)
I at least have external backups for important family pics and docs! But yea the homelab itself is severely lacking. If it dies, I get to start from scratch. Been gambling for years that “I’ll get around to a backup solution before it dies”. I wouldn’t bet on me :|
logging is probably down
You do, of course have a dedicated rsyslogd server? An isolated system to which logs are sent, so that if someone compromises another one of your systems, they can’t wipe traces of that compromise from those systems?
Oh. You don’t. Well, that’s okay. Not every lab can be complete. That Raspberry Pi over there in the corner isn’t actually doing anything, but it’s probably happy where it is. You know, being off, not doing anything.
Ah. The approach that squirrel@piefed.zip suggested. ;)
Thanks for the tutorial though.
Hmmm. My pi{VPN,hole,dhcp,HA} has a little bit of overhead left…
Time to start documenting it!
At 71, I have to document. I started a long time ago. I worked for a mec. contractor long ago, and the rule was: ‘If you didn’t write it down, it didn’t happen.’ That just carried over to everything I do.
Do you write down what you write down on the internet?
As in a blog or wiki? I do not because I am not authoritative. What I know came from reading, doing, screwing it up, ad nauseam. When something finally clicks for me, I write it down because 9 times out of 10, I will need that info later. But my writing would be so full of inaccuracies that it would be embarrassing and possibly lead someone astray.
It’s how cults start!
I’ve started to take a l lot more notes at work I guess there will be a time where I take notes of what month it is!
I guess there will be a time where I take notes of what month it is!
You may jest, but there are times when I can’t remember what I had for breakfast. They say that you never truly forget anything, but that our recall mechanism fades over time. For a myriad of reasons, including age, my recall mechanism is shit.
Offt depends what you had and your version of health. I am hopeful that technology helps when I am that age, only a few years but ai agents seem to be a start. Just need to let go of those big data fears.
NEVER1!!!11!!
Don’t look too closely you can jinx it.











