Some HomeLab Lore & How I accidentally wiped my own wiki

Yep. You heard me right. I accidentally wiped my own wiki.

For some context, for the past two weeks, I've been working on completely renovating my homelab situation, so I can finally stop manually downloading very legal Ubuntu Torrents manually and instead have some automation do it.

The problem I had was my very weak server. See I have a laptop as my entire homelab, ignoring the networking jazz. This laptop is a beast for the specs it has (Intel i3-5005U 2 Core CPU at 2Ghz with 4GB of RAM) and for the past 3 or so years I've been on this apartment it has been my primary server for Home Assistant, some media stuff (e.g. Immich) and lately my Paperless server.

Now running all these services is typically fine, I also was running PiHole and UniFi OS Gateway on it, which basically meant if this laptop went down (which it did! I had:

1. No internet and
2. Zero home control

Ignoring the fact that I'd lose access also to my Bitwarden password sync, and my phone image backup, losing two of the most two essential parts of my home was a bit of problem. These downtimes weren't really that common, it would typically be after the laptop randomly chockes from running for too long, or accidentally doing too many DNS queries and overloading PiHole. A quick restart usually did the trick, albeit sometimes I had to run fsck before boot (MANUALLY!) to fix some random partition errors on the SSD, probably related to the crashes.

I solved one of these problems by upgrading my network setup to a Ubiquiti gateway, and that REALLY helped. Apparently the UniFi gateway docker image gets quite chatty when devices connect and reconnect, so a lot of the WiFi issues I'd have and the random CPU spikes that would cause PiHole to choke up and begin avalanche of issues, were mostly related to that.

Since the day I offloaded UniFi Gateway to an actual UniFi device, in this case the Dream Machine SE, I've noticed that the laptop-server would actually run much quieter (cpu wise). I think it also helped that PiHole no longer acted as a DHCP and full authority DNS server, as those were now handed off to Dreamy (the Dream machine as I've nicknamed it)

This entire idea of re-working on my homelab and properly setting up, prompted me to setup a Git repo on my GitTea instance that I had laying around and document my own setup for my own sanity, but also provide a nice overview of what I've been doing on my setup for future employers that want to see my abilities in this sector, but also to share it with the world (and especially my best friend who I desperately need to get him to setup a decent homelab setup for the work he does lol)

On top of the Git repo (which by the way you can find here: https://git.alepouna.net/alenet/setup), I decided to also create a wiki (using wiki.js) so that I can document the more "yappy" parts of the configurations and setup there. What I didn't really anticipate was how into it I would get. Within a few hours of setting it up, I've already filled 3 full pages of information for what I run on my setup, network security (and a quick 101 about it) and a few other stuff.

And then, I did the most stupid thing ever. I took down the container to do some maintenance on it (fix an OAUTH issue I had where it would take 10-20 seconds to login).

I kinda got distracted by something else, and noticed my server had about 70GB used space out of nowhere. The culprit? Docker. I was experimenting with so many services lately to write on the Wiki and see what I could usee for stuff, that I have managed to download so many images and had so many unused containers that those needed to go.

So I decided to do something that I thought was pretty reasonable at the time and run:

docker system prune --volumes

This command removes all stopped containers, all unused networks, all unused images, all build cache and all the unused volumes.

At the time I thought "yay more space!", and then I got a notification from Uptime Kuma

❌ Your service wiki.alepouna.net went down. ❌

"Oh."

It was at this moment, that I knew, I fucked up. I quickly realized what just happened, and I checked docker to see that indeed, the wiki was gone.

10+ days of writing, over 5000 words, 20 or so images of various configurations, router settings and more, now all gone.

Initially I was mad at my self for running the command, then I was mad at my self for not backing up the container to my Hetzner bucket, then I realized I had backups, but it was only of the letsencrypt volume (for some reason?!), then I got mad again at my self for all of the above. Then disappointment.

All this work, gone, reduced to atoms.

Welp, lesson learned. Never run a prune command with volumes, manually remove them. Never not backup documents that you use in production. And finally, check if the dang backup works.

With that being said, and this frustration & grief now out of my system, I am reconciling a dispute with my own thoughts.
Was the wiki even a good idea to begin with? Maybe just having everything in a Git repo would work? Maybe the wiki should just render the Git repo? I don't know. Let me know your thoughts in the comments, or my DMs, you know where to reach me ;)

Anyways, to end in a happy note, I still have in the back of my head a lot of the content I wrote, and I've already started to write some of it back in the Git repo, soon to commit, but yeah. Take backup kids.

PS: I will be posting a lot more stuff regarding this very homelab, troubles I've gone through, my experiences and other cool beans stuff, so stay tuned.