Start Backup <computer explodes>

A computer burns¹

I recently bought a new 2TB NVMe M.2 drive with the intention of rebuilding my system. I was running ubuntu 20.04 but the next LTS release of ubuntu (22.04) was already available and canonical is leaning hard into snaps. I hate snaps. They solve the same problem as fedora's flatpacks, pollute my network list, and the last time I needed some weird software ubuntu offered to sell it to me through their snap store. The store was filled with commercial software. I specifically moved to open source to get away from this short of shit.

The last few weeks I've been researching different distros and decided to move to an arch based distro called manjaro. Arch, unlike debian, uses a rolling release process so the packages stay more up to date at the expense of having things break more often. I played around with it in a vm and it seemed like it would work.

Friday, I started backing up files and decided the best way to handle the upgrade was to just copy my entire /home/ directory to one of my backup drives so I could pick and choose what I wanted to bring back into the new system. The old M.2 drive would be my safety net. If I couldn't get everything working on the new drive, I could just put the old M.2 drive back in and it would be like nothing had happened.

The backup of my home directory had been running for a couple hours when my desktop locked up. This was unusual but it didn't bother me until I hit the reset button and the computer claimed the bootloader was corrupt. This was when I started to panic.

I pulled out my laptop and started looking up the arcane instructions for rebuilding a bootloader from a live boot. The grub rebuild command seemed to work but when I tried to boot from the drive it still failed.

So I was stuck. The first thing I did was to go online and order a M.2 - USB adapter because the data was probably still there even if I couldn't boot into that disk but even with expedited shipping it was going to arrive on the following Monday.

I work at home and my computer is a critical tool. I removed the old M.2 drive, plugged in the new one, and starting installing manjaro.

Though manjaro is a new linux distribution for me the main practical difference is it uses a package management system called pacman instead of apt. Installing software on linux systems is a delight. If I need mpv, vifm, mpd, nginx, and neovim I can just type:²

pacman -S mpc vifm mpd nginx neovim

It will download these applications from a secure, centrally managed software repository, work out any required dependencies, and start any new system services when it's done. Oh, and it's all free. The last time I used windows I still had to do google searches and download binaries from random people's web pages. I haven't used windows in 20 years but I'm guessing they have an app store that's filled with ads and commercial software. I recently got a mac book for work and that app store is the template for selling software to your customers. My guess is windows is desperate to condition their users to start paying for software and ubuntu wants to do the same.

There are a few packages that weren't in the extensive repositories. My password manager is an old tcl/tk app, my comic reader is abandonware (a project I've considered adopting), and my newsreader is a gtk app that no-one uses anymore. In each of these cases I was able to compile the application from source, without much fuss, getting the required dependencies with pacman commands.

computer explodes³

Because I didn't manage to make a proper backup before this forced upgrade I had to scramble to save the changes I made since my last backup (the night before). Also while restoring from backup I discovered I had neglected to add a '--delete' flag to one of my nightly rsync backup script which meant every backup directory contained every file they ever held the last several years (every test project long deleted and every steam game long since uninstalled).

During my live boot recovery I copied a few files to a mounted backup drive and to a thumbdrive but I was unable to read this thumbdrive later in manjaro. I spent hours combing the internet for clues why the kernel was mounting this thumbdrive as read-only only to realize that my thumbdrive had died. That it decided to die on the same day my other drive failed was just bad luck.

During the rebuild, after rebooting several times, my BIOS changed and started trying to boot from a non-bootable drive. I fixed this and then realized that the problem could have been as simple as a flaky M.2 socket making my NVMe drive go away, and then the BIOS detecting this and 'helpfully'⁴ reordering my boot order to load from a non-bootable drive.

Knowing this, I tried to go back to my original drive but I ran into the same missing bootloader issue which made me think the NVMe drive didn't fail at all and I broke it while trying to restore the bootloader after a socket hiccup.

The moral of this story is to keep backups. Even with backups from the night before I still had to scramble to rescue some files. A younger na might have thought I was being paranoid when I was running from a live boot and I backed up my latest changes in two places; a second drive and my thumbdrive, but this saved me when the thumbdrive chose that moment to die.

Every time I go through a re-install my backup scripts get sharper and things get easier but I always miss something. I can't imagine what normal people would suffer.

For all the bad luck of having the drive go down before I could finish my backup scripts, and having a thumbdrive fail during the recovery, I was extraordinarily lucky to have it happen when I already had a replacement 2TB NVMe drive on hand. I've still got the old M.2 drive and I suspect it's still OK but take this as a warning.

Make backups!

posted by na on June 2, 2022 19:38