Services fail when booting into snapshot

[dhasial]

I think this may be related to #64.
Debian 10 Stable

Whenever I boot into a snapshot from GRUB, a bunch of services don't boot and I end up in a TTY console instead of the GUI like normal. My /boot/efi is on a separate partition, and is not a BTRFS subvol, I wonder if this is related?

[max-baz]

My /boot/efi is on a separate partition, and is not a BTRFS subvol, I wonder if this is related?

Definitely not, this is very normal.

I can't think of any specific recommendation for what to investigate, try to see get to the root cause of why services fail, check the status, journal, maybe you'll spot something?

[trochdewei]

I probably have same problem when trying to boot snapper snapshots. My os is Manjaro

[mattiabiondi]

The snapshot you are trying to boot into is probably set as read-only.

You can check it with:

$ sudo btrfs property get -ts /path/to/snapshot/

If it returns ro=true, you have to make it writable with:

$ sudo btrfs property set -ts /path/to/snapshot/ ro false

[max-baz]

While that might be the case, I am able to boot into readonly snapshots just fine, and there are cases where you want to keep them readonly (e.g. this snapshot is your only backup and you can't risk destroying it while attempting to recover your system).

Quote from #88 (comment):

They are r/o for a reason: Booting into a snapshot changes the data and might just destroy the last available backup.
Sometimes, one needs several tries until one achieves to fix a problem, so there should be an option to start over.

[trochdewei]

I have the following layout, don't know if it can help

[...]
UUID=xxx /              btrfs   subvol=@,defaults,noatime,space_cache,ssd,compress=zstd,commit=120 0 1
UUID=xxx /home          btrfs subvol=@home,defaults,noatime,space_cache,ssd,compress=zstd,commit=120 0 2
UUID=xxx /.snapshots    btrfs   subvol=@snapshots,defaults,noatime,space_cache,ssd,compress=zstd,commit=120 0 2
[...]

[Kr1ss-XD]

FWIW, from my experience one caveat preventing you from booting r/o snapshots can be systemd trying to write the journal onto the r/o file system. You could try to mitigate this by setting storage=volatile or storage=none in /etc/systemd/journald.conf or /etc/systemd/journald.conf.d/*.conf.

Be aware though that you won't have any persistent logs (or even none at all, depending on the setting you chose) then.
I ended up doing double snapshots each time via pacman hooks - the default r/o one, and a writeable snapshot of the same subvolume which I can boot into if I need to.

More info regarding the journal storage in the manual - journald.conf(5).

[trochdewei]

I solved issue by moving /var to separate subolume

UUID=xxx /                     btrfs   subvol=@,defaults,noatime,space_cache,ssd,compress=zstd,commit=120 0 1
UUID=xxx /home            btrfs   subvol=@home,defaults,noatime,space_cache,ssd,compress=zstd,commit=120 0 2
UUID=xxx /.snapshots   btrfs   subvol=@snapshots,defaults,noatime,space_cache,ssd,compress=zstd,commit=120 0 2
UUID=xxx /var                btrfs   subvol=@var,defaults,noatime,space_cache,ssd,compress=zstd,commit=120 0 2

[max-baz]

This was really a nice catch, I completely forgot my I also have /var/log on a separate subvolume, so that's why I am able to boot without any problems on a readonly snapshot.

Thanks guys, I will document this in README!

[ckotte]

@maximbaz Can you still boot into your read-only snapshots? If yes, is your var on a separate subvolume?

This is my layout:
@
@/boot/grub/x86_64-efi
@/var/cache
@/var/tmp
@/var/lib/machines
@/var/lib/portables
@/var/lib/libvirt/images
@home
@var-log

Now the system hangs at boot and I get this:

Oct 07 17:33:48 vagrant-btrfs-restore systemd[1]: Dependency failed for Multi-User System.
Oct 07 17:33:48 vagrant-btrfs-restore systemd[1]: Dependency failed for Graphical Interface.
Oct 07 17:33:48 vagrant-btrfs-restore systemd[1]: graphical.target: Job graphical.target/start failed with result 'dependency'.
Oct 07 17:33:48 vagrant-btrfs-restore systemd[1]: multi-user.target: Job multi-user.target/start failed with result 'dependency'.

I tested read-only snapshot boot and restore two months ago and it was working with GNOME etc. pp. installed. It still works on a new machine if no graphical userinterface is installed.

Either something has happened in the meantime or I didn't test it properly. I can boot if complete /var is on a subvolume..

[max-baz]

@ckotte just tested: yes I can still boot boot in my read-only snapshots, and no I don't have the entire /var on a separate subvolume. For full disclosure, I no longer use GRUB and thus this project, but my new setup is semantically very similar to what grub-btrfs is doing so I'm pretty sure it's not important to in this case.

Could you double-check that your /var/log is indeed mounted to @var-log subvolume? I'm not sure but it looks a bit suspicious that your subvolume is called @var-log and not say @/var/log.

As another idea, maybe our graphical environments differ, and your environment contains something that cannot work on readonly filesystem, while mine can 🤔 I'm using sway, and although it is launched via systemd, it doesn't have any dependencies. Could you try to investigate perhaps what exact dependency has failed in your case?

For reference, here's my layout.

[ckotte]

@maximbaz @var-log subvolume is mounted. I can read the logs of the failed boot after booting normally. @/var/log would be a nested subvolume of @ like the other nested subvolumes in /var.

I guess some services cannot deal with read only /var/lib anymore and therefore it fails to boot. I cannot get access to the system. I can only read the logs after booting normally. This makes fixing this issue very difficult for me..

May I ask which bootloader you are using now? I cannot find the boot loader installation in your script. It also looks like you are using encrypted /boot? I thought only grub can handle encrypted /boot!?

[max-baz]

So I used GRUB exactly because I had this same belief as yourself, that encrypted /boot is only possible with GRUB, and without encrypted /boot you are open to evil-maid attacks, and all hope is lost.

But I researched this topic and it turned out that there are alternative approaches that are even better!

---

Here's how I do normal boot:

I'm using direct UEFI boot + Secure Boot now, no bootloaders like GRUB or systemd-boot - less code means less vulnerabilities to guard yourself against.

Here's how it works:

1. Remove built-in Secure Boot keys, generate your own keys, leave private key on encrypted disk and register public key in BIOS (this part I had even when I was using GRUB, you probably have it as well).
2. Keep /boot on encrypted disk, and mount unencrypted ESP FAT32 partition to /efi.

Now comes the interesting part:

3. Generate a new .efi file that you will register in BIOS as boot target (i.e. instead of grub.efi), which consists of:

   • CPU microcode (ucode.img)
   • initramfs
   • vmlinuz
   • hardcoded kernel cmdline (that specifies exact kernel arguments to boot, including root btrfs subvolume)

All of the above components are taken from encrypted /boot, so cannot be tampered with while your computer is turned off.

4. Sign this .efi file with your own Secure Boot key and put this one file in unencrypted /efi.

5. Configure in your BIOS that this is the boot target (instead of GRUB)

Now evil-maid attack is not possible because the only unencrypted file is your signed .efi file, and if it is being tampered with, Secure Boot will refuse to load it.

Because cmdline is hardcoded in the image, Secure Boot also guarantees that you or attackers cannot just change it (e.g. to boot in an old subvolume).

In addition, because there is less steps in the process, and especially because you aren't decrypting your disk twice (like it is the case with GRUB), the boot is soooooooooooo much faster, you wouldn't believe it.

---

Here's how I do recovery if my main subvolume fails to boot:

1. Generate not only the .efi file described above, but also another .efi recovery file:

   • initramfs
   • vmlinuz
   • NO microcode (in case it causes boot failures)
   • NO hardcoded cmdline (so that we can later select which subvolume to boot in)

In fact I create two such recovery .efi files, one with latest kernel and one with LTS kernel, in case boot failures are caused by kernel upgrade.

2. Sign these .efi files as well for Secure Boot, but do NOT add them in BIOS yet - so that if you want to boot into them, you must first use your BIOS password.

Attackers cannot use this .efi file because to boot into it they need to know your BIOS password.

Evil maid attack is not possible because this image is signed with Secure Boot keys and at no point in time do we disable Secure Boot.

Because cmdline is NOT hardcoded in these recovery images, Secure Boot will let us specify a custom one, one where you specify rootflags=subvol=snapshots/123/snapshot and boot into a snapshot 123.

---

There are a couple of projects that automate most of these steps, but none of them satisfied my needs precisely, so I chose to build my own tool that does exactly what I want 😄

It might be very much tailored to my personal needs and although feedback is welcome I'm not sure that I want to make it more generic (again, the less code there is, the more I can trust it).

But if you want to have a look, it is available on Github: arch-secure-boot

It automates all of the steps described above, plus contains a couple of extra integrations that I personally use:

• pacman hooks (similar to snap-pac-grub, so I can enable the tool and forget about it)
• integration with fwupd
• custom EFI shell (because I use Dell laptop, and their built-in implementation is buggy)
• a simple script that provides UI for selecting snapshot to boot into, so that I don't need to type cmdline, just boot in recovery and lazily select the snapshoot, almost as easy as grub-btrfs 😄

And here's how I configure the tool during OS installation, it's just one line to execute, and I don't need to remember about this ever again.

---

In order not to spam people with unwanted notifications, if you have some questions or feedback or just want to chat more about this setup, please open an issue in arch-secure-boot 👍

[ckotte]

@maximbaz Thank you very much for your detailed explanation! This sounds interesting. I also have a Dell laptop and I don't like their EFI implementation as well. I will try it out once I figured out a working setup for my read-only snapshots. 😉

[ckotte]

I found out why I'm not able to boot read-only snapshot anymore. It's an issue with gdm and accounts-daemon.

The errors about the other services go away if I use subvolumes for /var/cache and /var/tmp. However, the system still hangs and doesn't provide a login. It started probably with gdm 3.36. I can boot into read-only snapshots again if I create subvolumes for /var/lib/gdm/ and /var/lib/AccountsService.

Not sure if I want to implement this. Using a subvolume for /var is probably the best option when using GNOME with the disadvantage of a writeable/var when booting the read-only snapshot. I'm also not sure if I can boot the read-only snapshot if /var is corrupted...