LUKS encrypted devices that are linked to TPM2 are no longer being automatically decrypted during the boot process

Hi,

Following a network-based system installation using Kickstart with a %post script configured to handle automatic LUKS device unlocking via TPM2, the LUKS-encrypted devices fail to unlock automatically during boot.

A system whose LUKS devices were previously configured to unlock automatically via TPM2 binding is no longer decrypting those devices during boot, despite this functionality working correctly before.

How to resolve it.

Thank you, Julia :smiley:

Assuming PCR register 7 is already being used to bind and unlock the LUKS devices, this issue can occur in the following scenarios:

On already-installed systems:

After the shim-x64 package has been updated and the system has been rebooted

After the Secure Boot state has been modified or toggled

On systems immediately after installation:

When a network-based installation was performed and the BOOTX64.efi bootloader specified during the UEFI PXE or HTTP boot phase is outdated compared to the version shipped with the shim-x64 package that gets installed on the system

When the CentOS 8.x DVD was used for installation, but the packages were pulled from external repositories rather than those included with the DVD

Regarding other PCR registers:

If LUKS device unlocking relies on PCR registers other than PCR 7, the issue may also surface following a firmware update or due to various other conditions, since these registers are inherently unstable and subject to frequent changes.

The scenario in which this problem occurs on the very first boot following a network-based installation performed via Kickstart

Reinstall the system after making sure you follow the guide lines below.

When performing a network-based installation, ensure that the BOOTX64.efi file corresponds to the exact OS release being installed, and use it during the UEFI PXE or HTTP boot phase.

Under no circumstances should the grubx64.efi bootloader be used β€” this applies even when the system is booting with Secure Boot disabled.

Examples:

  • If you are installing CentOS 8.4 and plan to use packages from the DVD, set the BOOTX64.efi file found on the CentOS 8.4 DVD as the bootloader in your DHCP configuration.
  • If packages will be pulled from the CDN repository instead, download the shim-x64 package and extract the BOOTX64.efi file it contains β€” you can use the rpm2cpio utility to extract the package contents.

Additionally, ensure that PCR 7 is the register used when binding the LUKS devices to TPM2, as demonstrated in the example below.

%post
clevis luks bind -f -k - -d /dev/vda3 tpm2 '{"key":"ecc", "pcr_ids":"7"}' <<< "temppass"
%end

In the example above, an ECC key is used in place of an RSA key because ECC is significantly faster in performance β€” roughly 12 times faster than RSA.

Always set a strong and complex passphrase for your LUKS devices as a fallback measure. If TPM2-based automatic unlocking fails for any reason, having no passphrase could make system recovery completely impossible.

The scenario in which this problem arises after the system has been rebooted following the installation of security updates that included an upgrade to the shim-x64 package

The steps required to recover access to the system will vary depending on whether or not you have a backup passphrase available to manually unlock the LUKS devices.

If you have previously set a passphrase for your LUKS devices, follow the steps outlined below.

These operations must be carried out on every LUKS device present on the system. In the example below, there is only one LUKS device: /dev/vda3.

  • Manually unlock the LUKS devices using the passphrase
  • Identify and note the slot that contains the Clevis token
# clevis luks list -d /dev/vda3
0: tpm2 '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'

From above, we can see that the slot is use is slot 0.

Regenerate the binding.

# clevis luks regen -d /dev/vda3 -s 0
Regenerating binding (device /dev/vda3, slot 0):
Pin: tpm2, Config: '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'
Do you want to proceed? [ynYN] Y
Enter existing LUKS password: <YOUR PASS PHRASE>
Warning: Value 512 is outside of the allowed entropy range, adjusting it.
Binding regenerated successfully

If no passphrase has been configured for the LUKS devices, follow the recovery procedure outlined below.

Boot the system using the DVD and select Troubleshooting mode to begin the recovery process.

Access the shell directly by choosing 3) Skip to shell, which bypasses any attempt to mount the root device.

Mount the UEFI filesystem to /boot/efi and check the date/timestamp of the bootloader file.

In the example below, the UEFI partition is located on /dev/vda1.

sh-4.4# mkdir /boot/efi && mount /dev/vda1 /boot/efi
sh-4.4# ls -ld /boot/efi/EFI/centos /boot/efi/EFI/centos/shimx64.efi
drwxr-xr-x. 3 root root   4096 Jul  9 07:40 /boot/efi/EFI/centos
-rwxr-xr-x. 1 root root 924888 Apr  7 15:17 /boot/efi/EFI/centos/shimx64.efi

In the output shown above, the shimx64.efi file carries a timestamp of Apr 7 at 15:17, while the parent directory /boot/efi/EFI/centos shows a last modified date of Jul 9 at 07:40. This discrepancy strongly suggests that an update to the shim-x64 package was most likely applied on July 9th.

On a Linux system, visit CentOS’s Package Browser to download the two most recent releases of the shim-x64 package, then extract them to inspect the timestamps of the files they contain.

For this example, we assume that shim-x64-15.4-2.el8_1 is the latest release as indicated on the Package Browser page, and shim-x64-15-16.el8 is the previous version:

Extract the contents of both packages.

mydesktop $ cd Downloads
mydesktop $ LATEST=shim-x64-15.4-2.el8_1
mydesktop $ PREVIOUS=shim-x64-15-16.el8
mydesktop $ for pkg in $LATEST $PREVIOUS; do (mkdir $pkg && cd $pkg && rpm2cpio ../$pkg.x86_64.rpm | cpio -icdmu); done

Check dates.

mydesktop $ for pkg in $LATEST $PREVIOUS; do ls -l $pkg/boot/efi/EFI/centos/shimx64.efi; done
-rwx------. 1 rmetrich rmetrich 924888 Apr  7 17:17 shim-x64-15.4-2.el8_1/boot/efi/EFI/centos/shimx64.efi
-rwx------. 1 rmetrich rmetrich 1244488 Sep 22  2020 shim-x64-15-16.el8/boot/efi/EFI/centos/shimx64.efi

From the output above, we can confirm that the timestamps of the files in shim-x64-15.4-2.el8_1 match the date observed on the system being recovered β€” note that minor differences may appear due to timezone variations.

Based on this, we can reasonably conclude that the shim-x64 package was upgraded to version shim-x64-15.4-2.el8_1 on the affected system. Therefore, the previous version β€” shim-x64-15-16.el8 β€” is the one we need to use in order to restore the system to a working state.

Configure the network interface and retrieve the shim-x64 package from the Linux system.

Note: Since the rescue environment does not run an sshd daemon, the package must be transferred using either the ftp or scp command instead.

Network configuration must be done manually β€” use the ip link and ip addr commands for static network setups, or run dhclient if the network relies on DHCP for automatic address assignment.

sh-4.4# dhclient enp1s0
sh-4.4# scp rmetrich@mydesktop:Downloads/shim-x64-15-16.el8.x86_64.rpm /tmp/

Extract the package to overwrite the files in /boot/efi and verify the date of the boot loader file.

sh-4.4# cd / && rpm2cpio /tmp/shim-x64-15-16.el8.x86_64.rpm | cpio -icdmu
sh-4.4# ls -ld /boot/efi/EFI/centos/shimx64.efi
-rwxr-xr-x. 1 root root 1244488 Sep 22  2020 /boot/efi/EFI/centos/shimx64.efi

In the example shown above, the existing file was replaced by overwriting it with the older version, which carries a timestamp of September 22, 2020.

Exit from the rescue and reboot on the hard disk.

sh-4.4# exit

If the system still fails to boot with automatic LUKS unlocking after the bootloader rollback, repeat the same procedure using an even older version of the shim-x64 package.

If the system boots successfully, proceed with the final steps below to configure a fallback passphrase on the LUKS devices as a safety measure.

These operations must be carried out on every LUKS device on the system. In the example below, /dev/vda3 is the LUKS device being used.

Identify and note the slot that contains the Clevis token.

# clevis luks list -d /dev/vda3
0: tpm2 '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'

From the output above, we can identify that the Clevis token is stored in slot 0.

Retrieve the Clevis passphrase that is saved in the slot identified above.

# echo $(clevis luks pass -d /dev/vda3 -s 0)
<CLEVISPASSPHRASE>

Set up a strong recovery passphrase on the LUKS device to serve as a secure fallback option in case automatic TPM2-based unlocking fails in the future.

# cryptsetup luksAddKey /dev/vda3
Enter any existing passphrase: <CLEVISPASSPHRASE>
Enter new passphrase for key slot: <SOME FAIL SAFE PASS PHRASE>
Verify passphrase: <SOME FAIL SAFE PASS PHRASE>

Reinstall the shim-x64 package to ensure that the latest available version is now properly installed and in use on the system.

# yum -y reinstall shim-x64

Restart the system and go through the steps once more to complete the recovery process.

Cheers :+1: