Running Confidential VM with AMD SEV-SNP

What is AMD SEV-SNP?

AMD SEV (Secure Encrypted Virtualization) is a technology to isolates guest VMs from the hypervisor.

AMD SEV provides only Memory Encryption.

AMD SEV-ES (SEV Encrypted State) provides encryption for guest VM's registers.

AMD SEV-SNP (SEV Secure Nested Paging) is the latest SEV techonology and it provides strong memory integrity protection and improved attestation. SEV-SNP has the concept of memory page owner. Only page owner (guest VM) can read memory page and others cannot read it. It prevents others from altering memory contents.

(cited from page 7 in AMD SEV-SNP:Strengthening VM Isolation with Integrity Protection and More)

The detail is described in AMD SEV-SNP:Strengthening VM Isolation with Integrity Protection and More

Tested Environment

CPU: AMD EPYC 7313P
MotherBoard: Supermicro H12SSL-I (BIOS Version 2.6a)
Memory: 128GB
OS: Ubuntu 23.10

Used Software

SNP Firmware: 1.55.16
Host Kernel: https://github.com/AMDESE/linux snp-host-latest branch, 5a170ce1a08259ac57a9074e1e7a170d6b8c0cda
Guest Kernel: https://github.com/AMDESE/linux snp-guest-latest branch, 690558b32fe6402c309341fbae65f41decb81592
QEMU: https://github.com/AMDESE/qemu snp-latest branch, fe4c9e8e7e7ddac4b19c4366c1f105ffc4a78482
OVMF: https://github.com/AMDESE/ovmf snp-latest branch, 09fbe92dc545779671d7fd89a5bd4f1b14f7e69b

Configure Host BIOS

We need to configure BIOS to enable SEV-SNP support

Advanced -> NB Configuration

Configure SEV-SNP Support to Enabled

Advanced -> CPU Configuration

Configure SMEE to Enabled
Configure SEV ASID Count to 509 ASIDs
Configure SEV-ES ASID Space Limit Control to Manual
Configure SEV-ES ASID Space Limit to 100
Configure SNP Memory (RMP Table) Coverage to Enabled

Install the Latest Firmware

Download latest SNP firmware from AMD official Website. CPU is EPYC 7313P (Millan), so here we download amd_sev_fam19h_model0xh_1.55.16.zip

~$ wget https://download.amd.com/developer/eula/sev/amd_sev_fam19h_model0xh_1.55.16.zip
~$ unzip amd_sev_fam19h_model0xh_1.55.16.zip
Archive:  amd_sev_fam19h_model0xh_1.55.16.zip
  inflating: amd_sev_fam19h_model0xh_1.55.16.sbin
  inflating: Milan Release Notes.txt
~$ sudo rm -rf /lib/firmware/amd/amd_sev_fam19h_model0xh.sbin
~$ sudo cp amd_sev_fam19h_model0xh_1.55.16.sbin /lib/firmware/amd/amd_sev_fam19h_model0xh.sbin

Build Host Kernel for SEV-SNP

Mainline kernel does not have support for SEV-SNP host. We need to build out-of-tree SEV-SNP supported host kernel. If you want to build kernel manually, clone source code from https://github.com/AMDESE/linux (branch is snp-host-latest).

Currently(2023/12/30), host kernel commit id is 5a170ce1a08259ac57a9074e1e7a170d6b8c0cda

~$ sudo apt -y install build-essential bison libssl-dev flex libelf-dev libncurses-dev
~$ git clone --single -b snp-latest https://github.com/AMDESE/AMDSEV/
~$ cd AMDSEV
~/AMDSEV$ ./build.sh kernel host
~/AMDSEV$ ls -l linux/
total 342996
drwxrwxr-x 27 naoki naoki      4096 Dec 30 06:42 guest
drwxrwxr-x 28 naoki naoki      4096 Dec 30 06:52 host
-rw-r--r--  1 naoki naoki   9683256 Dec 30 06:54 linux-headers-6.6.0-rc1-snp-host-5a170ce1a082_6.6.0-rc1-g5a170ce1a082-2_amd64.deb
-rw-r--r--  1 naoki naoki  88048938 Dec 30 06:54 linux-image-6.6.0-rc1-snp-host-5a170ce1a082_6.6.0-rc1-g5a170ce1a082-2_amd64.deb
-rw-r--r--  1 naoki naoki 252103406 Dec 30 06:54 linux-image-6.6.0-rc1-snp-host-5a170ce1a082-dbg_6.6.0-rc1-g5a170ce1a082-2_amd64.deb
-rw-r--r--  1 naoki naoki   1362734 Dec 30 06:54 linux-libc-dev_6.6.0-rc1-g5a170ce1a082-2_amd64.deb
-rw-rw-r--  1 naoki naoki      7744 Dec 30 06:54 linux-upstream_6.6.0-rc1-g5a170ce1a082-2_amd64.buildinfo
-rw-rw-r--  1 naoki naoki      2961 Dec 30 06:55 linux-upstream_6.6.0-rc1-g5a170ce1a082-2_amd64.changes

Sucessfully built, deb packages exist. Install them and reboot. To enable SEV-SNP enabled KVM, module options are required. debug_swap=0 is required to disable new VMSA fields. The detail is in https://github.com/AMDESE/AMDSEV/issues/195.

~/AMDSEV$ sudo dpkg -i linux/*.deb
~/AMDSEV$ echo "options kvm_amd sev-snp=1 sev=1 sev-es=1 debug_swap=0" | sudo tee /etc/modprobe.d/kvm.conf
~/AMDSEV$ sudo reboot

Boot from the built kernel. If everything works correctly, we can see below log.

$ uname -r
6.6.0-rc1-snp-host-5a170ce1a082
$ sudo dmesg | grep -e SEV
[sudo] password for naoki:
[    0.716831] SEV-SNP: RMP table physical address [0x0000000087800000 - 0x00000000a7dfffff]
[    5.068960] ccp 0000:47:00.1: SEV firmware update successful
[    6.904814] ccp 0000:47:00.1: SEV API:1.55 build:16
[    6.904820] ccp 0000:47:00.1: SEV-SNP API:1.55 build:16
[    6.916442] kvm_amd: SEV-ES and SEV-SNP supported: 99 ASIDs
[    6.916444] kvm_amd: SEV enabled (ASIDs 100 - 509)
[    6.916446] kvm_amd: SEV-ES enabled (ASIDs 1 - 99)
$ ls -l /dev/sev
crw------- 1 root root 10, 122 Dec 30 07:08 /dev/sev

Build SEV-SNP supported QEMU

Build SEV-SNP supported QEMU. Ubuntu 23.10 has default gcc version 13, but gcc-13 cannot build QEMU correctly. So, we use gcc version 12 to build QEMU.

~/AMDSEV$ sudo apt install gcc-12 g++-12
~/AMDSEV$ CC=gcc-12 CXX=g++-12 ./build.sh qemu
~/AMDSEV$ ls -l usr/local/bin/qemu-system-x86_64
-rwxr-xr-x 1 naoki naoki 65718848 Dec 30 08:35 usr/local/bin/qemu-system-x86_64

Build the dedicated UEFI Firmware

To run linux kernel in SEV-SNP VM, the dedicated UEFI Firmware (OVMF) is required. To build the dedicated OVMF, we need to patch build script common.sh.

~/AMDSEV$ git diff
diff --git a/common.sh b/common.sh
index e4087d5..171779a 100755
--- a/common.sh
+++ b/common.sh
@@ -126,7 +126,7 @@ build_install_ovmf()
                GCCVERS="GCC5"
        fi

-       BUILD_CMD="nice build -q --cmd-len=64436 -DDEBUG_ON_SERIAL_PORT=TRUE -n $(getconf _NPROCESSORS_ONLN) ${GCCVERS:+-t $GCCVERS} -a X64 -p OvmfPkg/OvmfPkgX64.dsc"
+       BUILD_CMD="nice build -q --cmd-len=64436 -DDEBUG_ON_SERIAL_PORT=TRUE -n $(getconf _NPROCESSORS_ONLN) ${GCCVERS:+-t $GCCVERS} -a X64 -p OvmfPkg/AmdSev/AmdSevX64.dsc"

        # initialize git repo, or update existing remote to currently configured one
        if [ -d ovmf ]; then
@@ -150,11 +150,11 @@ build_install_ovmf()
                run_cmd git submodule update --init --recursive
                run_cmd make -C BaseTools
                . ./edksetup.sh --reconfig
+               run_cmd touch OvmfPkg/AmdSev/Grub/grub.efi
                run_cmd $BUILD_CMD

                mkdir -p $DEST
-               run_cmd cp -f Build/OvmfX64/DEBUG_$GCCVERS/FV/OVMF_CODE.fd $DEST
-               run_cmd cp -f Build/OvmfX64/DEBUG_$GCCVERS/FV/OVMF_VARS.fd $DEST
+               run_cmd cp -f Build/AmdSev/DEBUG_$GCCVERS/FV/OVMF.fd $DEST

                COMMIT=$(git log --format="%h" -1 HEAD)
                run_cmd echo $COMMIT >../source-commit.ovmf

After modifying common.sh, build the dedicated OVMF.

~$ ./build.sh ovmf
~$ ls -l usr/local/share/qemu/
total 4096
-rw-rw-r-- 1 naoki naoki 4194304 Dec 30 08:25 OVMF.fd

Build Guest Linux Kernel

Build guest linux kernel with SEV_GUEST enabled.

~/AMDSEV$ git diff
diff --git a/common.sh b/common.sh
index e4087d5..b2634d6 100755
--- a/common.sh
+++ b/common.sh
@@ -84,7 +84,7 @@ build_kernel()
                        run_cmd ./scripts/config --disable SYSTEM_TRUSTED_KEYS
                        run_cmd ./scripts/config --disable SYSTEM_REVOCATION_KEYS
                        run_cmd ./scripts/config --disable MODULE_SIG_KEY
-                       run_cmd ./scripts/config --module  SEV_GUEST
+                       run_cmd ./scripts/config --enable  SEV_GUEST
                        run_cmd ./scripts/config --disable IOMMU_DEFAULT_PASSTHROUGH
                        run_cmd ./scripts/config --disable PREEMPT_COUNT
                        run_cmd ./scripts/config --disable PREEMPTION

~/AMDSEV$ ./build.sh kernel guest
~/AMDSEV$ ls -l linux/guest/arch/x86/boot/bzImage
-rw-rw-r-- 1 naoki naoki 11440128 Dec 30 09:17 linux/guest/arch/x86/boot/bzImage

Run with Measurement

I prepared scripts to launch guest VM with measurement. Measurement validates memory integrity when the SEV-SNP guest VM starts.

# Setup Rust environment
~$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
~$ rustup target add x86_64-unknown-linux-musl

~$ git clone --recursive https://github.com/naoki9911/simple-sev-snp
~$ cd simple-sev-snp

# Build SEV-SNP guest tools
~/simple-sev-snp$ cd snpguest
~/simple-sev-snp/snpguest$ cargo build --target x86_64-unknown-linux-musl
~/simple-sev-snp/snpguest$ cd ../

# Create small initrd with alpine rootfs
~/simple-sev-snp$ ./create_rootfs.sh

# Generate private keys to sign launch digests 
~/simple-sev-snp$ ./gen_key.sh

# Then, launch with measurement!
~/simple-sev-snp$ sudo ./launch.sh

# You can get Attestation Report with snpguest
~ # ./snpguest report report.bin req.txt --random
~ # ./snpguest display report report.bin

Attestation Report (1184 bytes):
Version:                      2
Guest SVN:                    0

    Guest Policy (196608):
    ABI Major:     0
    ABI Minor:     0
    SMT Allowed:   1
    Migrate MA:    0
    Debug Allowed: 0
    Single Socket: 0
Family ID:
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
...