This guide installs a ZFS mirror on LUKs encrypted containers. This leads to a performance hit, since writing requires each block of data to be encrypted twice. Addituionally the HP N54L has no AES CPU instruction. All IO is very CPU bound after this. I get a write throughput of about 8M/s. I am using the system to receive ZFS backups via zfs send/receive over a DSL connection. This is working fine since the connection speed is much slower than the writing speed. This setup is not for you if you need faster write speeds.
I used the Ubuntu 13.10 64bit Desktop Live CD and copied it to an USB stick with dd. Creating the USB stick with tools like unetbootin will not work, since the HP N54L will think it is a floppy and halt at the syslinux stage.
Install the ZFS native repository for Ubuntu:
apt-add-repository --yes ppa:zfs-native/stable apt-get update apt-get install debootstrap ubuntu-zfs
I have 2 WD Red 4 TB disks in the system. We need a small partition for grub and a small RAID for the unencrypted /boot partition. I create an additional very small raid, with a LUKS container inside, so that all keys can be derived from that. The rest of the space is for the LUKS container with a ZFS pool and the root filesystem inside.
parted -a optimal /dev/sda mklabel gpt parted -a minimal /dev/sda mkpart primary 1MiB 2MiB parted -a optimal /dev/sda set 1 bios_grub on parted -a optimal /dev/sda mkpart primary 2MiB 200MiB parted -a optimal /dev/sda set 2 raid on parted -a optimal /dev/sda mkpart primary 200MiB 220MiB parted -a optimal /dev/sda set 3 raid on parted -a optimal /dev/sda mkpart primary 220MiB 100% parted -a optimal /dev/sdb mklabel gpt parted -a minimal /dev/sdb mkpart primary 1MiB 2MiB parted -a optimal /dev/sdb set 1 bios_grub on parted -a optimal /dev/sdb mkpart primary 2MiB 200MiB parted -a optimal /dev/sdb set 2 raid on parted -a optimal /dev/sdb mkpart primary 200MiB 220MiB parted -a optimal /dev/sdb set 3 raid on parted -a optimal /dev/sdb mkpart primary 220MiB 100%
Next we create the raid
apt-get install mdadm --no-install-recommends mdadm --create --verbose /dev/md/0 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2 #Answer Yes, if asked for 1.x metadata format mdadm --create --verbose /dev/md/1 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3 #Answer Yes, if asked for 1.x metadata format
And then create the LUKS containers
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/md/1 cryptsetup luksOpen /dev/md/1 cryptroot /lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup -s 512 -c aes-xts-plain64 -h sha512 luksFormat /dev/sda4 /lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sda4 crypt1 /lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup -s 512 -c aes-xts-plain64 -h sha512 luksFormat /dev/sdb4 /lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sdb4 crypt2
Create the zpool
zpool create -O mountpoint=none -o ashift=12 rpool mirror /dev/mapper/crypt1 /dev/mapper/crypt2 zfs create -o mountpoint=/ rpool/ROOT zfs create -o mountpoint=none rpool/DATA zpool set bootfs=rpool/ROOT rpool zpool export rpool zpool import -R /mnt rpool
install the base system
debootstrap saucy /mnt mount --bind /dev/ /mnt/dev
The HP N54L needs the uhci-hcd module in the initramfs. The release version of initramfs-tools is broken. There is an update in the proposed section, that fixes this. Without you won’t be able to enter your password on boot.
wget http://launchpadlibrarian.net/154931909/initramfs-tools-bin_0.103ubuntu1.1_amd64.deb wget http://launchpadlibrarian.net/154931949/initramfs-tools_0.103ubuntu1.1_all.deb mv initramfs-tools* /mnt/ chroot /mnt /bin/bash --login dpkg -i initramfs-tools* exit
Change into the new system
chroot /mnt /bin/bash --login
You are inside the new system now.
mkfs.ext4 /dev/mapper/cryptroot mkfs.ext4 /dev/md0 mount /dev/md0 /boot mount -t proc proc /proc mount -t sysfs sysfs /sys ln -sf /dev/mapper/cryptroot /dev/cryptroot ln -sf /dev/mapper/crypt1 /dev/crypt1 ln -sf /dev/mapper/crypt2 /dev/crypt2 locale-gen $LANG locale-gen en_US.UTF-8 apt-get update apt-get install software-properties-common apt-add-repository --yes ppa:zfs-native/stable apt-add-repository --yes ppa:zfs-native/grub #At the time of writing, no saucy ifs grub was released, edit /etc/apt/sources.list.d/zfs-native-grub-saucy.list to use the raring version sed -i 's|saucy|raring|' /etc/apt/sources.list.d/zfs-native-grub-saucy.list apt-get update apt-get install linux-image-server linux-headers-server ubuntu-minimal #skip installation of grub for now apt-get install ubuntu-zfs cryptsetup apt-get install mdadm --no-install-recommends apt-get install grub-pc mountall #grub will be downgraded grub-install /dev/sda grub-install /dev/sdb apt-get install zfs-initramfs echo "cryptroot UUID=$(blkid /dev/md/1 -s UUID -o value) none luks">> /etc/crypttab echo "crypt1 UUID=$(blkid /dev/sda4 -s UUID -o value) cryptroot luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived" >> /etc/crypttab echo "crypt2 UUID=$(blkid /dev/sdb4 -s UUID -o value) cryptroot luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived" >> /etc/crypttab echo "UUID=$(blkid /dev/md0 -s UUID -o value) /boot ext4 rw,relatime,stripe=4,data=ordered 0 0"> /etc/fstab #we need a fake root system so that the crypt tools get added to the initramfs #we cannot use /dev/mapper/cryptroot, since the initramfs is missing the derived script then... echo "/dev/mapper/crypt1 / none none 0 0">> /etc/fstab echo "target=cryptroot,source=UUID=$(blkid /dev/md/1 -s UUID -o value),key=none,rootdev"> /etc/initramfs-tools/conf.d/cryptroot echo "target=crypt1,source=UUID=$(blkid /dev/sda4 -s UUID -o value),key=cryptroot,keyscript=/lib/cryptsetup/scripts/decrypt_derived,rootdev" >> /etc/initramfs-tools/conf.d/cryptroot echo "target=crypt2,source=UUID=$(blkid /dev/sdb4 -s UUID -o value),key=cryptroot,keyscript=/lib/cryptsetup/scripts/decrypt_derived,rootdev" >> /etc/initramfs-tools/conf.d/cryptroot update-initramfs -u -k all sed -i 's|GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"|GRUB_CMDLINE_LINUX_DEFAULT="noplymouth"|' /etc/default/grub update-grub adduser <myuser> adduser <myuser> sudo /etc/init.d/mdadm stop umount /boot umount /sys umount /proc exit
You are back in the host system now
umount /mnt/dev zpool export rpool reboot
Rescue Mode
Boot the lived again and then:
apt-add-repository --yes ppa:zfs-native/stable apt-get update apt-get install debootstrap ubuntu-zfs apt-get install mdadm --no-install-recommends mdadm --assemble /dev/md/0 mdadm --assemble /dev/md/1 cryptsetup luksOpen /dev/md/1 cryptroot /lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sda4 crypt1 /lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sdb4 crypt2 zpool import -R /mnt rpool mount --bind /dev/ /mnt/dev chroot /mnt /bin/bash --login mount /dev/md0 /boot mount -t proc proc /proc mount -t sysfs sysfs /sys ln -sf /dev/mapper/cryptroot /dev/cryptroot ln -sf /dev/mapper/crypt1 /dev/crypt1 ln -sf /dev/mapper/crypt2 /dev/crypt2
To leave the system again:
umount /boot umount /sys umount /proc exit umount /mnt/dev zpool export rpool
Optionally unlock your server via SSH
apt-add-repository universe apt-get update apt-get install openssh-server #make sure that openssh-server is installed before dropbear apt-get install dropbear #use the same host key in initramfs as in real system cp /etc/dropbear/dropbear_dss_host_key /etc/initramfs-tools/etc/dropbear/dropbear_dss_host_key cp /etc/dropbear/dropbear_rsa_host_key /etc/initramfs-tools/etc/dropbear/dropbear_rsa_host_key #add your public keys to /etc/initramfs-tools/root/.ssh/authorized_keys
Add the file /etc/initramfs-tools/hooks/unlock with the following contents:
#!/bin/sh -e
PREREQS=""
prereqs() { echo "$PREREQS"; }
case "$1" in
prereqs)
prereqs
exit 0
;;
esac
## hook-functions provides copy_exec()
. /usr/share/initramfs-tools/hook-functions
echo "/sbin/cryptsetup luksOpen /dev/md/1 cryptroot">$DESTDIR/bin/unlock
echo 'kill $(pidof cryptsetup)'>>$DESTDIR/bin/unlock
echo 'kill $(pidof plymouth)'>>$DESTDIR/bin/unlock
chmod a+x $DESTDIR/bin/unlock
chmod a+x /etc/initramfs-tools/hooks/unlock update-initramfs -u -k all
You can now log into your initramfs during boot via shh with the root user and unlock the disks by calling:
unlock exit
Final remarks
You should probably backup your LUKs headers and store them in a safe place:
cryptsetup luksHeaderBackup /dev/md/1 --header-backup-file md1.luks cryptsetup luksHeaderBackup /dev/sda4 --header-backup-file sda4.luks cryptsetup luksHeaderBackup /dev/sdb4 --header-backup-file sdb4.luks
If you have sugustions on how to improve this setup, feel free to post them in the comments.
Sources
http://www.larsko.org/ZfsUbuntu
http://blog.neutrino.es/2011/unlocking-a-luks-encrypted-root-partition-remotely-via-ssh/