This guide installs a ZFS mirror on LUKs encrypted containers. This leads to a performance hit, since writing requires each block of data to be encrypted twice. Addituionally the HP N54L has no AES CPU instruction. All IO is very CPU bound after this. I get a write throughput of about 8M/s. I am using the system to receive ZFS backups via zfs send/receive over a DSL connection. This is working fine since the connection speed is much slower than the writing speed. This setup is not for you if you need faster write speeds.

I used the Ubuntu 13.10 64bit Desktop Live CD and copied it to an USB stick with dd. Creating the USB stick with tools like unetbootin will not work, since the HP N54L will think it is a floppy and halt at the syslinux stage.

Install the ZFS native repository for Ubuntu:

apt-add-repository --yes ppa:zfs-native/stable
apt-get update
apt-get install debootstrap ubuntu-zfs

I have 2 WD Red 4 TB disks in the system. We need a small partition for grub and a small RAID for the unencrypted /boot partition. I create an additional very small raid, with a LUKS container inside, so that all keys can be derived from that. The rest of the space is for the LUKS container with a ZFS pool and the root filesystem inside.

parted -a optimal /dev/sda mklabel gpt
parted -a minimal /dev/sda mkpart primary 1MiB 2MiB
parted -a optimal /dev/sda set 1 bios_grub on
parted -a optimal /dev/sda mkpart primary 2MiB 200MiB
parted -a optimal /dev/sda set 2 raid on
parted -a optimal /dev/sda mkpart primary 200MiB 220MiB
parted -a optimal /dev/sda set 3 raid on
parted -a optimal /dev/sda mkpart primary 220MiB 100%

parted -a optimal /dev/sdb mklabel gpt
parted -a minimal /dev/sdb mkpart primary 1MiB 2MiB
parted -a optimal /dev/sdb set 1 bios_grub on
parted -a optimal /dev/sdb mkpart primary 2MiB 200MiB
parted -a optimal /dev/sdb set 2 raid on
parted -a optimal /dev/sdb mkpart primary 200MiB 220MiB
parted -a optimal /dev/sdb set 3 raid on
parted -a optimal /dev/sdb mkpart primary 220MiB 100%

Next we create the raid

apt-get install mdadm --no-install-recommends
mdadm --create --verbose /dev/md/0 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2
#Answer Yes, if asked for 1.x metadata format
mdadm --create --verbose /dev/md/1 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
#Answer Yes, if asked for 1.x metadata format

And then create the LUKS containers

cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/md/1
cryptsetup luksOpen /dev/md/1 cryptroot
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup -s 512 -c aes-xts-plain64 -h sha512 luksFormat /dev/sda4
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sda4 crypt1
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup -s 512 -c aes-xts-plain64 -h sha512 luksFormat /dev/sdb4
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sdb4 crypt2

Create the zpool

zpool create -O mountpoint=none -o ashift=12 rpool mirror /dev/mapper/crypt1 /dev/mapper/crypt2
zfs create -o mountpoint=/ rpool/ROOT
zfs create -o mountpoint=none rpool/DATA
zpool set bootfs=rpool/ROOT rpool
zpool export rpool
zpool import -R /mnt rpool

install the base system

debootstrap saucy /mnt
mount --bind /dev/ /mnt/dev

The HP N54L needs the uhci-hcd module in the initramfs. The release version of initramfs-tools is broken. There is an update in the proposed section, that fixes this. Without you won’t be able to enter your password on boot.

wget http://launchpadlibrarian.net/154931909/initramfs-tools-bin_0.103ubuntu1.1_amd64.deb
wget http://launchpadlibrarian.net/154931949/initramfs-tools_0.103ubuntu1.1_all.deb
mv initramfs-tools* /mnt/
chroot /mnt /bin/bash --login
dpkg -i initramfs-tools*
exit

Change into the new system

chroot /mnt /bin/bash --login

You are inside the new system now.

mkfs.ext4 /dev/mapper/cryptroot
mkfs.ext4 /dev/md0 
mount /dev/md0 /boot
mount -t proc proc /proc
mount -t sysfs sysfs /sys
ln -sf /dev/mapper/cryptroot /dev/cryptroot
ln -sf /dev/mapper/crypt1 /dev/crypt1
ln -sf /dev/mapper/crypt2 /dev/crypt2

locale-gen $LANG
locale-gen en_US.UTF-8

apt-get update
apt-get install software-properties-common

apt-add-repository --yes ppa:zfs-native/stable
apt-add-repository --yes ppa:zfs-native/grub
#At the time of writing, no saucy ifs grub was released, edit /etc/apt/sources.list.d/zfs-native-grub-saucy.list to use the raring version
sed -i 's|saucy|raring|' /etc/apt/sources.list.d/zfs-native-grub-saucy.list
apt-get update

apt-get install linux-image-server linux-headers-server ubuntu-minimal
#skip installation of grub for now

apt-get install ubuntu-zfs cryptsetup

apt-get install mdadm --no-install-recommends
apt-get install grub-pc mountall
#grub will be downgraded
grub-install /dev/sda
grub-install /dev/sdb

apt-get install zfs-initramfs

echo "cryptroot UUID=$(blkid /dev/md/1 -s UUID -o value) none luks">> /etc/crypttab
echo "crypt1 UUID=$(blkid /dev/sda4 -s UUID -o value) cryptroot luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived" >> /etc/crypttab 
echo "crypt2 UUID=$(blkid /dev/sdb4 -s UUID -o value) cryptroot luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived" >> /etc/crypttab

echo "UUID=$(blkid /dev/md0 -s UUID -o value) /boot ext4 rw,relatime,stripe=4,data=ordered 0 0"> /etc/fstab
#we need a fake root system so that the crypt tools get added to the initramfs
#we cannot use /dev/mapper/cryptroot, since the initramfs is missing the derived script then...
echo "/dev/mapper/crypt1 / none none 0 0">> /etc/fstab

echo "target=cryptroot,source=UUID=$(blkid /dev/md/1 -s UUID -o value),key=none,rootdev"> /etc/initramfs-tools/conf.d/cryptroot
echo "target=crypt1,source=UUID=$(blkid /dev/sda4 -s UUID -o value),key=cryptroot,keyscript=/lib/cryptsetup/scripts/decrypt_derived,rootdev" >> /etc/initramfs-tools/conf.d/cryptroot
echo "target=crypt2,source=UUID=$(blkid /dev/sdb4 -s UUID -o value),key=cryptroot,keyscript=/lib/cryptsetup/scripts/decrypt_derived,rootdev" >> /etc/initramfs-tools/conf.d/cryptroot

update-initramfs -u -k all

sed -i 's|GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"|GRUB_CMDLINE_LINUX_DEFAULT="noplymouth"|' /etc/default/grub

update-grub

adduser <myuser>
adduser <myuser> sudo

/etc/init.d/mdadm stop
umount /boot
umount /sys
umount /proc
exit

You are back in the host system now

umount /mnt/dev
zpool export rpool
reboot

 Rescue Mode

Boot the lived again and then:

apt-add-repository --yes ppa:zfs-native/stable
apt-get update
apt-get install debootstrap ubuntu-zfs
apt-get install mdadm --no-install-recommends

mdadm --assemble /dev/md/0
mdadm --assemble /dev/md/1

cryptsetup luksOpen /dev/md/1 cryptroot
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sda4 crypt1
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sdb4 crypt2

zpool import -R /mnt rpool

mount --bind /dev/ /mnt/dev
chroot /mnt /bin/bash --login

mount /dev/md0 /boot
mount -t proc proc /proc
mount -t sysfs sysfs /sys
ln -sf /dev/mapper/cryptroot /dev/cryptroot
ln -sf /dev/mapper/crypt1 /dev/crypt1
ln -sf /dev/mapper/crypt2 /dev/crypt2

To leave the system again:

umount /boot
umount /sys
umount /proc
exit
umount /mnt/dev
zpool export rpool

 

Optionally unlock your server via SSH

apt-add-repository universe
apt-get update

apt-get install openssh-server
#make sure that openssh-server is installed before dropbear
apt-get install dropbear

#use the same host key in initramfs as in real system
cp /etc/dropbear/dropbear_dss_host_key /etc/initramfs-tools/etc/dropbear/dropbear_dss_host_key
cp /etc/dropbear/dropbear_rsa_host_key /etc/initramfs-tools/etc/dropbear/dropbear_rsa_host_key

#add your public keys to /etc/initramfs-tools/root/.ssh/authorized_keys

Add the file /etc/initramfs-tools/hooks/unlock with the following contents:

#!/bin/sh -e

PREREQS=""

prereqs() { echo "$PREREQS"; }

case "$1" in
    prereqs)
    prereqs
    exit 0
    ;;
esac

## hook-functions provides copy_exec()
. /usr/share/initramfs-tools/hook-functions

echo "/sbin/cryptsetup luksOpen /dev/md/1 cryptroot">$DESTDIR/bin/unlock
echo 'kill $(pidof cryptsetup)'>>$DESTDIR/bin/unlock
echo 'kill $(pidof plymouth)'>>$DESTDIR/bin/unlock

chmod a+x $DESTDIR/bin/unlock

 

chmod a+x /etc/initramfs-tools/hooks/unlock
update-initramfs -u -k all

You can now log into your initramfs during boot via shh with the root user and unlock the disks by calling:

unlock
exit

Final remarks

You should probably backup your LUKs headers and store them in a safe place:

cryptsetup luksHeaderBackup /dev/md/1 --header-backup-file md1.luks
cryptsetup luksHeaderBackup /dev/sda4 --header-backup-file sda4.luks
cryptsetup luksHeaderBackup /dev/sdb4 --header-backup-file sdb4.luks

If you have sugustions on how to improve this setup, feel free to post them in the comments.

Sources

http://www.larsko.org/ZfsUbuntu

http://blog.neutrino.es/2011/unlocking-a-luks-encrypted-root-partition-remotely-via-ssh/

http://wiki.ubuntuusers.de/LUKS

This guide installs a ZFS mirror on LUKs encrypted containers. This leads to a performance hit, since writing requires each block of data to be encrypted twice. Addituionally the HP...