This guide installs a ZFS mirror on LUKs encrypted containers. This leads to a performance hit, since writing requires each block of data to be encrypted twice. Addituionally the HP N54L has no AES CPU instruction. All IO is very CPU bound after this. I get a write throughput of about 8M/s. I am using the system to receive ZFS backups via zfs send/receive over a DSL connection. This is working fine since the connection speed is much slower than the writing speed. This setup is not for you if you need faster write speeds.
I used the Ubuntu 13.10 64bit Desktop Live CD and copied it to an USB stick with dd. Creating the USB stick with tools like unetbootin will not work, since the HP N54L will think it is a floppy and halt at the syslinux stage.
Install the ZFS native repository for Ubuntu:
apt-add-repository --yes ppa:zfs-native/stable
apt-get update
apt-get install debootstrap ubuntu-zfs
I have 2 WD Red 4 TB disks in the system. We need a small partition for grub and a small RAID for the unencrypted /boot partition. I create an additional very small raid, with a LUKS container inside, so that all keys can be derived from that. The rest of the space is for the LUKS container with a ZFS pool and the root filesystem inside.
parted -a optimal /dev/sda mklabel gpt
parted -a minimal /dev/sda mkpart primary 1MiB 2MiB
parted -a optimal /dev/sda set 1 bios_grub on
parted -a optimal /dev/sda mkpart primary 2MiB 200MiB
parted -a optimal /dev/sda set 2 raid on
parted -a optimal /dev/sda mkpart primary 200MiB 220MiB
parted -a optimal /dev/sda set 3 raid on
parted -a optimal /dev/sda mkpart primary 220MiB 100%
parted -a optimal /dev/sdb mklabel gpt
parted -a minimal /dev/sdb mkpart primary 1MiB 2MiB
parted -a optimal /dev/sdb set 1 bios_grub on
parted -a optimal /dev/sdb mkpart primary 2MiB 200MiB
parted -a optimal /dev/sdb set 2 raid on
parted -a optimal /dev/sdb mkpart primary 200MiB 220MiB
parted -a optimal /dev/sdb set 3 raid on
parted -a optimal /dev/sdb mkpart primary 220MiB 100%
Next we create the raid
apt-get install mdadm --no-install-recommends
mdadm --create --verbose /dev/md/0 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2
#Answer Yes, if asked for 1.x metadata format
mdadm --create --verbose /dev/md/1 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
#Answer Yes, if asked for 1.x metadata format
And then create the LUKS containers
cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512 /dev/md/1
cryptsetup luksOpen /dev/md/1 cryptroot
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup -s 512 -c aes-xts-plain64 -h sha512 luksFormat /dev/sda4
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sda4 crypt1
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup -s 512 -c aes-xts-plain64 -h sha512 luksFormat /dev/sdb4
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sdb4 crypt2
Create the zpool
zpool create -O mountpoint=none -o ashift=12 rpool mirror /dev/mapper/crypt1 /dev/mapper/crypt2
zfs create -o mountpoint=/ rpool/ROOT
zfs create -o mountpoint=none rpool/DATA
zpool set bootfs=rpool/ROOT rpool
zpool export rpool
zpool import -R /mnt rpool
install the base system
debootstrap saucy /mnt
mount --bind /dev/ /mnt/dev
The HP N54L needs the uhci-hcd module in the initramfs. The release version of initramfs-tools is broken. There is an update in the proposed section, that fixes this. Without you won’t be able to enter your password on boot.
wget http://launchpadlibrarian.net/154931909/initramfs-tools-bin_0.103ubuntu1.1_amd64.deb
wget http://launchpadlibrarian.net/154931949/initramfs-tools_0.103ubuntu1.1_all.deb
mv initramfs-tools* /mnt/
chroot /mnt /bin/bash --login
dpkg -i initramfs-tools*
exit
Change into the new system
chroot /mnt /bin/bash --login
You are inside the new system now.
mkfs.ext4 /dev/mapper/cryptroot
mkfs.ext4 /dev/md0
mount /dev/md0 /boot
mount -t proc proc /proc
mount -t sysfs sysfs /sys
ln -sf /dev/mapper/cryptroot /dev/cryptroot
ln -sf /dev/mapper/crypt1 /dev/crypt1
ln -sf /dev/mapper/crypt2 /dev/crypt2
locale-gen $LANG
locale-gen en_US.UTF-8
apt-get update
apt-get install software-properties-common
apt-add-repository --yes ppa:zfs-native/stable
apt-add-repository --yes ppa:zfs-native/grub
#At the time of writing, no saucy ifs grub was released, edit /etc/apt/sources.list.d/zfs-native-grub-saucy.list to use the raring version
sed -i 's|saucy|raring|' /etc/apt/sources.list.d/zfs-native-grub-saucy.list
apt-get update
apt-get install linux-image-server linux-headers-server ubuntu-minimal
#skip installation of grub for now
apt-get install ubuntu-zfs cryptsetup
apt-get install mdadm --no-install-recommends
apt-get install grub-pc mountall
#grub will be downgraded
grub-install /dev/sda
grub-install /dev/sdb
apt-get install zfs-initramfs
echo "cryptroot UUID=$(blkid /dev/md/1 -s UUID -o value) none luks">> /etc/crypttab
echo "crypt1 UUID=$(blkid /dev/sda4 -s UUID -o value) cryptroot luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived" >> /etc/crypttab
echo "crypt2 UUID=$(blkid /dev/sdb4 -s UUID -o value) cryptroot luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived" >> /etc/crypttab
echo "UUID=$(blkid /dev/md0 -s UUID -o value) /boot ext4 rw,relatime,stripe=4,data=ordered 0 0"> /etc/fstab
#we need a fake root system so that the crypt tools get added to the initramfs
#we cannot use /dev/mapper/cryptroot, since the initramfs is missing the derived script then...
echo "/dev/mapper/crypt1 / none none 0 0">> /etc/fstab
echo "target=cryptroot,source=UUID=$(blkid /dev/md/1 -s UUID -o value),key=none,rootdev"> /etc/initramfs-tools/conf.d/cryptroot
echo "target=crypt1,source=UUID=$(blkid /dev/sda4 -s UUID -o value),key=cryptroot,keyscript=/lib/cryptsetup/scripts/decrypt_derived,rootdev" >> /etc/initramfs-tools/conf.d/cryptroot
echo "target=crypt2,source=UUID=$(blkid /dev/sdb4 -s UUID -o value),key=cryptroot,keyscript=/lib/cryptsetup/scripts/decrypt_derived,rootdev" >> /etc/initramfs-tools/conf.d/cryptroot
update-initramfs -u -k all
sed -i 's|GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"|GRUB_CMDLINE_LINUX_DEFAULT="noplymouth"|' /etc/default/grub
update-grub
adduser <myuser>
adduser <myuser> sudo
/etc/init.d/mdadm stop
umount /boot
umount /sys
umount /proc
exit
You are back in the host system now
umount /mnt/dev
zpool export rpool
reboot
Rescue Mode
Boot the lived again and then:
apt-add-repository --yes ppa:zfs-native/stable
apt-get update
apt-get install debootstrap ubuntu-zfs
apt-get install mdadm --no-install-recommends
mdadm --assemble /dev/md/0
mdadm --assemble /dev/md/1
cryptsetup luksOpen /dev/md/1 cryptroot
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sda4 crypt1
/lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetup luksOpen /dev/sdb4 crypt2
zpool import -R /mnt rpool
mount --bind /dev/ /mnt/dev
chroot /mnt /bin/bash --login
mount /dev/md0 /boot
mount -t proc proc /proc
mount -t sysfs sysfs /sys
ln -sf /dev/mapper/cryptroot /dev/cryptroot
ln -sf /dev/mapper/crypt1 /dev/crypt1
ln -sf /dev/mapper/crypt2 /dev/crypt2
To leave the system again:
umount /boot
umount /sys
umount /proc
exit
umount /mnt/dev
zpool export rpool
Optionally unlock your server via SSH
apt-add-repository universe
apt-get update
apt-get install openssh-server
#make sure that openssh-server is installed before dropbear
apt-get install dropbear
#use the same host key in initramfs as in real system
cp /etc/dropbear/dropbear_dss_host_key /etc/initramfs-tools/etc/dropbear/dropbear_dss_host_key
cp /etc/dropbear/dropbear_rsa_host_key /etc/initramfs-tools/etc/dropbear/dropbear_rsa_host_key
#add your public keys to /etc/initramfs-tools/root/.ssh/authorized_keys
Add the file /etc/initramfs-tools/hooks/unlock with the following contents:
#!/bin/sh -e
PREREQS=""
prereqs() { echo "$PREREQS"; }
case "$1" in
prereqs)
prereqs
exit 0
;;
esac
## hook-functions provides copy_exec()
. /usr/share/initramfs-tools/hook-functions
echo "/sbin/cryptsetup luksOpen /dev/md/1 cryptroot">$DESTDIR/bin/unlock
echo 'kill $(pidof cryptsetup)'>>$DESTDIR/bin/unlock
echo 'kill $(pidof plymouth)'>>$DESTDIR/bin/unlock
chmod a+x $DESTDIR/bin/unlock
chmod a+x /etc/initramfs-tools/hooks/unlock
update-initramfs -u -k all
You can now log into your initramfs during boot via shh with the root user and unlock the disks by calling:
unlock
exit
Final remarks
You should probably backup your LUKs headers and store them in a safe place:
cryptsetup luksHeaderBackup /dev/md/1 --header-backup-file md1.luks
cryptsetup luksHeaderBackup /dev/sda4 --header-backup-file sda4.luks
cryptsetup luksHeaderBackup /dev/sdb4 --header-backup-file sdb4.luks
If you have sugustions on how to improve this setup, feel free to post them in the comments.
Sources
http://www.larsko.org/ZfsUbuntu
http://blog.neutrino.es/2011/unlocking-a-luks-encrypted-root-partition-remotely-via-ssh/
http://wiki.ubuntuusers.de/LUKS
This guide installs a ZFS mirror on LUKs encrypted containers. This leads to a performance hit, since writing requires each block of data to be encrypted twice. Addituionally the HP...