Setting up ZFS on CoreOS

Recently I needed to expand my disk storage in my server. I previously had 2x2TB of old WD Green disks in a BTRFS RAID1. Now I upgraded to 8x3TB and a dedicated SAS2008-based controller.

As a filesystem I opted for ZFS because I wanted parity-based RAID and Btrfs's implementation is considered broken at the moment. Because my server (just like all servers at DolanSoft) run on CoreOS I needed to compile ZFS for CoreOS.

That is not easily done though, because there is no kernel module compilation environment in CoreOS and because it is an immutable operating system, there is not even the possibility of installing one.

Edit 01/18: I now published torcx-zfs which is a much cleaner and quicker way of installing ZFS on CoreOS

After a lot of digging, I found this mail by a CoreOS dev which linked to a container that is automatically built for every CoreOS version and has the needed tools to compile the ZFS kernel module and userspace tools.

So I pulled the right container for my CoreOS version (stable)

wget http://stable.release.core-os.net/amd64-usr/current/coreos_developer_container.bin.bz2
bunzip2 coreos_developer_container.bin.bz2

and started the resulting container using sudo systemd-nspawn -i coreos_developer_container.bin --share-system.

The mail above also had instructions on how to prepare that container for kernel module development, which I'm repeating here:

emerge-gitclone
emerge -gKav coreos-sources
cd /usr/src/linux
zcat /proc/config.gz >.config
make modules_prepare

After that, we're ready to build SPI and ZFS:

wget -O - https://github.com/zfsonlinux/zfs/releases/download/zfs-0.6.5.7/zfs-0.6.5.7.tar.gz | tar -xzf -
wget -O - https://github.com/zfsonlinux/spl/archive/spl-0.6.5.7.tar.gz | tar -xzf -
cd spl && make && make install
cd zfs && make && make install

The resulting coreos_developer_container.bin can now be used anywhere to install the ZFS kernel module and userspace.

sudo systemd-nspawn -i coreos_developer_container.bin --bind /:/target --capability=CAP_SYS_MODULE --share-system
modprobe zfs
cp /usr/local/sbin/* /target/opt/bin/
cp -r /usr/local/lib64 /target/usr/share/oem/
chown -R 755 /target/usr/share/oem

Now we need to run ldconfig -v to update the library path and restart the session.

We can now finally create a ZFS pool:

sudo zpool create data raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sdf

Have fun with your new ZFS!