Building a NixOS image for VirtualBox

About

NixOPS allows to deploy a network of machine configurations into various backends. One of the supported backends is Virtualbox. It can be used to conveniently test out a network configuration locally with ease.

Currently NixOPS uses a quite dated image as a base to create the virtualbox machines. This results in a delay of a few minutes to transfer a big chunk of derivations from the local store into the virtual machine.

This post come together based on my research how to build an up to date base image.

Overview of the bootstrap process

NixOPS can create new machines as needed if the backend supports this. For the backend based on Virtualbox this works by creating a new virtual machine. The base disk of the machine is automatically created as a clone of a pre-defined base image.

The base image is first downloaded into the local nix store. This step is skipped if the image resides already in the store.

Then the new disk is created and attached to a new virtual machine.

There is a special service inside of the machine image which allows to inject a ssh key, so that the automatic login via ssh is possible for the later deployment steps.

The regular deployment starts once the machine is up and reachable via ssh.

Definition of the base image

The base image is defined as a regular NixOS machine configuration. The sources can be found inside of the NixOPS repository at the following URL:

https://github.com/NixOS/nixops/blob/master/nix/virtualbox-image-nixops.nix

Building an OVA

My first attempt was with the following command:

nix-build \
    -A config.system.build.virtualBoxOVA \
    -I nixos-config=/Users/johannes/wo/nixops/nix/virtualbox-image-nixops.nix \
    --argstr system x86_64-linux

It turned out that this is a bit too much, since it does build the whole appliance and not just the disk. Still, a few bits are worth explaining:

-A config.system.build.virtualBoxOVA

This is a NixOS attribute which resolved to a derivation, that's why it can be built.

The definition can be found at the following URL: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/virtualbox-image.nix

-I nixos-config=...

The NixOS configuration to apply is expected to be on the import path as nixos-config.

--argstr system x86_64-linux

This is only needed if the current system is of a different type.

Since I did run the experiments from my macOS machine, I had to specify it. Note that also remote building has to be set up correctly so that this works. You'll need a remote linux machine with the flag kvm set.

The resulting appliance did work, running it in Virtualbox worked without any trouble. The next step would be to find out how to just build the disk image instead of the full appliance.

Implementation to build a disk image

The next step is to find out how to just build the disk image. The goal is to have a VMDK file as the build result.

NixOPS' maintainer scripts

NixOPS contains a shell script which can be used as a guide:

#! /bin/sh -e
export NIXOS_CONFIG=$(readlink -f $(dirname $0)/..)/nix/virtualbox-image-nixops.nix
version=$(nix-instantiate --eval-only '<nixpkgs/nixos>' -A config.system.nixosVersion | sed s/'"'//g)
echo "version = $version"
nix-build '<nixpkgs/nixos>' -A config.system.build.virtualBoxOVA --keep-going --fallback
mkdir ova && tar -xf ./result/*.ova -C ova && mv ova/{nixos*,nixos}.vmdk
name=virtualbox-nixops-$version.vmdk.xz
xz < ./ova/nixos.vmdk > $name
rm -fr ova
aws s3 cp $name s3://nix-releases/nixos/virtualbox-nixops-images/$name
sha256sum $name

Approach

The script tells us a few things:

Starting with the OVA was not too bad, we can keep it as an interim result.
Another derivation is needed, to extract the VMDK from the OVA and place it into the store. The details are already contained inside of the maintainer script.
It should be possible to hook this up with Hydra, so that the latest images are automatically prepared.

Add OVA to relese.nix

Building the appliance is more or less straight forward:

First the machine-configuration is needed. It defined what will go into the appliance.
Based on this a machine can be created by passing in the configuration and also the target system.
Finally the appliance can be found in the attribute config.system.build.virtualBoxOVA of the machine. This is the same as in the original experiment above.

The code looks as follows:

{ nixos ? <nixpkgs/nixos>
, system ? builtins.currentSystem
}:

let
  machine-configuration = import ./nix/virtualbox-image-nixops.nix;

  machine = import nixos {
    inherit system;
    configuration = machine-configuration;
  };

in rec {
  ova = machine.config.system.build.virtualBoxOVA;
}

Create VMDK from the OVA

Based on the script maintainers/build-virtualbox-image.sh the vmdk can be created with the following updated Nix file:

{ nixos ? <nixpkgs/nixos>
, system ? builtins.currentSystem
}:

let
  machine-configuration = import ./nix/virtualbox-image-nixops.nix;

  machine = import nixos {
    inherit system;
    configuration = machine-configuration;
  };

  # TODO: Not yet sure if using the local packages is the best approach
  pkgs = import <nixpkgs> { };

in rec {
  ova = machine.config.system.build.virtualBoxOVA;

  nixos-disk = pkgs.stdenv.mkDerivation rec {
    name = "virtualbox-nixops-image-${version}";
    version = machine.config.system.nixosVersion;
    phases = [ "installPhase" ];
    nativeBuildInputs = [
      ova
    ];
    installPhase = ''
      mkdir ova
      tar -xf ${ova}/*.ova -C ova
      mv ova/{nixos*,nixos}.vmdk

      mkdir -p $out
      name=$out/virtualbox-nixops-${version}.vmdk.xz
      xz < ./ova/nixos.vmdk > $name
      sha256sum $name > $name.sha256
    '';
  };
}

Hooking this up with Hydra

It should be possible to build the jobset from the previous section with Hydra. To make it work the remote builder should have at least one machine tagged with kvm. Otherwise the job will pile up in the queue.

Conclusion

I did hook up a local test with NixOPS to deploy a Virtualbox machine based on the new image. The following snipped might give the needed inspiration to hook things up:

{
  # Using a more recent base image speeds things up quite drastically.
  #
  # The symlink ./vbox-image has to be created and it has to point to the
  # base image to use.

  defaults = {
    # TODO: Manually created based on work-johbo on code.bo-tech.de of nixops
    # nix-build -I nixpkgs=/Users/johannes/wo/nixpkgs-17.09 \
    #     release-vbox-image.nix --argstr system x86_64-linux -A baseImage
    deployment.virtualbox.disks.disk1.baseImage = ./vbox-image;
  };

}

Having it as a layer on its own allows to include it easily into a deployment, e.g. by using nixops modify.

This did speed up the creation of new machines significantly compared to the 16.09 based image which is used by default.