NixOPS Container

About

This post is an export of the original slides plus the speaker notes of a talk about NixOPS and its container backend. It's mainly there to allow attendees of the talk to get the pointers as convenient links.

Missing parts: We did have a few good discussions afterwards which I did not capture. In case you were there and do remember them, add a comment please or drop me a message and I'll add a section at the end.

It's raw, no details added. :-)

NixOPS

Deploy a network of machines

Note

  • Introduction
  • Nix as a base
    • Most operations carried out by Nix itself
  • NixOS as a base
    • Mainly the module system is in use here
    • A tiny bit of the NixOS container support
      • would love to change this
  • NixOPS a set of machines
    • network of machines
    • manages state
      • generated some Nix expressions based on state
    • delegates the heavy work to Nix

Containers

NixOS containers

Note

  • Focus on NixOS containers
  • Based on systemd-nspawn in general
  • declarative vs. imperative
    • declarative supports many more options
    • see the manual
  • NixOPS throws imperative containers

Use NixOPS to deploy containers

How do we get a container deployed?

Note

  • Experience so far?

Goal

  • Existing application
  • Custom NixOS module to run it as a service
  • Simple network with just this container
  • Deploy to an existing NixOS machine

Note

  • Using tryton
    • http://www.tryton.org/
    • Planning to use it to organize my business
    • Invoices, Projects, Time tracking, etc.
    • Having a ton of tweaked repos locally
    • All hooked together with nix
    • For now only the result is interesting
  • How do we get this app into a container?
  • How do we deploy this into a NixOS host?

Simple network

 1 {
 2   network.description = "Tryton Daemon";
 3   trytond = { config, pkgs, ... }: {
 4     imports = [ ./modules/trytond.nix ];
 5     deployment.owners = [ "joh@bo-tech.de" ];
 6     environment.systemPackages = with pkgs; [ tcpdump lsof ];
 7 
 8     services.trytond = {
 9       enable = true;
10       openFirewall = true;
11       pkg = let
12         tryton-nixpkgs = import <tryton-nixpkgs> {
13           inherit (pkgs) system;
14         };
15         trytond = import ./../default.nix {
16           pkgs = tryton-nixpkgs;
17         };
18       in trytond;
19     };
20   };
21 }

Note

  • module for the application
  • import the build
    • bit tricky here, since I did use a different nixpkgs source to build the app
    • look at line 12, tryton-nixpkgs
    • want to use a different checkout of nixpkgs

Container specific configuration

1 {
2   trytond = { config, pkgs, ... }: {
3     deployment.targetEnv = "container";
4     deployment.container.host = "nixvm1";
5   };
6 }

Note

  • container part is already adjusted
    • that's how I would like it to work
    • experimental code is there
    • more later
  • Current state would assign IP address etc. automatically.

Create a deployment

1 $ nixops create \
2     ops/trytond.nix ops/trytond-container.nix \
3     -I nixpkgs=/Users/johannes/wo/nixpkgs-17.03 \
4     -I tryton-nixpkgs=/Users/johannes/wo/nixpkgs

Inspect the state

 5 $ nixops list
 6 +--------------------------------------+--------+---------------+------------+-----------+
 7 | UUID                                 | Name   | Description   | # Machines |    Type   |
 8 +--------------------------------------+--------+---------------+------------+-----------+
 9 | dec26407-4221-11e7-91e8-3c15c2d6d1d4 | (none) | Tryton Daemon |          1 | container |
10 +--------------------------------------+--------+---------------+------------+-----------+

Note

  • lines 1 and 2 are pretty much regular
  • line 3 selects a local clone to use as nixpkgs
  • line 4 uses a different set for the application
  • it does not have to be this wild, but I did end up at this point

Build and deploy

 1 $ nixops deploy
 2 trytond> building initial configuration...
 3 these derivations will be built:
 4   /nix/store/advgiccgk8w638kc2jmvv9ahxn2vg0d0-root-authorized_keys.drv
 5 # [...]
 6 trytond> creating container...
 7 # [...]
 8 trytond> copying closure...
 9 # [...]
10 trytond> activation finished successfully

Note

  • in my case
    • macOS and remote building
    • was not without trouble
    • did not try in daemon mode

NixOS container internals

Let's poke around and see what we find.

Note

  • Took me a few nights to get it all together
  • Got a serious cold afterwards :(
  • BUT
    • Now things do fit together in my mind
    • Learning curve was really hard for me
  • Might be due to the way how I did approach it :D

Context

Container trytond

Configuration file

/etc/containers/trytond.conf

1 RIVATE_NETWORK=1
2 HOST_BRIDGE=br0
3 HOST_ADDRESS=10.233.0.1
4 LOCAL_ADDRESS=10.233.0.2
5 INTERFACES=""
6 MACVLANS=""
7 EXTRA_NSPAWN_FLAGS=""

Who is creating this?

Note

  • start script
  • uses variables from /etc/containers
  • used as arguments to systemd-nspawn
  • look at the host system
    • ps aux | grep nspawn

nixos-container

8 nixos-container create trytond

pkgs/tools/virtualization/nixos-container/nixos-container.pl

Note

  • Look at the create part

systemd-nspawn

ps aux | grep nspawn

 1 root 24570 0.0 0.0 44912 2540 ? Ss 15:54 0:00
 2 /nix/store/4mhj2swvnacffi9zyj54gd5ig343jwga-systemd-232/bin/systemd-nspawn
 3 --keep-unit -M trytond -D /var/lib/containers/trytond --network-veth
 4 --network-bridge=br0 --notify-ready=yes --bind-ro=/nix/store
 5 --bind-ro=/nix/var/nix/db --bind-ro=/nix/var/nix/daemon-socket
 6 --bind=/nix/var/nix/profiles/per-container/trytond:/nix/var/nix/profiles
 7 --bind=/nix/var/nix/gcroots/per-container/trytond:/nix/var/nix/gcroots --setenv
 8 PRIVATE_NETWORK=1 --setenv HOST_BRIDGE=br0 --setenv HOST_ADDRESS=10.233.0.1
 9 --setenv LOCAL_ADDRESS=10.233.0.2 --setenv HOST_ADDRESS6= --setenv
10 LOCAL_ADDRESS6= --setenv HOST_PORT= --setenv
11 PATH=/nix/store/gxx67al6di155xb52zmnhnzbixzy45sg-iproute2-4.9.0/bin:/nix/store/5lfwsx1nkrcqp7p24qr8z4wiwfxx5idv-coreutils-8.26/bin:/nix/store/j7dsnnmws7fc2b1xzcvc5zys5bp3d1yg-findutils-4.6.0/bin:/nix/store/j4qc2qdmhmjxxgzfbhi4whznxr5sg68l-gnugrep-3.0/bin:/nix/store/2j490a1f6c0v6ycm1zfbmw507m4zkfbm-gnused-4.4/bin:/nix/store/4mhj2swvnacffi9zyj54gd5ig343jwga-systemd-232/bin:/nix/store/gxx67al6di155xb52zmnhnzbixzy45sg-iproute2-4.9.0/sbin:/nix/store/5lfwsx1nkrcqp7p24qr8z4wiwfxx5idv-coreutils-8.26/sbin:/nix/store/j7dsnnmws7fc2b1xzcvc5zys5bp3d1yg-findutils-4.6.0/sbin:/nix/store/j4qc2qdmhmjxxgzfbhi4whznxr5sg68l-gnugrep-3.0/sbin:/nix/store/2j490a1f6c0v6ycm1zfbmw507m4zkfbm-gnused-4.4/sbin:/nix/store/4mhj2swvnacffi9zyj54gd5ig343jwga-systemd-232/sbin
12 /nix/store/c85s4dj6rq174wkdx4zk35id46xmn2v1-container-init
13 /nix/var/nix/profiles/system/init

Note

  • Ok, where does this come from?
  • Found it in nixpkgs in the container module for NixOS.

Details

nixos/modules/virtualisation/containers.nix

Note

  • the module container does have an init script
  • it can be influenced by the container configuration

NixOPS internals

Theory of operation

  • allocate resources
  • build all machines
  • copy closures into targets

Note

  • main work is done by Nix
  • storing the state of the deployment
    • that's the core of what NixOPS actually does

Remote building

macOS - Linux headache

Note

  • possible
  • did have to tweak things slightly
  • basically nix remote building has to work
  • common struggle macOS users it seems

Machines

1 /nix/store/xs8dk440ml79wbp0cvjwa8mcwsl3hb3m-nixops-machines

Note

  • inspect what it did build and where I did put the container configuration

Improve container backend

Goal

Support many of the options which declarative containers do support currently.

Approach

Unify declarative and imperative containers.

Sneak in container options

 1 {
 2   trytond = { config, pkgs, ... }: {
 3     deployment.targetEnv = "container";
 4 
 5     deployment.container.host = "nixvm1";
 6 
 7     # This is already tweaked, does not work out of the box.
 8     container = {
 9       writeContainerConfig = true;
10       privateNetwork = true;
11       localAddress = "10.233.0.2";
12       hostAddress = "10.233.0.1";
13       hostBridge = "br0";
14     };
15   };
16 }

Note

  • This is what I would like to write
  • And in fact I got it working

How is it done

Show this in real code, slides are not made for this.

https://github.com/NixOS/nixops/compare/master...johbo:work-johbo?expand=1

Note

  • Idea
    • Allow to use the configuration of declarative containers
    • Put the result into the machine
    • Backend specific details
    • This way imperative and declarative could be done in a similar way

Desired configuration into machine

 1 [nix-shell:~/wbt/tryton/nix]$ tree /nix/store/xs8dk440ml79wbp0cvjwa8mcwsl3hb3m-nixops-machines/trytond/
 2 /nix/store/xs8dk440ml79wbp0cvjwa8mcwsl3hb3m-nixops-machines/trytond/
 3 ├── activate
 4 ├── bin
 5 │   └── switch-to-configuration
 6 ├── configuration-name
 7 ├── container
 8 │   └── container.conf -> /nix/store/kg52sg5slmkv9iyn79x8z6jcjp4wsnpv-container-conf
 9 ├── etc -> /nix/store/j3a9pqsx6a0s99zj92h39zg88ya9lir6-etc/etc
10 ├── extra-dependencies
11 ├── fine-tune
12 ├── init
13 ├── init-interface-version
14 ├── nixos-version
15 ├── sw -> /nix/store/vyipk22wszc0q8m72i4vhmv1rvpb32mk-system-path
16 ├── system
17 └── systemd -> /nix/store/5swjdn16ivaqp8x35df8w6n1h5miyv0a-systemd-232
18 
19 6 directories, 9 files

Status

Idea

Early proof of concept attributes

Pointers

The Manuals

For starters:

Also needed:

Note

  • NixOPS has a manual
    • not yet very sophisticated
    • code reading needed

The Code

Note

  • Python code took me a while to digest
  • Able to work with it once I got the overview

The Community

NixOPS anyone?

Container backend?

Note

  • NixOPS does not seem to have such a chatty userbase
    • guess people are busy
  • Thinner ice than with NixOS, Nixpkgs and Nix itself

See you at NixCON

http://nixcon2017.org/

;-)

or at the Internet

https://www.johbo.com