Reproducible Development Environments with Nix

About

It's a raw export of the slides from a talk at PyConWEB about my approach of using Nix to get reproducible development environments. It is based on my experience from the last few years of using this approach in various companies / projects. It's origins go back to the point of being introduced to Nix by Rok Garbas in 2014.

The main intend is to help attendees if they want to follow pointers just by clicking on them instead of typing the URLs themselves into Google ;)

Intro

Hello!

Note

  • Johannes Bornhold
  • 2011 - Python project, geo dependencies
  • 2014 - RhodeCode, challenging dependencies, Nix
  • 2016 - Focus on my company, more Nix

Dependencies

What is in the box?

Note

  • Q: Needs for typical Python based web project?
  • Q: Web specific?
  • thoughts
    • Python + X
    • mainly JS: npm, bower, grunt, webpack
    • System dependencies

Perspective

Developer's perspective

Note

  • Perspectives to look at dependencies
    • Ops
    • Legal
    • Dev
  • Multiple projects, easy switching
  • Multiple people, aka TEAM

Nix

  • Build tool
  • Package manager
  • Symlinks on steroids

Note

  • build tool
    • describe how to build an application
    • and in fact build it
    • store the binary result
  • package manager
    • describe a set of application
    • and build them
    • install the binary build result
  • symlinks
    • stores build results in a store
    • isolated per build
    • symlinks pointing to dependencies
    • symlinks pointing to build results

Building

 1 $ nix-build release.nix -A pip2nix.python35
 2 these derivations will be built:
 3   /nix/store/2hfsb0ixayq5y7nimziqz09zm1jj07nn-python3.5-MarkupSafe-0.23.drv
 4   /nix/store/4i5vhgnmdwf5pnjd2dza4ww7kiz4m600-py-1.4.32.tar.gz.drv
 5   /nix/store/c6skp7sf7c8jdcg63qryhl9hsack473k-python3.5-jinja2-2.8.drv
 6   /nix/store/f81liyzxq6ca1rb8vsnx7fcqzpl3c2lw-python3.5-contexter-0.1.3.drv
 7 # [ ... ]
 8 these paths will be fetched (58.10 MiB download, 259.00 MiB unpacked):
 9   /nix/store/04ccw2f40c0bvlhnhflprn19ia2fv44g-python2.7-ipaddress-1.0.16
10   /nix/store/0f4lw8jasc2sqp86k0czx1llpr5sbr8l-python2.7-pyasn1-0.1.9
11 # [ ... ]

Note

  • release.nix describes how to build the application
  • based on this description Nix knows what to do
    • compare setup.py for pip
    • package.json for npm
    • you name it
  • some dependencies will be
    • built
    • others fetched
    • benefit: binary substitution
  • Q: What's /nix/store/ ?

Store

Build result points to the store

1 $ cd ~/my-project
2 $ nix-build
3 # [ ... ]
4 $ ls -l result
5 result -> /nix/store/zic5aq2sx2fv19l9xyv265c2w4cacr8i\
6     -python3.5-pip2nix-0.7.0.dev1

Easy to run via symlink

 7 $ ./result/bin/pip2nix --help
 8 Usage: pip2nix [OPTIONS] COMMAND [ARGS]...
 9 # [ ... ]

Note

  • place to put the build results
  • hash based on all build inputs
    • two different configurations of the same package result in different hashes
  • works with garbage collection
    • important to be aware of it ;)
    • eventually your disk will fill up
  • Q: How do we get a development environment?

Shell

  • Fetch or build dependencies
  • Prepare a shell environment
1 $ nix-shell release.nix -A pip2nix.python35
2 [nix-shell:~/wo/pip2nix]$ pip2nix --help
3 Usage: pip2nix [OPTIONS] COMMAND [ARGS]...
4 # [ ... ]

Suitable for development

5 [nix-shell:~/wo/pip2nix]$ python -c 'import pip2nix; print(pip2nix)'
6 <module 'pip2nix' from '/Users/johannes/wo/pip2nix/pip2nix/__init__.py'>

Note

  • fetching and building dependencies
    • same is nix-build
  • nix-shell provides
    • environment
    • all dependencies available
    • shell
    • hooks - can be handy
  • can be used to develop

Nix and Python

Let's look at some code

👀

Note

  • Examples
  • Starting simple
  • Adding more and more trouble

Simple package with dependencies

 1 { pkgs ? (import <nixpkgs> {}) }:
 2 
 3 let
 4   pythonPackages = pkgs.python36Packages;
 5 
 6 in pythonPackages.buildPythonPackage {
 7   name = "sample-1.2.0";
 8   src = ./.;
 9 
10   propagatedBuildInputs = [
11     pythonPackages.lxml
12     pythonPackages.peppercorn
13   ];
14 }

Note

  • lxml itself does in fact bring libxml2 and libxslt into the mix
  • inside nixpkgs
    • the definition of lxml
    • have a look
  • reality check
    • you want tooling support in bigger projects

Nix as development environment

Note

  • Q: How can we use it?
  • Remember the shell?
  • Then other aspects

The shell

Environment with all buildtime dependencies available.

Tweak further via hooks.

Note

  • Developer's workhorse
  • Default behavior is already a good working environment
  • Tweak hooks to adjust to your project's needs
  • Experience so far
    • Avoid too much magic
    • Keep it hackable
    • Don't assume you know what your colleagues need and how they should work
      • instead provide sane defaults
      • and make it easy to override them

Default workflow

1 $ nix-shell
2 [nix-shell:~/wo/my-project]$ # do stuff!

Advanced level

3 $ nix-shell -A i18n-tools
4 $ nix-shell i18n-tools.nix

Note

  • The contract to enter an environment is minimalistic
  • Run nix-shell
  • Or if you have multiple environments
    • either select one attribute
    • or a different Nix file
  • Experience so far
    • Using Nix also for build and deployment
      • Minimal default.nix describing your package
      • A release.nix with build and release-specific things
      • A shell.nix with dev-env specific things
      • Worked out quite ok so far

Automation

pip2nix / pypi2nix

My default layout

1 .
2 ├── default.nix
3 ├── python-packages-overrides.nix
4 ├── python-packages.nix
5 └── setup.py

Note

  • developer's perspective
  • not about automating the deployment
    • do this as well, use nix ;)
  • freezing dependencies into a nix file
  • 80/20 approach
  • layers

Conclusions

Note

  • Based on using Nix during the last years
  • Mainly Python based web development
  • Target platform was Linux
  • Development on a mix of Linux / macOS
  • 1st conclusion: Not only Python

1+N nixperts

Note

  • Build up one expert in your team
  • Or have someone in reach available

pin nixpkgs by default

Note

  • nixpkgs comes via a channel
    • nixos-stable and unstable
    • each team member might be on a different commit
    • causes friction
  • whole team on same set by default
  • experts can still hack their way
  • buys you time if breaking changes are on their way
  • update on a regular base

start small

Note

  • don't make it too great in the first step
    • it would just blow up
  • give time to build experience
  • then incrementally put Nix everywhere ;)

CI and deployment

Note

  • Talk looks only at development environment
  • Nix makes a fantastic foundation for binary deployments
    • Rollbacks nearly for free
  • CI -> cache
  • CI can build your project also against latest nixpkgs
    • this way you will know if breaking changes are coming
  • Challenge
    • Keeping the balance between a good development environment and good support for the ops part.

Automate where possible

Note

  • focus here: dependency management
  • freezing Python requirements into a Nix expression
    • Tooling: pip2nix, pypi2nix
    • 80% approach
  • think in layers
    • get a base automated
    • put your tweaks on top of it
    • easy re-generation
  • similar for other languages' ecosystems

Pointers

The Manuals

For starters:

Next steps:

Note

  • The language section of the Nix manual is worth reading.
    • MUST READ, pays of fast
  • The Python section of the Nixpkgs manual contains a very good overview of various approaches and tooling support.
  • Treat Nixpkgs manual as a cookbook / reference
    • Poke around frequently
    • Lots to offer

The Tools

Note

  • Tooling available
  • Different approaches
    • pip2nix abuses pip internals
    • pypi2nix treats pip as a black box
  • Thinking of merging the tools
  • Experience so far
    • 80/20 approach works well
    • 80% automated is easy to achieve
    • 20% tweaking is sustainable
    • feels mode like 90/10 actually :)

The Code

Note

  • Examples
  • Utilities
  • Inspiration
  • Lots of wheels in there
    • no need to invent new ones

The Community

Notes added which were not part of the talk

Stickers

I am really sorry that I did not have a single sticker with me to distribute.