Reproducible development environments based on Nix for Python

A common challenge in a typical software project is that the dependencies of the application and the development environment change over time, and providing an environment to develop the application is a non-trivial task.

I am currently solving this challenge based on the Nix package manager and one of its tools called nix-shell. The following sections provide an overview of the approach which I found so far for development of Python projects.

Goal: Enter the environment with one command

Looking at the problem from the user's perspective, this means in this particular case from the developer's perspective: The day to day mission is usually stimulating enough, so that managing the development environment should be simple for the usual cases.

Ideally, there should be one command to start or enter an environment and after that I should be able to start my work. In terms of a nix-shell based setup, it should look like this:

  1. Enter the environment:

    nix-shell

  2. Start the application, get busy:

    pserve development.ini --reload

Approach: Generate the package set semi-automatic

After some initial experiments I settled on the tool pip2nix for the moment. At the time when I did my initial research, several other tools existed, which also provided working results. My assumption is that this principle can also be implemented based on those tools.

The goal in using pip2nix is to have 80% or more of the nix files generated based on existing infrastructure which we know from tools like pip and pypi. The idea was that this tool should be able to generate a package set of Python dependencies based on the information from sources like setup.py or requirements.txt which are often available in Python based projects already.

For the remaining details I accepted that some manual tweaking is needed, e.g. for dependencies on C libraries. My only requirement was that I wanted to be able to maintain the manual tweaks in a separate file, so that I could generate the other file again without risking to loose handcrafted adjustments.

Out of these thoughts I arrived at the idea to use a layered approach:

  • At the bottom, I am taking the python packages out of nixpkgs.
  • On top of that I apply the generated packages which I got from pip2nix.
  • The resulting package set is then further modified by applying my manual tweaks.

Example result

Based on the sample project I am showing how the approach works.

Overview

I ended up using the following file structure for my projects:

  • default.nix as a definition for my project's derivation. I try to keep special tweaks for the development out of this file, so that I can also use it for as input for other purposes like creating a release build.

  • shell.nix containing everything which is specific to a typical developer's needs. Since nix-shell prefers this file over default.nix, things will "just work out" for the regular day to day usage.

  • pkgs/python-packages.nix is file which is generated by pip2nix. I like to keep also a pip2nix.ini around, so that I can just generate this file by running the following command:

    pip2nix generate

  • pkgs/python-packages-overrides.nix contains my manual tweaks. Most of the time these tweaks are around adding C dependencies.

First run of pip2nix

I started with a local clone of the sample project and generated the first files with the following commands:

pip2nix scaffold --package=sample
pip2nix generate .

Afterwards the files default.nix and python-packages.nix were created. I was already able to enter the environment by running nix-shell and also call the program sample.

The generated version of default.nix did not yet fully meet my needs, so I tweaked it a bit go get it into the shape which is shown in the following section.

Combining the layers in default.nix

The following file is the current state which I use as default.nix in my Python based projects:

 1 { pkgs ? (import <nixpkgs> {}), pythonPackages ? "python35Packages" }:
 2 let
 3   inherit (pkgs.lib) fix extends;
 4   basePythonPackages = with builtins; if isAttrs pythonPackages
 5     then pythonPackages
 6     else getAttr pythonPackages pkgs;
 7 
 8   elem = builtins.elem;
 9   basename = path: with pkgs.lib; last (splitString "/" path);
10   startsWith = prefix: full: let
11     actualPrefix = builtins.substring 0 (builtins.stringLength prefix) full;
12   in actualPrefix == prefix;
13 
14   src-filter = path: type: with pkgs.lib;
15     let
16       ext = last (splitString "." path);
17     in
18       !elem (basename path) [".git" "__pycache__" ".eggs"] &&
19       !elem ext ["egg-info" "pyc"] &&
20       !startsWith "result" path;
21 
22   sample-src = builtins.filterSource src-filter ./.;
23 
24   pythonPackagesGenerated = self: basePythonPackages.override (a: {
25     inherit self;
26   })
27   // (scopedImport {
28     self = self;
29     super = basePythonPackages;
30     inherit pkgs;
31     inherit (pkgs) fetchurl fetchgit;
32   } ./pkgs/python-packages.nix);
33 
34   pythonPackagesOverrides = import ./pkgs/python-packages-overrides.nix {
35     inherit
36       basePythonPackages
37       pkgs;
38   };
39 
40   pythonPackagesLocalOverrides = self: super: {
41     sample = super.sample.override (attrs: {
42       src = sample-src;
43     });
44   };
45 
46   myPythonPackages =
47     (fix
48     (extends pythonPackagesLocalOverrides
49     (extends pythonPackagesOverrides
50              pythonPackagesGenerated)));
51 
52 in myPythonPackages.sample

The following tweaks have been applied compared to the generated variant:

  • Line 1: Change to python35Packages.
  • Line 3: I am using fix and extends from pkgs.lib.
  • Line 24 to 32: This has been turned into a function, so that I can pass in the parameter self. I also moved the file python-packages.nix into a subfolder pkgs, so that the project files are organized a bit better.
  • Line 34 to 38: This is the layer to apply my manual tweaks to the python packages. The import will return a function which takes the parameters self and super, so that it can be used together with fix and extends later in this file.
  • Line 40 to 44: It got the parameters self and super. By default it just tweaks the pointer to the sources of the current packages. In real projects it usually starts to get some project specific tweaks in addition which shall be visible in the file default.nix instead of being buried into pkgs/python-packages-overrides.nix.
  • Line 46 to 50: In these lines the layers are glued together, so that I end up with one package set.

Tweaking a dependency in pkgs/python-packages-overrides.nix

The following file shows how overrides work in principle. In this example I am just activating the test run of peppercorn.

 1 { pkgs, basePythonPackages }:
 2 
 3 self: super: {
 4 
 5   peppercorn = super.peppercorn.override (attrs: {
 6     doCheck = true;
 7   });
 8 
 9 }

Regular updates

Most updates will be straight forward once the initial structure has been set up. In most cases I am able to just run the following command:

pip2nix generate

This assumes that a file pip2nix.ini has been added as well, so that pip2nix will take its options from there.

Conclusion

During the last few months, this approach has worked out quite ok for me. Most of the time I am able to update the Python dependencies by running pip2nix generate.

Challenges come up when packages have special requirements. Currently I am in 90% of the cases able to find the correct tweaks inside of the nixpkgs repository and apply them in my overrides file.

Especially if a team of people is working with a codebase, this approach can help to reduce the friction by differences in the development environments. Most of the time starting a fresh nix-shell will bring in changed dependencies.

I've put the examples up on Github at https://github.com/johbo/sampleproject/tree/blog-pip2nix.

Comments

comments powered by Disqus