Python development with Nix - an update

Recap of the goal and approach

The goal was to run nix-shell and have a ready to start environment available. This should work on Linux as well as on Mac OS. My approach to get there was to use the tool pip2nix to get about 80% of the dependency management automated and find a way of how to easily provide the remaining 20%.

How it went so far

Freezing the Python dependencies via pip2nix usually works out quite well. By setting some configurations in pip2nix.ini I can just run the following command to update the generated files:

pip2nix generate

Especially minor updates work nearly always without any need for further tweaks.

Starting with a fresh project is usually also a straight forward process. If system dependencies are needed or special build tweaks, most of the time a quick research inside of the Nixpkgs repository's file python-packages.nix helps to get the needed inspiration. Once the needed tweaks are set up, they only have to be touched again in rare cases.

Also using a layered approach based on fix and extends from Nixpkgs library proved to be very solid and easy to work with.

How to improve from here

After hitting the goal to have 80% automated, I start to wonder if it would be possible to get closer to full automation of the process to freeze the dependencies into Nix derivations.

After using the approach in many business projects and personal projects, I noticed that most of the needed adjustments are repetitive. Only in rare cases special tweaks were needed.

Currently I am thinking that a source containing common needed tweaks for most Python packages would be very useful.

This could get inspiration from the way how it is done for the Ruby world. Within Nixpkgs a file which contains the needed system dependencies and special tweaks for the most common packages exists and is applied by default to enrich the generated expressions.

Applying this to the layered approach could be as simple as including just another layer be default and providing it from a common source.

Conclusions

Because the layered approached turned out to work so well I have decided to adjust the scaffolding of pip2nix. It is now generating a default.nix which combines the different layers and a file for the overrides as well.

The next improvement will most probably be to provide commonly needed tweaks, so that the process can be nearly fully automated.