My focus

I was mainly involved around tooling for Python based development with Nix. My goal was to improve pip2nix a little bit and to explore the future direction of it, also thinking about how it compares to pypi2nix.

pip2nix is a utility to freeze a set of Python dependencies into a Nix expression. It aims to automate a good chunk of the work to use Python dependencies with the Nix package manager.

Comparing pypi2nix and pip2nix

Both tools try to achieve a very similar result: A set of Python dependencies shall be frozen inside of a Nix expression, so that it can be used during development and deployment to increase reliability and reproducability.

pypi2nix

The idea is to avoid getting too deep into the internals of pip, which is used to install Python dependencies.

To generate a Nix expression for a given list of requirements, it will try to install all requirements and generate the Nix expressions based on the result of this installation.

The installation will fail, when there are dependencies which rely on C libraries or need special options to build. The process is then to add the needed dependencies and repeat the process until it passes. Once this passes, the dependencies and needed overrides are complete.

pip2nix

Here the approach is basically reversed: The idea is to use pip's internal APIs to retrieve the needed information. All dependencies will be fetched and unpacked, so that the dependencies can be detected based on the included file setup.py. After unpacking, no installation will happen.

The result is generated into a Nix expression.

Afterwards, needed tweaks have to be applied into an override file, so that external C based dependencies can be added.

The process finishes after the needed overrides have been added. At this moment the project can be built.

Comparison

The key difference seems to be in the approach:

  • pypi2nix installs the packages and examines them based on that. It ensures already during this phase that all needed system dependencies are added. The user may have to run a few iterations until the generating the Nix expression is successful, during each iteration missing system dependencies will be added.
  • pip2nix only discovers the dependencies, without having the needed system dependencies available during this phase. Usually only one run is needed to generate the Nix expression. The needed system dependencies are added into an empty template afterwards.

Both tools try to achieve the same result:

  • Get 80% there based on automating the repetitive tasks.
  • Allow to easily apply the needed manual tweaks into a separate file. This makes future updates of the generated file easier.

Conclusion

At this stage I don't see a big issue in having multiple tools. It is not yet fully clear to me which way works best, or if even having both ways makes sens. The best way to find this out is by exploration: Keep both ways, improve, learn and at some point it will become obvious which way to choose.

Most probably it is even possible to keep the results of the tools so similar, that it is easy to switch between them.

Outlook

After using pip2nix for quite a few projects during the last few years, I observed that the needed adjustments in the overrides layer are very repetitive. Compare also my recent post about Python development based on Nix.

Example overrides

The following file shows a typical example of needed overrides.

 1 # Generated by pip2nix 0.5.0.dev1
 2 # Adjust to your needs, e.g. to provide C libraries.
 3 
 4 { pkgs, basePythonPackages }:
 5 
 6 self: super: {
 7 
 8   cffi = super.cffi.override (attrs: {
 9     propagatedBuildInputs = attrs.propagatedBuildInputs ++ (with self; [
10       pkgs.libffi
11     ]);
12   });
13 
14   cryptography = super.cryptography.override (attrs: {
15     buildInputs = attrs.buildInputs ++ [
16       pkgs.openssl
17     ] ++
18     pkgs.lib.optional pkgs.stdenv.isDarwin pkgs.darwin.apple_sdk.frameworks.Security;
19     propagatedBuildInputs = attrs.propagatedBuildInputs ++ (with self; [
20       # Building with Python 2
21       enum34
22       ipaddress
23     ]);
24 
25   });
26 
27   Lektor = super.Lektor.override (attrs: {
28     # TODO: johbo: Workaround issues in watchdog
29     makeWrapperArgs = pkgs.stdenv.lib.optionals pkgs.stdenv.isDarwin [
30       "--set DYLD_FRAMEWORK_PATH /System/Library/Frameworks"
31     ];
32   });
33 
34   PyYAML = super.PyYAML.override (attrs: {
35     buildInputs = with self; [ pkgs.pyrex ];
36     propagatedBuildInputs = with self; [ pkgs.libyaml ];
37   });
38 
39   watchdog = super.watchdog.override (attrs: {
40     preShellHook = ''
41       # TODO: johbo: The develop install fails without running this first.
42       eval "$configurePhase"
43     '';
44     buildInputs = attrs.buildInputs ++ (
45       pkgs.stdenv.lib.optionals pkgs.stdenv.isDarwin [
46          pkgs.darwin.apple_sdk.frameworks.CoreServices
47          pkgs.darwin.cf-private
48       ]);
49   });
50 
51   # Keep these from nixpkgs unchanged.
52   inherit (basePythonPackages)
53     pip
54     setuptools;
55 
56 }

Next step

Based on this experience I think it would be beneficial to have a central repository which provides needed overrides for most Python packages. Based on these overrides, the process might already work fully automatic for many cases.

My hope is that this would help us to save some time when using Nix for Python based software. With a bit of luck this should also making it easier for newcomers to start out with their Python based projects and Nix.

Interim advise

Based on my own experience I know that after some time you will get used to Nix, and finding out the needed tweaks works most of the time already based on intuition. But in the first weeks or months, this can be a quite frustrating exercise.

My current recommendation is to look into the file python-packages.nix inside of the Nixpkgs repository, there is a lot of inspiration for needed tweaks.

That's it!

Happy that I got a few teaks for pip2nix pushed out, have a look: https://github.com/johbo/pip2nix/blob/v0.5.0/CHANGELOG.rst#050

Hope to see you next time: https://www.meetup.com/Berlin-NixOS-Meetup/


Comments

comments powered by Disqus