Modifying Python packages via config.nix

The package collection nixpkgs has a nice way to configure and tweak the package set based on the configuration file ~/.nixpkgs/config.nix.

Changing something in pythonPackages was not so obvious to me though. In my case I used dulwich on Darwin. It was pulled in as a dependency of Mercurial. One of it's Unicode related tests was failing, since there is a difference in the handling of file names between Linux and Mac OS X which leads to a failing test around reference names.

So before working on a fix for nixpkgs, I wanted to have a quick workaround for my case.

The failing test of dulwich

The starting point of this journey was the following failing test:

======================================================================
FAIL: test_non_ascii (dulwich.tests.test_refs.DiskRefsContainerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/private/var/folders/v2/kx2sg5693tb1h84zc2hmjjgr0000gn/T/nix-build-python2.7-dulwich-0.12.0.drv-0/dulwich-0.12.0/dulwich/tests/test_refs.py", line 453, in test_non_ascii
    self.assertEqual(expected_refs, self._repo.get_refs())
AssertionError: {'refs/heads/packed': '42d06bd4b77fed026b154d16493e5deab78f02ec', 'refs/tags/sch [truncated]... != {'HEAD': '42d06bd4b77fed026b154d16493e5deab78f02ec', 'refs/tags/scho\xcc\x88n':  [truncated]...
  {'HEAD': '42d06bd4b77fed026b154d16493e5deab78f02ec',
  'refs/heads/40-char-ref-aaaaaaaaaaaaaaaaaa': '42d06bd4b77fed026b154d16493e5deab78f02ec',
  'refs/heads/master': '42d06bd4b77fed026b154d16493e5deab78f02ec',
  'refs/heads/packed': '42d06bd4b77fed026b154d16493e5deab78f02ec',
  'refs/tags/refs-0.1': 'df6800012397fb85c56e7418dd4eb9405dee075c',
  'refs/tags/refs-0.2': '3ec9c43c84ff242e3ef4a9fc5bc111fd780a76a8',
-  'refs/tags/sch\xc3\xb6n': '0000000000000000000000000000000000000000'}
?                   ^  ^^

+  'refs/tags/scho\xcc\x88n': '0000000000000000000000000000000000000000'}
?                +   ^  ^^

It seems that it checks for refs/tags/schön and expects the correct byte sequence on Linux.

Although it is sad, for my case I wanted it to just work, so my aim for the workaround is to just switch off the test run.

Overriding packages

The documentation on overriding a package suggests to define packageOverrides inside of ~/nixpkgs/config.nix in the following way:

{
  packageOverrides = pkgs: rec {
    foo = pkgs.foo.override { ... };
  };
}

Overriding a Python package

The tricky part was to find out how I can override it in a way, so that every package which would use dulwich as a dependency would get my modified version. In the end I got reminded to a NixCon talk about the Haskell packaging infrastructure , where a way to use layers to override each other was presented.

Based on that I came up with the following configuration inside of ~/.nixpkgs/config.nix:

pythonPackageOverrides = {self, super}: {
  dulwich = super.dulwich.override {
    doCheck = false;
  };
};

packageOverrides = pkgs: rec {

  # Apply python package overrides in a way that everything will use the
  # overidden derivations.
  pythonPackages = pkgs.pythonPackages.override (oldAttrs: {
    self = pythonPackages;
    }) // (pythonPackageOverrides {
      self = pythonPackages;
      super = pkgs.pythonPackages;
    });
};

Conclusion

This allowed me to use pkgs.fetchhg again and did the trick as a workaround. I think that this might be a nice ad-hoc way for other tweaks to Python packages as well.

I did notice that it seems to clash with further calls to pkgs.overridePackages so it is better to consider this as an experimental approach.

The failing test of dulwich

Overriding packages

Overriding a Python package

Conclusion

Comments