Modifying Python packages via config.nix

The package collection nixpkgs has a nice way to configure and tweak the package set based on the configuration file ~/.nixpkgs/config.nix.

Changing something in pythonPackages was not so obvious to me though. In my case I used dulwich on Darwin. It was pulled in as a dependency of Mercurial. One of it's Unicode related tests was failing, since there is a difference in the handling of file names between Linux and Mac OS X which leads to a failing test around reference names.

So before working on a fix for nixpkgs, I wanted to have a quick workaround for my case.

The failing test of dulwich

The starting point of this journey was the following failing test:

======================================================================
FAIL: test_non_ascii (dulwich.tests.test_refs.DiskRefsContainerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/private/var/folders/v2/kx2sg5693tb1h84zc2hmjjgr0000gn/T/nix-build-python2.7-dulwich-0.12.0.drv-0/dulwich-0.12.0/dulwich/tests/test_refs.py", line 453, in test_non_ascii
    self.assertEqual(expected_refs, self._repo.get_refs())
AssertionError: {'refs/heads/packed': '42d06bd4b77fed026b154d16493e5deab78f02ec', 'refs/tags/sch [truncated]... != {'HEAD': '42d06bd4b77fed026b154d16493e5deab78f02ec', 'refs/tags/scho\xcc\x88n':  [truncated]...
  {'HEAD': '42d06bd4b77fed026b154d16493e5deab78f02ec',
  'refs/heads/40-char-ref-aaaaaaaaaaaaaaaaaa': '42d06bd4b77fed026b154d16493e5deab78f02ec',
  'refs/heads/master': '42d06bd4b77fed026b154d16493e5deab78f02ec',
  'refs/heads/packed': '42d06bd4b77fed026b154d16493e5deab78f02ec',
  'refs/tags/refs-0.1': 'df6800012397fb85c56e7418dd4eb9405dee075c',
  'refs/tags/refs-0.2': '3ec9c43c84ff242e3ef4a9fc5bc111fd780a76a8',
-  'refs/tags/sch\xc3\xb6n': '0000000000000000000000000000000000000000'}
?                   ^  ^^

+  'refs/tags/scho\xcc\x88n': '0000000000000000000000000000000000000000'}
?                +   ^  ^^

It seems that it checks for refs/tags/schön and expects the correct byte sequence on Linux.

Although it is sad, for my case I wanted it to just work, so my aim for the workaround is to just switch off the test run.

Overriding packages

The documentation on overriding a package suggests to define packageOverrides inside of ~/nixpkgs/config.nix in the following way:

{
  packageOverrides = pkgs: rec {
    foo = pkgs.foo.override { ... };
  };
}

Overriding a Python package

The tricky part was to find out how I can override it in a way, so that every package which would use dulwich as a dependency would get my modified version. In the end I got reminded to a NixCon talk about the Haskell packaging infrastructure , where a way to use layers to override each other was presented.

Based on that I came up with the following configuration inside of ~/.nixpkgs/config.nix:

pythonPackageOverrides = {self, super}: {
  dulwich = super.dulwich.override {
    doCheck = false;
  };
};

packageOverrides = pkgs: rec {

  # Apply python package overrides in a way that everything will use the
  # overidden derivations.
  pythonPackages = pkgs.pythonPackages.override (oldAttrs: {
    self = pythonPackages;
    }) // (pythonPackageOverrides {
      self = pythonPackages;
      super = pkgs.pythonPackages;
    });
};

Conclusion

This allowed me to use pkgs.fetchhg again and did the trick as a workaround. I think that this might be a nice ad-hoc way for other tweaks to Python packages as well.

I did notice that it seems to clash with further calls to pkgs.overridePackages so it is better to consider this as an experimental approach.