Witch of Git - web/blog/blob - posts/2017/package-manager-race-condition.md

   1 ---
   2 title: "You Got Your Race Condition Inside My Package Manager!"
   3 date: 2017-08-14
   4 author: Cassie Jones
   5 tags:
   6 - python
   7 - pypi
   8 - debugging
   9 summary: "We solve a fun, surreal debugging mystery involving pip and version compatibility."
  10 ---
  11
  12 # A Case of Broken Builds
  13
  14 The continuous integration servers at my current job are unfortunately stateful.
  15 Every week or so, we run a bunch of configuration processes to reinstall packages to keep the environment clean.
  16 One of these reinstalls `pip` and the Python libraries used by build tools.
  17 This morning, I got a message from one of the build engineers telling me that the Python libraries weren't installing correctly anymore.
  18 (Even though I'm an intern, I'm apparently one of the office Python experts now.)
  19 So, I opened up the build log, and began looking around.
  20
  21 What was failing was pretty clear:
  22
  23 ```
  24 Collecting ruamel.yaml
  25   Using cached ruamel.yaml-0.15.28.tar.gz
  26 Installing collected packages: ruamel.yaml
  27  ...
  28  error: Microsoft Visual C++ 9.0 is required
  29 ```
  30
  31 but why was this suddenly happening now, without us making any changes to our configuration?
  32 Also, why are we installing ruamel.yaml?
  33 We're not using that!
  34
  35 Long story short, ruamel.yaml was a transitive dependency of dateparser, an excellent library for parsing natural language dates.
  36 It wasn't clear to me why it would be suddenly failing though, so I decided to go investigating further.
  37 Looking at the release notes of dateparser, I saw that they had recently pinned ruamel.yaml to `<0.14`, which we clearly weren't getting.
  38 Previously, the version was un-pinned, so I decided to go look at the release notes for ruamel.yaml, and sure enough, there were releases over the weekend—those must've been what broke it.
  39
  40 We upgraded our dependency on dateparser to 0.6, and tried again... and it still failed while trying to build the newest version of ruamel.yaml.
  41 One period of looking at GitHub blame views, commit histories, and unpacking PyPI tarballs later, I determined that version 0.6 of dateparser released on PyPI doesn't actually have the pin the version of ruamel.yaml, despite what the changelog claims.
  42 (I opened [dateparser issue #342][] for this.)
  43
  44 Since the version wasn't pinned, we just asked pip to first install an older version of ruamel.yaml, to hopefully get priority when dateparser tried to install it.
  45 So, we put `ruamel.yaml==0.13.14` in our package list, and then tried again.
  46 Finally, everything worked perfectly.
  47
  48 Case closed.
  49
  50 ---
  51
  52 # This Fix is a Mystery
  53
  54 But wait, what's this?
  55 Looking closer at the successful build logs, we can see that both `ruamel.yaml-0.13.14` and `ruamel.yaml-0.15.29` are installing without complaint.
  56 What's stopped the error?
  57 Well, if you'll look at the version number up at the top, we were installing `ruamel.yaml-0.15.28` before—just one hour previously, while I was on my lunch break, an update to ruamel.yaml had been released.
  58 Looking back at previous versions on PyPI, I finally figured out what had gone wrong.
  59 If you look at the downloads on the PyPI page for [ruamel.yaml version 0.15.28][0.15.28], you'll see that there are no Windows wheels.
  60 (Wheels are the format that Python uses to distribute compiled C extensions and pre-packed libraries.)
  61 However, if you go to the page for [version 0.15.29][0.15.29], then you'll see that Windows wheels are finally present.
  62 So, I guess until dateparser fixes their version pinning, we'll just have to hope that ruamel.yaml stays packaged correctly.
  63
  64 Case closed.
  65
  66 ---
  67
  68 # We Get Very Unlucky
  69
  70 Oops, nope it's not.
  71 Later in the afternoon, I got another message that some of the builds had failed.
  72 Looking at the first build that started failing, again we see that...
  73
  74 ```
  75 Collecting ruamel.yaml
  76   Using cached ruamel.yaml-0.15.30.tar.gz
  77 Installing collected packages: ruamel.yaml
  78  ...
  79  error: Microsoft Visual C++ 9.0 is required
  80 ```
  81
  82 okay, this project releases *fast*, this is the fourth release in 2 days.
  83 In any case, the last few builds succeeded with `0.15.30`, so what happened?
  84 Well, I don't know for sure, but I have a pretty good guess.
  85 I suspect that the release process for ruamel.yaml isn't atomic, and that they upload their source releases first, and the wheels come a bit later.
  86 We were unlucky enough to start a build during that first upload, where only the source package was available, and no Windows wheels.
  87 But, the few builds that got held up and started 4 minutes after the others took long enough that the wheels were available, and so they installed without any fuss.
  88
  89 This was an exceptionally unlucky situation.
  90 But, I've got a very good story now—and also a much greater appreciation for various package manager `.lock` files.
  91
  92 [dateparser issue #342]: https://github.com/scrapinghub/dateparser/issues/342
  93 [0.15.28]: https://pypi.python.org/pypi/ruamel.yaml/0.15.28
  94 [0.15.29]: https://pypi.python.org/pypi/ruamel.yaml/0.15.29