Reflections on GHC's release schedule

Ben Gamari - 2017-08-01

Looking back on GHC’s past release schedule reveals a rather checkered past,

There are a few things to notice here:

• release cadence has swung rather wildly
• the release cycle has stretched in the last several releases
• time-between-releases generally tends to be on the order of a year

While GHC is far from the only compiler with such an extended release schedule, others (namely LLVM, Go, and, on the extreme end, Rust) have shown that shorter cycles are possible. I personally think that a more stable, shorter release cycle would be better for developers and users alike,

• developers have a tighter feedback loop, inducing less pressure to get new features and non-critical bugfixes into minor releases
• release managers have fewer patches to cherry-pick
• users see new features and bugfixes more quickly

With 8.2.1 at long last behind us, now is a good time to reflect on why these cycles are so long, what release schedule we would like to have, and what we can change to realize such a schedule. On the way we’ll take some time to examine the circumstances that lead to the 8.2.1 release which, while not typical, remind us that there is a certain amount of unpredictability inherent in developing large systems like GHC; a fact that must be born in mind when considering release policy.

Let’s dig in…

The release process today

Cutting a GHC release is a fairly lengthy process involving many parties and a significant amount of planning. The typical process for a major release looks something like this,

1. (a few months after the previous major release) A set of release priorities are defined determining which major features we want in the coming release
2. wait until all major features are merged to the master branch
3. when all features are merged, cut a stable branch
4. in parallel:
   1. coordinate with core library authors to determine which library versions the new release should ship
   2. prepare release documentation
   3. do preliminary testing against Hackage and Stackage to identify and fix early bugs
   4. backport significant fixes merged to master
5. when the tasks in (4) are sufficiently advanced, cut a source release for a release candidate
6. produce tier-1 builds and send source tarballs to binary packagers, wait a week to prepare binary builds; if anyone finds the tree is unbuildable, go back to (5)
7. upload release artifacts, announce release candidate
8. wait a few weeks for testing
9. if there are significant issues: fix them and return to (5)
10. finalize release details (e.g. release notes, last check over core library versions)
11. cut source tarball, send to binary build contributors, wait a week for builds
12. announce final release, celebrate!

Typically the largest time-sinks in this process are waiting for regression fixes and coordinating with core library authors. In particular, the coordination involved in the latter isn’t difficult, but merely high latency.

In the case of 8.2.1, the timeline looked something like this,

Unexpected set-backs

This timeline was a bit more extended than desired for a few reasons.

The first issues were #13426 and #13535, compile-time performance regressions which came to light shortly after the branch and after the first release candidate, respectively. In #13535 it was observed that the testsuite of the vector package (already known for its propensity to reveal compiler regressions) increased by nearly a factor of five in compile-time allocations over 8.0.2.

While a performance regression would rarely classify as a release blocker, both the severity of the regressions combined with the fact that 8.2 was intended to be a performance-oriented release made releasing before fixes were available quite unappealing. For this reason David Feuer, Reid Barton, and I invested significant effort to try to track down the culprits. Unfortunately, the timescale on which this sort of bug is resolved span days, stretching to weeks when time is split with other responsibilities. While Reid’s valiant efforts lead to the resolution of #13426, we were eventually forced to set #13535 aside as the release cycle wore on.

The second setback came in the form of two quite grave correctness issues (#13615, #13916) late in the cycle. GHC being a compiler, we take correctness very seriously: Users’ confidence that GHC will compile their programs faithfully is crucial for language adoption, yet also very easily shaken. Consequently, while neither of these issues were regressions from 8.0, we deemed it important to hold the 8.2 release until these issues were resolved (which ended up being significant efforts in their own right; a blog post on this will be coming soon).

Finally, there was the realization (#13739) after release candidate 2 that some BFD linker releases suffered from very poor performance when linking with split-sections enabled (the default behavior in 8.2.1). This served as a forcing function to act on #13541, which we originally planned for 8.4. As expected, it took quite some time to follow through on this in a way that satisfied users and distribution packagers in a portable manner.

Moving forward: Compressing the release schedule

Collectively the above issues set the release back by perhaps six or eight weeks in total, including the additional release candidate necessary to validate the raft of resulting patches. While set-backs due to long-standing bugs are hard to avoid, there are a few areas where we can do better,

1. automate the production of release artifacts
2. regularly test GHC against user packages in between releases
3. expand continuous integration of GHC to less common platforms to ensure that compatibility problems are caught before the release candidate stage
4. regularly synchronize with core library maintainers between releases to reduce need for version bound bumps at release time
5. putting in place tools to ease bisection, which is frequently a useful debugging strategy around release-time

As it turns out, nearly all of these are helped by our on-going effort to move GHC’s CI infrastructure to Jenkins (see #13716). As this is a rather deep topic in its own right, I’ll leave this more technical discussion for a second post (blog:jenkins-ci).

With the above tooling and process improvements, I think it would be feasible to get the GHC release cycle down to six months or shorter if we so desired. Of course, shorter isn’t necessarily better: we need to be careful to balance the desire for a short release cycle against the need for an adequate post-release “percolation” time. This time is crucial to allow the community to adopt the new release, discover and fix its regressions. In fact, the predictability that a short release schedule (hopefully) affords is arguably more important than the high cadence itself.

Consequently, we are considering tightening up the release schedule for future GHC releases in a slow and measured manner. Given that we are now well into the summer, I think positioning the 8.4 release around February 2018, around seven months from now, would be a sensible timeline. However, we would like to hear your opinions.

Here are some things to think about,

1. Do you feel that it takes too long for GHC features to make it to users’ hands?
2. How many times per year do you envision upgrading your compiler before the process becomes too onerous? Would the current load of interface changes per release be acceptable under a faster release cadence?
3. Should we adjust the three-release policy to counteract a shorter GHC release cycle?
4. Would you feel more likely to contribute to GHC if your work were more quickly available in a release?

We would love to hear your thoughts. Be sure to mention whether you are a user, GHC contributor, or both.