Simplify the release process by removing Code Thaw

[saschagrunert]

Background

Right now we apply Code Freeze with the first beta on the master branch. From that point in time, only PRs within the milestone can be merged. On Test Freeze we create the release branch and cut the first release candidate (rc). The master branch is still frozen until we create the final release, which is called Thaw.

More details about the release process can be found here:
https://github.com/kubernetes/sig-release/blob/master/releases/release_phases.md#code-freeze

Idea

Remove Thaw, and completely open the master branch on Test Freeze.

Benefits

• The discussion about having fast-forwards vs cherry-picks during the minor release process would be obsolete and we would always require cherry-picks
• The process aligns with the cherry-pick process for other release branches generally.
• We unlink contributors from the release process earlier

Before

After

Drawbacks

That's where I need your input, especially from @kubernetes/release-managers.

[saschagrunert]

@liggitt, would this make things easier when looking back to previous discussions of fast-forwards vs cherry-picks on the release branch?

[xmudrii]

I’m not sure that I like this approach at all. While it simplifies the workflow, I think it increases the risk by a large margin. I had to skip the retro, so I’m missing some context, but let’s try to get it documented here at least.

Right now, cherry-picks have to be created manually. This is error-prone, because someone might forgot the cherry-pick. This is fixed in a way that a Release Manager or a Release Team Lead compares the master and the release branch, and then cherry-picks what’s forgotten. This is easy, however, if we remove Thaw, we will have a huge influx of PRs that will make impossible to compare two branches and find what should be cherry-picked.

The key problem here is that it requires a very good communication and we’re not that good with that. Every contributor must know that they must cherry-pick the PR otherwise it will not land in the release. Again, when you have a huge number of PRs getting merged, you can’t control this at all.

IMO, I see two ways out of this:

• Switch to doing FFs
  • I don’t like this because it must be done manually (more burden), and must be done by a Release Manager (very small subset of people can do it), and on top of that it requires a lot of permissions in the repository (that’s why RM must do it)
  • On the other side, it’s a much safer approach and nothing will be forgotten
• Move cutting the release branch along with the final release
  • @justaugustus mentioned there are some technical obstacles with this that would have to be solved first
  • @liggitt prefers the current approach (where we cut 2 weeks earlier) because we get to test new jobs and prepare everything
  • I, personally, prefer this approach because we neither have FFs nor cherry-picks, so much less things to worry about. Especially if we keep Thaw, having master and release branch follow each other doesn’t make much sense to me
  • The problem is that in this case we would have to move Thaw for a day or two later until we don’t setup jobs and dashboards. There’s the problem @liggitt mentioned, but moving Thaw for a day could mitigate that because we would have a chance to test that everything works fine

[saschagrunert]

Move cutting the release branch along with the final release

This has a direct impact on downstream consumers which may rely on the existence of the release branch. I don't think that we want to introduce a possibly breaking process.

[liggitt]

@liggitt, would this make things easier when looking back to previous discussions of fast-forwards vs cherry-picks on the release branch?

no, it makes it worse... in that scenario, we would not only have to manage individual cherry-picks for the 1.x.0 release (with all the risk of omitting fixes or picking them out of order and conflicting), but letting master race ahead as soon as release-1.x is created almost guarantees that any release-blocking fixes needed for 1.x.0 will get stuck in the master merge flood, and will hit conflicts there as well.

[JamesLaverack]

… letting master race ahead as soon as release-1.x is created almost guarantees that any release-blocking fixes needed for 1.x.0 will get stuck in the master merge flood, and will hit conflicts there as well.

Is it required that release-blocking fixes must merge to master and then be cherry-picked to release-1.x? Could we instead merge them directly to release-1.x and cherry-pick them to master?

[liggitt]

Is it required that release-blocking fixes must merge to master and then be cherry-picked to release-1.x?

in my opinion, absolutely yes. we should never merge to a release branch before merging to master (unless the fix doesn't apply to master for some reason)

[xmudrii]

+1 to what @liggitt said. Otherwise, there’s a risk that something can get done on the release branch, but not on the master branch (e.g. due to an error when porting from the release to the master branch). Such situations can be very hard to catch when you have a lot of commits.

[saschagrunert]

Switch to doing FFs
I don’t like this because it must be done manually (more burden)

We already discussed that it would be technically possible to automate it in a periodic prow job. We just have to find a way to handle the credentials, like we do in GCB.

[JamesLaverack]

My current draft of the schedule for 1.24 does not include a change to code freeze vs 1.23.

[saschagrunert]

Following-up on the topic and proposing an automatic fast-forward in kubernetes/release#2386