Support merge trains (any platform)

[rarkins]

What would you like Renovate to be able to do?

Similar to the idea in #5573 but to be implemented completely within Renovate's own logic.

Describe the solution you'd like

Something that means multiple PRs don't need to be rebased after each merge.

Describe alternatives you've considered

Having a "roll up" PR that adds all passing updates (i.e. test them individually, then add to a single group PR after).

Additional context

renovatebot/config-help#610

[rarkins]

Thanks, but I don't think this will be helpful here. We don't have a problem with merging - we have a problem with lots of rebasing, which GitHub automerge wouldn't help.

[HonkingGoose]

For future reference: the alternative idea of "rollup" updates has its own issue:

• roll up updates into single PR #4404

[HonkingGoose]

@rarkins GitHub is starting work on something similar to GitLab's Merge train, they're calling it a "Pull Request Merge Queue".

Relevant quote from their issues: ^{1 2}

How will it work?

A developer can add their pull request to the merge queue when:

1. Merge queue is enabled on a branch via a new branch protection setting, and...
2. All merge requirements have been met (e.g. including required reviewer and status check)

Once in the queue, the pull request's changes will be:

1. validated to ensure it passes all required checks before merging
2. merged with the latest version of the branch along with any pull requests ahead of it in the queue

Footnotes

1. https://github.com/github/roadmap/issues/272 ↩

2. https://github.com/github/roadmap/issues/370 ↩

[rarkins]

I'm wondering if it's more about "keep rebasing until merged" rather than merge trains - which intentionally rebase themselves on top of the PR they trail in the queue (it's more efficient)

[HonkingGoose]

I'm wondering if it's more about "keep rebasing until merged" rather than merge trains - which intentionally rebase themselves on top of the PR they trail in the queue (it's more efficient)

I just checked the GitLab docs on merge trains so I'm on the same page as you. ¹

Merge train example

Three merge requests (A, B and C) are added to a merge train in order, which creates three merged results pipelines that run in parallel:

1. The first pipeline runs on the changes from A combined with the target branch.
2. The second pipeline runs on the changes from A and B combined with the target branch.
3. The third pipeline runs on the changes from A, B, and C combined with the target branch.

If the pipeline for B fails, it is removed from the train. The pipeline for C restarts with the A and C changes, but without the B changes.

If A then completes successfully, it merges into the target branch, and C continues to run. If more merge requests are added to the train, they now include the A changes that are included in the target branch, and the C changes that are from the merge request already in the train.

Wow! I'm really impressed with the smart behavior that GitLab managed to build! 🤯

Maybe GitHub is thinking of something similar, it's a bit hidden in this quote:

Once in the queue, the pull request's changes will be:

1. validated to ensure it passes all required checks before merging
2. merged with the latest version of the branch along with any pull requests ahead of it in the queue

The bolded part sounds a bit similar to GitLabs merge train?

In any case, even if GitHub "only" builds a "brute-force" mechanism that auto-updates the PR branch with main that would be a big help compared to the current behavior. 😄

Footnotes

1. https://docs.gitlab.com/ee/ci/pipelines/merge_trains.html#merge-train-example ↩

[rarkins]

Yes, that quote does sound like they're planning something similar

[kolkinn]

Just letting you know that some changes seem to be necessary in order for Renovate to merge using GitHubs new Merge Queue. We have enabled it on one of our repositories recently, and since then no Renovate PRs have been merged automatically for that repository.

[rarkins]

Are you attempting to manually add a Renovate PR to your branch's merge queue as per the description in https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/merging-a-pull-request-with-a-merge-queue ?

Please provide much more details of what you're trying to do and what doesn't work. E.g. if you have mandated a merge queue in a branch, and now Renovate can't automerge, that doesn't sound too surprising based on the docs I saw above

[kolkinn]

This might differ from one beta tester to another beta tester if GitHub is doing A/B testing, but in our case it seems necessary to require the use of merge queues in order to be able to use it at all.
It is with the mandatory merge queues that Renovate cannot automerge. Manually adding a Renovate PR to the merge queue using the "Merge when ready" button does work, just like a regular merge.

With mandatory merge queues enabled:

Without mandatory merge queues enabled:

[rarkins]

It sounds like the current approach of merge queues is potentially incompatible with the concept of automerging, unless we can access those options via API

[adamstrawson]

Is this a feature that is planned to be implemented? We're having to manually merge Renovate PR's at the moment as we're using GitHub merge queues, but it'll be nice to be able to auto-merge as before so that it's not a chore our engineers are having to maintain.

[rarkins]

Nothing in status:requirements is planned, because it's not ready to start. I looked in the above history and don't think there's any consensus on what needs to be done

[HonkingGoose]

Here's the new upstream issue for general release of the GitHub merge queue:

• Pull Request Merge Queue github/roadmap#370

The GitHub Docs, Merging a pull request with a merge queue have some more information about how it works in the GitHub interface. But that's probably not much help for the maintainers of Renovate.

The feature is still in beta, per the GitHub docs:

Note: The pull request merge queue feature is currently in limited public beta and subject to change.

[rarkins]

Functionally, is this a new type of "automerge" where Renovate adds it to a merge queue once requirements are met, instead of enabling automerge at PR creation or automerging itself?

Technically, does the API for this exist yet?

[HonkingGoose]

I don't know if there's an API yet, but there's a new webhook ¹ event and GitHub Actions workflow trigger ² called merge_group. Quote from the GitHub Blog: ³

A new webhook event and GitHub Actions workflow trigger (merge_group) makes it easier to run required status checks on merge groups created by merge queue. A merge group includes the changes from one or more pull requests and must pass the status checks required by the target branch.

I recommend you read all three links, and see if it's what you need. 😉

Footnotes

1. https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#merge_group ↩

2. https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#merge_group ↩

3. https://github.blog/changelog/2022-08-18-merge-group-webhook-event-and-github-actions-workflow-trigger/ ↩

[kaelig]

After reading the docs and this thread, I'm 90% sure of the answer to my question but I wanted to clarify...

At this time, is there a way to use Renovate + GitHub Merge Queue + Automerge (squash)? (if so, I'd love to know more!)

[HonkingGoose]

There's no way to use Renovate + GitHub Merge Queue + Automerge (squash). The Merge Queue feature is still in limited public beta, and we didn't get in the beta.

I think we need to have access to the Merge Queue feature first. That way we can test the feature and see how to integrate it into Renovate bot.

[rarkins]

The feature also needs to be available via API.

BTW I think once it's technically feasible, we should break GitHub Merge trains into its own issue. This one was intended originally as a "generic" way to achieve the same without native platform support

[kaelig]

Thank you for the context, I'll bring it up to GitHub in an upcoming discussion about the Netlify Deploy Preview <> GitHub Merge Queue developer workflow.

If you'd like, I can ask that they enroll you to the beta?

[rarkins]

Yes please do - best would be this open source org "renovatebot" but if they insist that it should be an enterprise customer only then my and Renovate's parent company "mend" would still be useful for me to test with if included.

[vchernin]

The github merge queue feature is in public beta now and I have been playing around with it. The merge_group event is currently broken but otherwise it seems usable.

[rarkins]

Does it support an API for Renovate to be able to add its PRs to the queue?

[vchernin]

As far as I can tell there isn't a REST api, but there is a github graphql mutation enablePullRequestAutoMerge. Tried it on my repo using https://docs.github.com/en/graphql/overview/explorer and it works as expected.

query get_pr_id {
    repository(owner: "$REPO_ORG", name: "$REPO_NAME") {
        pullRequest(number: PR_NUMBER) {
                  id
              }
        } 
}

mutation add_to_queue {
    enablePullRequestAutoMerge(input: {pullRequestId: "$ID_FROM_PREVIOUS_QUERY", mergeMethod: MERGE}) {
        clientMutationId
         }
}

[vchernin]

Actually, Renovate can already add to GitHub merge queues. It is necessary to enable auto merge in GitHub's repo settings (even though a merge queue is being used). Renovate is already making the right API call:

renovate/lib/modules/platform/github/index.ts

Lines 1455 to 1458 in bde9512

	const res = await githubApi.requestGraphql<GhAutomergeResponse>(
	enableAutoMergeMutation,
	queryOptions
	);

So it ends up looking like so:

This is using platform auto merge:

{
    "automerge": true,
    "automergeType": "pr",
    "platformAutomerge": true
}

[HonkingGoose]

BTW I think once it's technically feasible, we should break GitHub Merge trains into its own issue. This one was intended originally as a "generic" way to achieve the same without native platform support

Let's discuss the GitHub Merge Queue in this issue from now on:

• Explain how to get GitHub's pull request merge queue working with Renovate #20321

[rarkins]

Considering how good GitHub's merge queue feature is, I no longer plan for us to have a "generic" merge train, so have converted this to an Ideas discussion

[HonkingGoose]

So we're not building "our own merge train feature for all Renovate platforms that we support" anymore, correct? Do you still want to support GitLab's Merge Trains:

• Support GitLab "merge trains" #5573

[rarkins]

Correct and yes