RELEASE.md

Lodestar Release Guidelines

Lodestar is a blockchain node securing the Ethereum Beacon chain network. It is run by external individuals and operator entities outside of the control of the Lodestar team. We, as most other core dev teams, choose a slow conservative approach to releasing to ensure those node runners always update to stable, safe, and performant versions of our software.

Lodestar uses a modified version of gitflow to manage releases. Gitflow is a well-known strategy that suits our needs for security and stability.

Stable release

When to a release

Lodestar does not have a fixed schedule for releases. Instead, they are published as the developers see fit.

The causes for a release are varied, however here are several common ones:

• To include a major bug-fix, optimization, or feature.
• To include a series of small changes which have shown to improve performance, usability, etc.
• If significant or risky PRs are waiting to merge and we wish to try and isolate those changes to a future release. I.e., to leave a "clean slate" for future PRs to apply to.
• An external team or entity needs a specific feature that the team has agreed to publish.

To start a new release, one of the Lodestar developers will communicate this via the Lodestar chat channel and seek consensus from the other developers.

1. Create release candidate

All-in-one script (for example version v1.1.0, commit 9fceb02):

• The team selects a commit from unstable as a "release candidate" for a new version release.
  • NOTE: In some rare circumstances, the team may select a commit from stable. This may happen if a tight deadline needs to be met and unstable can't be stabilized in time. We should avoid doing this unless absolutely necessary as merge conflicts and unintended consequences of cherry-picking commits may arise.
• yarn release:create-rc 1.1.0 9fceb02
  • Must be run locally from a write-access account capable of triggering CI.
  • This script may alternatively be run on the checked out HEAD:
    • git checkout 9fceb02
    • yarn release:create-rc 1.1.0
• Open draft PR from rc/v1.1.0 to stable with title chore: v1.1.0 release.

Manual steps (for example version v1.1.0, commit 9fceb02):

• The team selects a commit from unstable as a "release candidate" for a new version release.
• Create a new release branch rc/v1.1.0 at commit 9fceb02.
  • git checkout -b rc/v1.1.0 9fceb02
• Set monorepo version to v1.1.0.
  • lerna version v1.1.0 --no-git-tag-version --force-publish --yes
• Commit changes
  • git commit -am "v1.1.0"
  • git push origin rc/v1.1.0
• Open draft PR from rc/v1.1.0 to stable with title chore: v1.1.0 release.

2. Tag release candidate

Tagging a release candidate will trigger CI to publish to NPM, dockerhub, and Github releases.

All-in-one script (for example version v1.1.0, commit 8ab7cef):

• The team selects a commit from rc/v1.1.0 as a commit to tag and publish.
• yarn release:tag-rc 1.1.0 8ab7cef
  • Must be run locally from a write-access account capable of triggering CI.
  • This script may alternatively be run on the checked out HEAD:
    • git checkout 8ab7cef
    • yarn release:tag-rc 1.1.0

Manual steps (for example version v1.1.0, commit 8ab7cef):

• Check out the commit:
  • git checkout 8ab7cef
• Tag resulting commit as v1.1.0-rc.0 with an annotated tag, and push the tag.
  • git tag -am "v1.1.0-rc.0" v1.1.0-rc.0
  • git push origin v1.1.0-rc.0

3. Test release candidate

Once a release candidate is created, the Lodestar team begins a testing period.

If there is a bug discovered during the testing period which significantly impacts performance, security, or stability, and it is determined that it is no longer prudent to promote the rc.x candidate to stable, then it will await a bug fix by the team. The fix will be committed to unstable first, then cherrypicked into the rc/v1.1.0 branch. Then we publish and promote the new commit to rc.x+1. The 3-day testing period will reset.

For example: After 3-5 days of testing, is performance equal to or better than latest stable?

• Yes: Continue to the next release step
• No: If it a small issue fixable quickly (hotfix)?
  • Yes: Merge fix(es) to unstable, push the fix(es) to rc/v1.1.0 branch, go to step 2, incrementing the rc version
  • No: abort the release. Close the chore: v1.1.0 release PR, delete the branch, and start the whole release process over.

4. Merge release candidate

• Ensure step 2 testing is successful and there is sufficient consensus to release v1.1.0.
• Approving the chore: v1.1.0 release PR means a team member marks the release as safe, after personally reviewing and / or testing it.
• Merge chore: v1.1.0 release PR to stable with "merge commit" strategy to preserve all history.
• Merge stable stable into unstable with merge commit strategy. Due to branch protections in unstable must open a PR. If there are conflicts, those must be resolved manually. Gitflow may cause changes that conflict between stable and unstable, for example due to a hotfix that is backported. If that happens, disable branch protections in unstable, merge locally fixing conflicts, run lint + tests, push, and re-enable branch protections.

5. Tag stable release

Tagging a stable release will trigger CI to publish to NPM, dockerhub, and Github releases.

All-in-one script (for example version v1.1.0):

• git checkout stable
• yarn release:tag-stable 1.1.0
  • Must be run locally from a write-access account capable of triggering CI.

Manual steps (for example version v1.1.0):

• Check out the new stable
  • git checkout stable
• Tag it as v1.1.0 with an annotated tag, push commit and tag.
  • git tag -am "v1.1.0" v1.1.0
  • git push origin v1.1.0

6. Announce

• Double check that Github release is correct
• Follow Publish to Social Media steps

Hotfix release

If a stable version requires an immediate fix before the next release, a hotfix release is started.

A similar process for a stable release is used, with the three differences.

• The candidate commit must be chosen from the stable branch instead of the unstable branch.
• Depending on the severity of the bug being fixed, the testing window may be decreased.
• All hotfixes are committed with an unstable first strategy rather than directly on the RC branch itself. Hotfixes are always merged to unstable first, then cherry-picked into hotfix release candidates.

1. Create hotfix release candidate from stable branch

All-in-one script (for example version v1.1.1, stable commit 8eb8dce):

• Select the latest commit from stable as the "hotfix release candidate" for a new hotfix version release.
• git fetch origin stable
• git checkout stable
• yarn release:create-rc 1.1.1
  • Must be run locally from a write-access account capable of triggering CI.
• Switch to the hotfix release branch and cherrypick the inclusion(s) from the unstable branch to the hotfix release.
  • git checkout rc/v1.1.1
  • git cherry-pick {commit}
• Open draft PR from rc/v1.1.1 to stable with the title chore: v1.1.1 release.

Manual steps (for example version v1.1.1, commit 8eb8dce):

• Select the latest commit from stable as the "hotfix release candidate" for a new hotfix release.
• Checkout stable branch
  • git checkout stable
• Create a new release branch rc/v1.1.1 at commit 8eb8dce
  • git checkout -b rc/v1.1.1 8eb8dce
• Set monorepo version to v.1.1.1.
  • lerna version v1.1.1 --no-git-tag-version --force-publish --yes
• Commit changes
  • git commit -am "v1.1.1"
  • git push origin rc/v1.1.1 Open draft PR from rc/v1.1.1 to stable with the title chore: v1.1.1 release.

2. Tag release candidate

Tagging a release candidate will trigger CI to publish to NPM, dockerhub, and Github releases.

All-in-one script (for example version v1.1.1, commit f3df9f8):

• Select the latest commit from rc/v1.1.1 to tag and publish.
• yarn release:tag-rc 1.1.1
  • Must be run locally from a write-access account capable of triggering CI.

Manual steps (for example version v1.1.1, commit f3df9f8):

• Tag latest commit as v1.1.1-rc.0 with an annotated tag, and push the tag.
  • git tag -am "v1.1.1-rc.0" v1.1.1-rc.0
  • git push origin v1.1.1-rc.0

3. Test hotfix release candidate

Once a hotfix release candidate is created, the Lodestar team may begin a modified hotfix testing period consisting of a quick sanity check or longer if required.

If the hotfix does not address the purpose of the hotfix release, or there is another bug discovered during this modified hotfix testing period which significantly impacts performance, security, or stability, and it is determined that it is no longer prudent to promote the rc.x candidate to stable, then it will await an additional fix by the team. The fix will be committed to unstable first, then cherrypicked into the rc/v1.1.1 hotfix branch. Then we publish and promote the new commit to rc.x+1. The modified hotfix testing period will reset.

For example: After modified hotfix testing period, is the original bug resolved? Is performance equal to or better than latest stable?

• Yes: Continue to the next release step
• No: If it a small issue fixable quickly with another hotfix?
  • Yes: Merge fix(es) to unstable, push the fix(es) to rc/v1.1.1 hotfix branch, go to step 2, incrementing the rc version
  • No: Abort the release. Close the chore: v1.1.v release PR, delete the branch, and start the whole release process over.

4. Merge hotfix release candidate

• Ensure step 3 testing is successful and there is sufficient consensus to release v1.1.1.
• Approving the chore: v1.1.1 release PR means a team member marks the release as safe, after personally reviewing and / or testing it.
• Merge chore: v1.1.1 release PR to stable with "merge commit" strategy to preserve all history.
• Merge stable into unstable with merge commit strategy. Due to branch protections in unstable must open a PR. If there are conflicts, those must be resolved manually. Gitflow may cause changes that conflict between stable and unstable, for example due to a hotfix that is backported. If that happens, disable branch protections in unstable, merge locally fixing conflicts, run lint + tests, push, and re-enable branch protections. See "Backporting merge conflicts from stable to unstable".

Pull the latest commits on both stable and unstable branches:

• git checkout stable && git pull origin stable
• git checkout unstable && git pull origin unstable

Merge stable into unstable, resolving conflicts:

• git checkout unstable && git merge stable
• Resolve conflicts
• Sanity check locally before pushing by using: git diff origin/unstable unstable
• Disable unstable branch protection
• git push
• Enable unstable branch protection

5. Tag stable hotfix release

Tagging a stable release will trigger CI to publish to NPM, dockerhub, and Github releases.

All-in-one script (for example version v1.1.1):

• git checkout stable
• yarn release:tag-stable 1.1.1
  • Must be run locally from a write-access account capable of triggering CI.

Manual steps (for example version v1.1.1):

• Check out the new stable
  • git checkout stable
• Tag it as v1.1.1 with an annotated tag, push commit and tag.
  • git tag -am "v1.1.1" v1.1.1
  • git push origin v1.1.1

6. Announce

• Double check that Github release is correct
• Follow Publish to Social Media steps

Dev release

On every commit to unstable a dev release is done automatically in CI. A dev release:

• is not tagged
• does not have a release page
• is published to NPM
• is pushed to Dockerhub

The source code is mutated before release to set a version string of format v1.1.0-dev.da9f72360, where the appended hash is the merge commit hash to unstable that triggered this CI build. The semver version that prefixes is expected to be the next minor version from the current code. The target consumers of such versions are automatic testing environments and other developers. They are expected to install via next tags and refer to the whole version for debugging.

Details

Publishing a release

The publishing of stable releases and release candidates is triggered by pushing a tag.

CI ensures the validity of the stable release and releases candidates by checking:

• the tag matches the version in the source
• for stable releases, the commit is the latest in the stable branch

This prevents accidentally publishing an incorrect version of Lodestar.

Dev releases are triggered on every new push to unstable.

Github workflows publish:

• to NPM registry
• to Dockerhub
• to Github releases

The behavior differs based on whether a stable release, a release candidate, or dev release is being performed.

• Stable release
  • published to npm with latest dist tag
  • published to docker with latest tag
  • a full Github release is published
• Release candidate
  • published to npm with rc dist tag
  • published to docker with rc tag
  • a prerelease Github release is published
• Dev release
  • published to npm with next dist tag
  • published to docker with next tag
  • no Github release is published

How to test release candidates

We test the pre-release candidate on multiple servers with a variety of connected validators on a stable testnet for a minimum of three (3) days.

The following observations must be taken into consideration before promoting the release candidate to stable:

• Are there any critical issues observed?
  • Examples: Memory leaks, abnormally high memory usage, abnormally high CPU performance, etc.
• Has profitability been affected?
  • Has profitability decreased and whether or not the variance is acceptable.
• Has any performance metric degraded comparably to the previous stable release? If yes, is the variance acceptable?
  • Example: Block processing times, validator miss ratios, etc.

Edit the Release

Releases are published automatically via CI.

Any additional release notes should be professional, comprehensive, and well-considered.

Have someone else review the release notes and then edit the release.

Publish to Social Media

The release should be announced on the following social channels:

• Discord: Use the #lodestar-announcements channel. Ensure it is published to all downstream channels
• Twitter: Short and sweet in a single tweet or thread with twitter.com/lodestar_eth
• Blog post (if necessary): To outline specific changes that require additional context for users

Release Manager Checklist

This section is to guide the Release Manager tasked with the next version release to ensure all items have been completed.

• Start thread on communication channels for new release
• Confirm consensus on unstable release candidate commit
• Complete Step 1: Create release candidate
• Complete Step 2: Tag release candidate
• Deploy rc.x candidate to beta group of servers
• If there are rc.x hot fixes, push to branch and increment the rc.x version.
• Team members conduct Release Candidate Metrics Review
• A Lodestar team member must mark the release candidate as safe, after personally reviewing and / or testing it
• Backup stable and unstable branches locally for restoration in case of accidental use of the incorrect merge method
• Temporarily enable "Allow merge commits" under the Lodestar repository settings
• Release Manager can now complete Step 4: Merge release candidate.
• Disable "Allow merge commits" under the Lodestar repository settings
• Complete Step 5: Tag stable release
• Double check that Github release is correct and inform the Project Manager of completion
• Project Manager to follow up with Devops updating both bn and vc stable servers

Alternatives considered

Click to expand!

Historical context and reasons against valid alternatives to help future discussions

Version branches

Lodestar used master as the single target for feature branches.

• Main branch = master
• Features merged to master
• To trigger rc, branch from master to v1.1.x
• master had package.json preemptively updated to the "next" version
• QA is done on v1.1.x branch
• Fixes on rc are done on v1.1.x, then re-tag
• Once released final v1.1.0 tag is on a branch that is never merged
• Hotfixes are either cherry-picked from master or done on the v1.1.x branch, never merged

However, this had some issues:

• Aborted releases left master in awkward version 2 minors ahead of master. When triggering the release again, we had to rollback master
• Almost all release tags ended in branches not part of the master tree. This caused issues since it's not straightforward to compute the diff between commits that are not direct parents of each other

Continuous integration

Always releasing master is another popular approach used by some entities but unsuitable for Lodestar. Given the complexity of a blockchain node, it's not possible to guarantee stable performance unless running the software for days under special conditions, not available in regular CI environments.