From 12f622acdb23683a49d886edf616a85fea4d27d6 Mon Sep 17 00:00:00 2001 From: Stewart X Addison Date: Wed, 1 May 2024 12:32:13 +0100 Subject: [PATCH 1/4] April 2024 checklist updates Signed-off-by: Stewart X Addison --- .github/ISSUE_TEMPLATE/release-checklist.md | 22 +++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/release-checklist.md b/.github/ISSUE_TEMPLATE/release-checklist.md index 23b97bf..b671ba2 100644 --- a/.github/ISSUE_TEMPLATE/release-checklist.md +++ b/.github/ISSUE_TEMPLATE/release-checklist.md @@ -32,7 +32,7 @@ Everyone participating in a release, including the release champion are requeste - [ ] **Notify release branching of build repositories** : [Slack message, branching build repositories](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#branching-message-for-build-related-repositories) - [ ] **Create build repositories release Branches** : [Create build repository release branches](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#create-release-branch-on-below-repositories) -- [ ] **Identify the aqa branch name for the upcoming release** +- [ ] **Identify the aqa branch name for the upcoming release (Note, April and October PSU updates generally use same branch as the March/September new releases** Ensure ALL nodes online prior to running these following TC steps: - [ ] TC: Run the DeleteJCKMultiNode process cleaning job on all ci.role.test nodes, to remove any now redundent jck-versions, to ensure healthy state, verify all nodes successful: https://ci.eclipse.org/temurin-compliance/job/DeleteJCKMultiNode @@ -42,6 +42,7 @@ Ensure ALL nodes online prior to running these following TC steps: - [ ] **Check the nagios server to ensure there are no critical infrastructure issues** Log in to the public [nagios](https://nagios.adoptopenjdk.net/nagios/) server, and check the Problems / Services page. If you do not have access, please request it via an issue in the infrastructure repository. If there are any issues, then please log an issue in the infrastructure repository. - [ ] **Regenerate The Release Build Pipeline Jobs In Jenkins** + - [ ] **Update testenv.properties in the AQA release branch to use the -dryrun-ga branches** ([Sample PR](https://github.com/adoptium/aqa-tests/pull/5202/files)) - [ ] **Prepare & Perform Dry Run Of Build & Tests** : [Dry-run](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#auto-way---before-release-week-dry-run-release-test) - [ ] **Triage dry-run TCK job results** - [ ] **Restore aqa-tests release branch testenv.properties JDK_BRANCH values to the "-ga" tag after dry-run has completed** @@ -70,21 +71,20 @@ After 1 day, then :- - [ ] TC: Run the Setup_JCK_Run_Multinode job with CLEAN_DIR=true (to purge any old release contents/results) on all ci.role.test nodes, this will extract the jck_run folder with all the temurin.jtx exclude files, verify all nodes successful : https://ci.eclipse.org/temurin-compliance/job/Setup_JCK_Run_Multinode/build?delay=0sec - [ ] Check the nagios server to ensure there are no critical infrastructure issues - [ ] **Trigger a trial release pipeline dry-run** to ensure less surprises on release day (typically against a milestone build), see here for [details](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#auto-way---before-release-week-auto-test) - - [ ] Confirm successful trial release pipelines and successful jck completion -- [ ] Calculate the "expected" openjdk build tags for the releases being published, and update all the JDKnn_BRANCH values in the testenv.properties file for the aqa-tests release branch, eg: https://github.com/adoptium/aqa-tests/blob/v0.9.6-release/testenv/testenv.properties ------- Release Week Checklist: -- [ ] -- Check All Nodes Online https://ci.eclipse.org/temurin-compliance/label/ci.role.test/ -- [ ] Run https://ci.eclipse.org/temurin-compliance/job/ProcessCheckMultiNode/ -- with defaults -- [ ] Run Setup_JCK_Multinode with CLEAN_DIR=true for ( ci.role.test ) +- [ ] Check All Nodes Online https://ci.eclipse.org/temurin-compliance/label/ci.role.test/ +- [ ] Run https://ci.eclipse.org/temurin-compliance/job/ProcessCheckMultiNode/ -- with defaults +- [ ] Run Setup_JCK_Run_Multinode with CLEAN_DIR=true for ( ci.role.test ) - [ ] Disable Setup_JCK_Run_Multinode To Ensure Test Evidence Is Not Lost - [ ] As detailed earlier, again check the nagios server to ensure there are no critical infrastructure issues -- [ ] Create the Github Issues for tracking progress against each Java version in the adoptium/temurin repo -- [ ] Create the Github issues for the Adoptium public retro & TC retro in the adoptium/temurin repo +- [ ] [Create the Github Issues for tracking progress](https://github.com/adoptium/temurin/issues/new?assignees=&labels=&projects=&template=release-status.md&title=%3Cmonth%3E+%3Cyear%3E+Release+Status+per+Platform%2C+Version+%26+Binary+Type) in the adoptium/temurin repo +- [ ] Create the Github issues for the [Adoptium public retro](https://github.com/adoptium/temurin/issues/new?assignees=&labels=Retrospective&projects=&template=retrospectives.md&title=General+Retrospective+for+%3Cmonth%3E+%3Cyear%3E+Releases) in the adoptium/temurin repo +- [ ] Create the retrospective issue for the Temurin Compliance project - [ ] Update the links on the slack channel for the release status and retrospective issues. #### Release Day Onwards @@ -124,7 +124,8 @@ Release Week Checklist: - [ ] **Fix** blocking failures if they exist and confirm others are non-blocking. - [ ] **Confirm Temurin-compliance items completed**, per platform/version/binaryType - [ ] **Get PMC 'ready to publish' approval**, once no blocking failures exist. -- [ ] **Generate The Release Notes Per JDK Version **, ( Use https://ci.adoptium.net/job/build-scripts/job/release/job/create_release_notes/ ) +- [ ] **Generate The Release Notes Per JDK Version **, ( Use https://ci.adoptium.net/job/build-scripts/job/release/job/create_release_notes/ ) and publish them with the [release tool](https://ci.adoptium.net/job/build-scripts/job/release/job/refactor_openjdk_release_tool/) using `UPSTREAM_JOB_NAME` of `create_release_notes` and the appropriate JOB NUMBER +- [ ] **Verify that the release notes are live (as [we've had problems with this](https://github.com/adoptium/temurin/issues/28#issuecomment-2077786176)) and remediate if required. - [ ] **Publish the release** (run the restricted access [release tool job](https://ci.adoptopenjdk.net/job/build-scripts/job/release/job/refactor_openjdk_release_tool/) on Jenkins) ( also publish release notes ) - [ ] **Consider updating the API** as required via the relevant parts of [the Adoptium API model constants](https://github.com/adoptium/api.adoptium.net/blob/main/adoptium-models-parent/adoptium-api-v3-models/src/main/kotlin/net/adoptium/api/v3/models/Versions.kt). - [ ] **Verify binaries published successfully** to github releases repo and website (_automate_*, this could also be an automated test) @@ -142,5 +143,6 @@ Release Week Checklist: - [ ] **Do all of the above for the jdk8u/aarch32 build: Ensure to specify overridePublishName param** - [ ] **Archive/upload all TCK results** - [ ] **Use EclipseMirror job in the Temurin Compliance jenkins to store a backup** of the release artifacts -- [ ] **Declare the release complete** and close this issue - [ ] **Create a draft PR for a release blog post for next release** in the [adoptium.net](https://github.com/adoptium/adoptium.net/pulls) repository and ensure to delegate the task of finalizing and publishing the existing draft PR for this release's blog post. +- [ ] **Ensure the [adoptium calendar](https://calendar.google.com/calendar/u/0/embed?src=c_56d7263c0ceda87a1678f6144426f23fb53721480b5ff71b073afb51091e5492@group.calendar.google.com) is updated for the next cycle at a minimum** +- [ ] **Declare the release complete** and close this issue From 48db31944c8160b8ec639019723059512bc3f4d9 Mon Sep 17 00:00:00 2001 From: Stewart X Addison Date: Tue, 7 May 2024 18:21:09 +0100 Subject: [PATCH 2/4] Updates following retrospective meeting (Round 1) Signed-off-by: Stewart X Addison --- .github/ISSUE_TEMPLATE/release-checklist.md | 42 +++++++++++++-------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/release-checklist.md b/.github/ISSUE_TEMPLATE/release-checklist.md index b671ba2..cbb25ac 100644 --- a/.github/ISSUE_TEMPLATE/release-checklist.md +++ b/.github/ISSUE_TEMPLATE/release-checklist.md @@ -23,17 +23,19 @@ Everyone participating in a release, including the release champion are requeste ------- -### Two Weeks Prior To Release +### 2 Weeks Prior To Release - [ ] **Release Champion named** whose responsibility is to ensure every item in this checklist gets completed - [ ] **Release Checklist Created** Create this issue to track the release and the preparation tasks. - [ ] **Identify Expected Release Versions** - Find out the version numbers from [here](https://www.java.com/releases/) - - [ ] **Notify release branching of build repositories** : [Slack message, branching build repositories](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#branching-message-for-build-related-repositories) - [ ] **Create build repositories release Branches** : [Create build repository release branches](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#create-release-branch-on-below-repositories) - - [ ] **Identify the aqa branch name for the upcoming release (Note, April and October PSU updates generally use same branch as the March/September new releases** +### 1-1½ weeks prior to release + +1½ weeks would typically mean running on the Friday so the dry-run results are available on the Monday before release week. + Ensure ALL nodes online prior to running these following TC steps: - [ ] TC: Run the DeleteJCKMultiNode process cleaning job on all ci.role.test nodes, to remove any now redundent jck-versions, to ensure healthy state, verify all nodes successful: https://ci.eclipse.org/temurin-compliance/job/DeleteJCKMultiNode - [ ] TC: Run the ProcessCheckMultiNode process cleaning job on all ci.role.test nodes, to ensure healthy state, verify all nodes successful: https://ci.eclipse.org/temurin-compliance/job/ProcessCheckMultiNode/build?delay=0sec @@ -46,9 +48,10 @@ Ensure ALL nodes online prior to running these following TC steps: - [ ] **Prepare & Perform Dry Run Of Build & Tests** : [Dry-run](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#auto-way---before-release-week-dry-run-release-test) - [ ] **Triage dry-run TCK job results** - [ ] **Restore aqa-tests release branch testenv.properties JDK_BRANCH values to the "-ga" tag after dry-run has completed** - - [ ] **Perform TCK Auto-manuals on x64Linux for each dry-run version** + - [ ] (Optional) **Perform TCK Auto-manuals on x64Linux for each dry-run version** + +### Thursday or Friday prior to release -### One Week Prior To Release - [ ] **Final Code Freeze Warning** post a message to the build & release slack channels : [Slack message](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#code-freeze-message) After 1 day, then :- @@ -66,12 +69,10 @@ After 1 day, then :- **Wait For All Of The Above To Complete Successfully Before Proceeding!** -- [ ] Log a helpdesk ticket with EF , to get all test materials updated +- [ ] Log a helpdesk ticket with EF , to get all test materials updated and prepare for any manual activities - [ ] TC: Run the ProcessCheckMultiNode process cleaning job on all ci.role.test nodes, to ensure healthy state, verify all nodes successful: https://ci.eclipse.org/temurin-compliance/job/ProcessCheckMultiNode/build?delay=0sec - [ ] TC: Run the Setup_JCK_Run_Multinode job with CLEAN_DIR=true (to purge any old release contents/results) on all ci.role.test nodes, this will extract the jck_run folder with all the temurin.jtx exclude files, verify all nodes successful : https://ci.eclipse.org/temurin-compliance/job/Setup_JCK_Run_Multinode/build?delay=0sec - [ ] Check the nagios server to ensure there are no critical infrastructure issues -- [ ] **Trigger a trial release pipeline dry-run** to ensure less surprises on release day (typically against a milestone build), see here for [details](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#auto-way---before-release-week-auto-test) -- [ ] Confirm successful trial release pipelines and successful jck completion ------- @@ -93,17 +94,27 @@ Release Week Checklist: - [ ] Check the published GA tags are the "expected" tags entered in the aqa-tests release branch testenv.properties. If they are not then update. - [ ] **Check Tags have been Mirrored** [Mirrors](https://ci.adoptopenjdk.net/view/git-mirrors/job/git-mirrors/job/adoptium/). - [ ] **Check "auto-trigger" pipelines or Launch build pipelines** for each version being released. Verify if the release pipline "auto-triggered", if not (maybe expected tag was wrong), then manually launch [(as per release doc](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#steps-for-every-version)) once release tags are available via [launch page](https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/build) in Jenkins. Provide links in this issue to each version's pipeline build(s). There may be multiple pipelines per version if primary and secondary platforms are separated to expedite the release. In some cases, where there are unforeseen configuration or infrastructure issues, reruns may be needed. - - jdk8 pipeline(s): + - LTS jdk8 pipeline(s): - **primary jdk8 pipeline:** - rerun(s): - **secondary jdk8 pipeline:** - reruns(s): - - jdk11 pipeline(s): + - LTS jdk11 pipeline(s): - **primary jdk11 pipeline:** - rerun(s): - **secondary jdk11 pipeline:** - rerun(s): - - jdkxx pipeline(s): + - LTS jdk17 pipeline(s): + - **primary jdk17 pipeline:** + - rerun(s): + - **secondary jdk17 pipeline:** + - rerun(s): + - LTS jdk21 pipeline(s): + - **primary jdk21 pipeline:** + - rerun(s): + - **secondary jdk21 pipeline:** + - rerun(s): + - STS jdkxx pipeline(s): - **primary jdkxx pipeline:** - rerun(s): - **secondary jdkxx pipeline:** @@ -117,10 +128,11 @@ Release Week Checklist: - [ ] **Remind** TCK testers (via Slack comment) to update a TCK triage issue with ownership and machine IP **before** running any interactive/automanual tests. - [ ] **Summarize test results**. Find each launched build pipeline in [TRSS](https://trss.adoptium.net/) to view a summary of test results. Can use the Release Summary Report feature in TRSS to generate a summary of failures, history and possible issues in markup format to be added to this issue as a comment. - [ ] **Triage** each build and test failure in the release summary report (following the [Triage guidelines](https://github.com/adoptium/aqa-tests/blob/master/doc/Triage.md)) and determine blocking or non-blocking. Supply links to triage issues or docs for each version here. - - jdk8 triage summary: - - jdk11 triage summary: - - jdk17 triage summary: - - jdkXX triage summary: + - LTS jdk8 triage summary: + - LTS jdk11 triage summary: + - LTS jdk17 triage summary: + - LTS jdk21 triage summary: + - STS jdkXX triage summary: - [ ] **Fix** blocking failures if they exist and confirm others are non-blocking. - [ ] **Confirm Temurin-compliance items completed**, per platform/version/binaryType - [ ] **Get PMC 'ready to publish' approval**, once no blocking failures exist. From 4ef2ca08e55be0b25472e0d9111575fe1b2a6261 Mon Sep 17 00:00:00 2001 From: Stewart X Addison Date: Tue, 7 May 2024 18:55:10 +0100 Subject: [PATCH 3/4] Updates following retrospective meeting (Round 2) Signed-off-by: Stewart X Addison --- .github/ISSUE_TEMPLATE/release-checklist.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/release-checklist.md b/.github/ISSUE_TEMPLATE/release-checklist.md index cbb25ac..69231cd 100644 --- a/.github/ISSUE_TEMPLATE/release-checklist.md +++ b/.github/ISSUE_TEMPLATE/release-checklist.md @@ -155,6 +155,6 @@ Release Week Checklist: - [ ] **Do all of the above for the jdk8u/aarch32 build: Ensure to specify overridePublishName param** - [ ] **Archive/upload all TCK results** - [ ] **Use EclipseMirror job in the Temurin Compliance jenkins to store a backup** of the release artifacts -- [ ] **Create a draft PR for a release blog post for next release** in the [adoptium.net](https://github.com/adoptium/adoptium.net/pulls) repository and ensure to delegate the task of finalizing and publishing the existing draft PR for this release's blog post. +- [ ] **Create a draft PR for a release blog post for next release** in the [adoptium.net](https://github.com/adoptium/adoptium.net/pulls) repository and ensure to delegate the task of finalizing and publishing the existing draft PR for this release's blog post. (Use [this link](https://openjdk.org/groups/vulnerability/advisories) to get the vulnerability list) - [ ] **Ensure the [adoptium calendar](https://calendar.google.com/calendar/u/0/embed?src=c_56d7263c0ceda87a1678f6144426f23fb53721480b5ff71b073afb51091e5492@group.calendar.google.com) is updated for the next cycle at a minimum** - [ ] **Declare the release complete** and close this issue From 419b02576a538021347de04bcaa5b1ffc3794357 Mon Sep 17 00:00:00 2001 From: Stewart X Addison Date: Wed, 8 May 2024 16:12:07 +0100 Subject: [PATCH 4/4] Split out win32 pipelines and add note on full update to API for release notes Signed-off-by: Stewart X Addison --- .github/ISSUE_TEMPLATE/release-checklist.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/release-checklist.md b/.github/ISSUE_TEMPLATE/release-checklist.md index 69231cd..762a29a 100644 --- a/.github/ISSUE_TEMPLATE/release-checklist.md +++ b/.github/ISSUE_TEMPLATE/release-checklist.md @@ -23,7 +23,7 @@ Everyone participating in a release, including the release champion are requeste ------- -### 2 Weeks Prior To Release +### Two Weeks Prior To Release - [ ] **Release Champion named** whose responsibility is to ensure every item in this checklist gets completed - [ ] **Release Checklist Created** Create this issue to track the release and the preparation tasks. @@ -44,11 +44,11 @@ Ensure ALL nodes online prior to running these following TC steps: - [ ] **Check the nagios server to ensure there are no critical infrastructure issues** Log in to the public [nagios](https://nagios.adoptopenjdk.net/nagios/) server, and check the Problems / Services page. If you do not have access, please request it via an issue in the infrastructure repository. If there are any issues, then please log an issue in the infrastructure repository. - [ ] **Regenerate The Release Build Pipeline Jobs In Jenkins** - - [ ] **Update testenv.properties in the AQA release branch to use the -dryrun-ga branches** ([Sample PR](https://github.com/adoptium/aqa-tests/pull/5202/files)) + - [ ] **Update testenv.properties in the AQA release branch to use the -dryrun-ga branches** ([Sample PR](https://github.com/adoptium/aqa-tests/pull/5202/files)) - [ ] **Prepare & Perform Dry Run Of Build & Tests** : [Dry-run](https://github.com/adoptium/temurin-build/blob/master/RELEASING.md#auto-way---before-release-week-dry-run-release-test) - [ ] **Triage dry-run TCK job results** - [ ] **Restore aqa-tests release branch testenv.properties JDK_BRANCH values to the "-ga" tag after dry-run has completed** - - [ ] (Optional) **Perform TCK Auto-manuals on x64Linux for each dry-run version** + - [ ] (Optional based on perceived risk with any machine updates) **Perform TCK Auto-manuals on one platform** ### Thursday or Friday prior to release @@ -87,6 +87,7 @@ Release Week Checklist: - [ ] Create the Github issues for the [Adoptium public retro](https://github.com/adoptium/temurin/issues/new?assignees=&labels=Retrospective&projects=&template=retrospectives.md&title=General+Retrospective+for+%3Cmonth%3E+%3Cyear%3E+Releases) in the adoptium/temurin repo - [ ] Create the retrospective issue for the Temurin Compliance project - [ ] Update the links on the slack channel for the release status and retrospective issues. +- [ ] Remove x32Windows from release-openjdk{8,11,17}-pipeline (they will be manually triggered later as secondary pipelines) #### Release Day Onwards @@ -97,17 +98,19 @@ Release Week Checklist: - LTS jdk8 pipeline(s): - **primary jdk8 pipeline:** - rerun(s): - - **secondary jdk8 pipeline:** + - **secondary jdk8 pipeline (inc. Win32):** + - reruns(s): + - **aarch32-jdk8u pipeline:** - reruns(s): - LTS jdk11 pipeline(s): - **primary jdk11 pipeline:** - rerun(s): - - **secondary jdk11 pipeline:** + - **secondary jdk11 pipeline (inc. Win32):** - rerun(s): - LTS jdk17 pipeline(s): - **primary jdk17 pipeline:** - rerun(s): - - **secondary jdk17 pipeline:** + - **secondary jdk17 pipeline (inc. Win32):** - rerun(s): - LTS jdk21 pipeline(s): - **primary jdk21 pipeline:** @@ -137,7 +140,7 @@ Release Week Checklist: - [ ] **Confirm Temurin-compliance items completed**, per platform/version/binaryType - [ ] **Get PMC 'ready to publish' approval**, once no blocking failures exist. - [ ] **Generate The Release Notes Per JDK Version **, ( Use https://ci.adoptium.net/job/build-scripts/job/release/job/create_release_notes/ ) and publish them with the [release tool](https://ci.adoptium.net/job/build-scripts/job/release/job/refactor_openjdk_release_tool/) using `UPSTREAM_JOB_NAME` of `create_release_notes` and the appropriate JOB NUMBER -- [ ] **Verify that the release notes are live (as [we've had problems with this](https://github.com/adoptium/temurin/issues/28#issuecomment-2077786176)) and remediate if required. +- [ ] **Verify that the release notes are live - may require a full update on the API [ we've had problems with this recently](https://github.com/adoptium/temurin/issues/28#issuecomment-2077786176)) and remediate if required. - [ ] **Publish the release** (run the restricted access [release tool job](https://ci.adoptopenjdk.net/job/build-scripts/job/release/job/refactor_openjdk_release_tool/) on Jenkins) ( also publish release notes ) - [ ] **Consider updating the API** as required via the relevant parts of [the Adoptium API model constants](https://github.com/adoptium/api.adoptium.net/blob/main/adoptium-models-parent/adoptium-api-v3-models/src/main/kotlin/net/adoptium/api/v3/models/Versions.kt). - [ ] **Verify binaries published successfully** to github releases repo and website (_automate_*, this could also be an automated test)