Bugzilla to GitHub Summary

[makubacki]

Overview

❗The Tianocore Bugzilla to GitHub issue migration occurred on 12/19/2024 from ~14:30 PST to ~20:30 PST.

The Tianocore project is migrating from Bugzilla to GitHub issues. Many of the details are covered in the RFC - #5926.

All of the attachments to Bugzilla issues is uploaded to the tianocore/user-attachments repository. These are reference as raw files from links in GitHub issues.

This page summarizes some details about the migration process for reference.

Script Details

The original tool used to programmatically migrate issues is berestovskyy/bugzilla2github.

Ultimately, many adjustments and enhancements were made in a Tianocore customized version hosted in the bz2gh_tianocore branch of makubacki/bugzilla2github.

Main changes include:

• Adoption of Python argparse for CLI arguments
• Adoption of Python logging for output
• Algorithm changes:
  • Always take the next available GitHub ID (no blank issues)
  • Do not update existing GitHub issue comments (one shot transfer)
• Compatibility fixes for authorizing against current GitHub REST API requirements
• Customization for Tianocore GitHub project labels (and package details)
• Customization for preserving Tianocore-specific Bugzilla issue information
• Customized Bugzilla <-> GitHub user mapping based on custom scripts for the Tianocore project
• Introduction of journal during issue submission to GitHub to resume from last issue write
• Introduction of stat tracking with stat table printed after script run
• Modernization of some code up to Python 3.12
• Output of GitHub REST status to understand GitHub REST API usage per issue submitted
• Support for routing issues to multiple GitHub orgs (tianocore and tianocore-docs) and repos within those orgs based on Bugzilla issue product and component information
• The ability to download Bugzilla attachments using the Bugzilla REST API
  • Storage of attachments organized by organization/repo/BZ-<BZ_ATTACHMENT_ID>-<BZ_ATTACHMENT_FILENAME>
• Updates to make Bugzilla XML parsing more robust

Data Source Details

The Bugzilla data was acquired manually by performing the following steps:

1. Start a new "Simple Search" on Tianocore Bugilla
2. Gather all open issues
   
   • Click "See all search results for this query."
     
   • At the bottom of the search page, click the "XML" orange button.
     
   • When the XML document opens in the Web browser, right-click, "Save file as...", "tianocore_all_open.xml"
3. Get all closed issues

   • Note: Due to the high number of closed issues, Bugzilla fails to render the XML file for a complete search. Therefore, the results are limited to Bugzilla IDs in groups of 1,000.

     

   • Click "See all search results for this query."

4. Paginate closed issues
   • Copy the URL (it should be https://bugzilla.tianocore.org/buglist.cgi?bug_status=__closed__&limit=0&no_redirect=1&order=priority%2Cbug_severity&query_format=specific)
   • Modify the URL to request only Bugzilla IDs [0, 1000): https://bugzilla.tianocore.org/buglist.cgi?bug_status=__closed__&limit=0&no_redirect=1&order=priority%2Cbug_severity&query_format=specific&f2=bug_id&o2=greaterthaneq&v2=0&f1=bug_id&o1=lessthan&v1=1000
     • Note: This is accomplished by appending &f2=bug_id&o2=greaterthaneq&v2=0&f1=bug_id&o1=lessthan&v1=1000 to the original query URL
   • At the bottom of the search page, click the "XML" orange button.
   • When the XML document opens in the Web browser, right-click, "Save file as...", "tianocore_closed_0_1000.xml"
5. Repeat Step 4 with the following queries, adjusting the file name saved each time.
   • tianocore_closed_1000_2000.xml - https://bugzilla.tianocore.org/buglist.cgi?bug_status=__closed__&f1=bug_id&f2=bug_id&limit=0&no_redirect=1&o1=lessthan&o2=greaterthaneq&order=priority%2Cbug_severity&query_format=specific&v1=2000&v2=1000
   • tianocore_closed_2000_3000.xml - https://bugzilla.tianocore.org/buglist.cgi?bug_status=__closed__&f1=bug_id&f2=bug_id&limit=0&no_redirect=1&o1=lessthan&o2=greaterthaneq&order=priority%2Cbug_severity&query_format=specific&v1=3000&v2=2000
   • tianocore_closed_3000_4000.xml - https://bugzilla.tianocore.org/buglist.cgi?bug_status=__closed__&f1=bug_id&f2=bug_id&limit=0&no_redirect=1&o1=lessthan&o2=greaterthaneq&order=priority%2Cbug_severity&query_format=specific&v1=4000&v2=3000
   • tianocore_closed_4000_5000.xml - https://bugzilla.tianocore.org/buglist.cgi?bug_status=__closed__&f1=bug_id&f2=bug_id&limit=0&no_redirect=1&o1=lessthan&o2=greaterthaneq&order=priority%2Cbug_severity&query_format=specific&v1=5000&v2=4000

That is the process used to get the files used as the data source, uploaded to the final_xmls directory of the bugzilla2github fork.

Script Invocation

For an actual run of the script that pushes issues to GitHub, it was executed as follows:

> python bugzilla2github \
        -x ./xmls/<filename>.xml \
        -j tianocore-bz2gh-journal.txt \
        -t <github-token> \
        -a <bugzilla-web-api-key \
        --beta \
        --force

Execution Details

• Total Issues: 4,765
• Total Open: 1,236
• Total Closed: 3,529

---

• Total Attachments: 1,413

---

• Issues with Attachements: 808 (17%)
• Issues with Assignees: 4,171 (88%)

---

Open Issues Per Repo

Repo	Issues
tianocore-docs/edk2-BuildSpecification	2
tianocore-docs/edk2-CCodingStandardsSpecification	6
tianocore-docs/edk2-DscSpecification	6
tianocore-docs/edk2-InfSpecification	3
tianocore/edk2	1,003
tianocore/edk2-libc	14
tianocore/edk2-platforms	168
tianocore/edk2-test	15
tianocore/tianocore.github.io	19

Closed Issues Per Repo

Repo	Issues
tianocore-docs/edk2-BuildSpecification	23
tianocore-docs/edk2-CCodingStandardsSpecification	10
tianocore-docs/edk2-DecSpecification	3
tianocore-docs/edk2-DscSpecification	7
tianocore-docs/edk2-FdfSpecification	13
tianocore-docs/edk2-InfSpecification	11
tianocore-docs/edk2-MinimumPlatformSpecification	2
tianocore-docs/edk2-TemplateSpecification	1
tianocore-docs/edk2-UniSpecification	4
tianocore/edk2	2,984
tianocore/edk2-libc	39
tianocore/edk2-platforms	290
tianocore/edk2-test	103
tianocore/tianocore.github.io	39

[mraiser]

If the migration is over, can you please reactivate TianoCore Bugzilla?
(https://bugzilla.tianocore.org/editparams.cgi?section=general)
by clearing the shutdownhtml parameter.

[kevindavisinsyde]

Why do you need access to the bugzilla database? I’m not sure the plan was to leave it enabled long term. Thanks, Kevin

…

_______________________________________________________________ On Dec 25, 2024, at 9:02 AM, mraiser ***@***.***> wrote:  If the migration is over, can you please reactivate TianoCore Bugzilla? (https://bugzilla.tianocore.org/editparams.cgi?section=general) by clearing the shutdownhtml parameter. — Reply to this email directly, view it on GitHub<#6544 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIA55THRNTXVUQ2XAFCMC2T2HK3HFAVCNFSM6AAAAABTXCW3HKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRWGM4DQNI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mraiser]

Trying to build firmware updates for Jetson Orin Nano on NixOS, which apparently requires artifacts in the bugzilla database.
https://github.com/anduril/jetpack-nixos/

Specifically it is looking for:
https://bugzilla.tianocore.org/attachment.cgi?id=1436

[mdkinney]

All attachments from TianoCore Bugzilla were migrated to https://github.com/tianocore/user-attachments

Searching for 1436 in that repo results in:

https://github.com/tianocore/user-attachments/blob/main/tianocore/edk2/BZ-1436-4118.patch

[mraiser]

Looks like they have your old url hard-coded. Sounds like this is a them problem and not a you problem. Thanks!
https://github.com/search?q=repo%3Aanduril%2Fjetpack-nixos%20https%3A%2F%2Fbugzilla.tianocore.org&type=code

[alkasm]

This is now fixed in jetpack-nixos.