[Master issue] Harden data pipeline infrastructure with extra goodies and cleanup like retries, unit testing, gtest junit xml support, and documentation

[tt-rkim]

Transferring cleanup list from #9319

Cleanup tasks

• More defensive programming / re-tries for missing logs / copied jobs for workflow run retries and artifact downloading since we can hang there / error out there because a job doesn't exist anymore
• Create unit tests and related test infrastructure for the data collection tools so we can have better refactoring flexibility later #12885
• gtest information support
• Fixing errors in the data:
  • Handle data collection cases where artifacts aren't carried over between attempts  #12840
  • re-runs can create new submission time stamps, but we still look for the earliest job in the earliest run, causing negative queue times
• Mypy / pydantic / strict type checking because we have so many random fields
• Investigate and fix high incidence of CI/CD data generation failure with JSON blob generation exception #12271
• Investigate using qooman/actions--context to get github_job_id instead of using the cursed uuid infrastructure #11711
• Upload benchmarking data for single-card demos to sftp server #10691
• Restrict workflow data collection to main branch for now and submit a data team ticket to have a branch field, and then use it in our collection process #11469
• Save environment data and upload benchmark data even when CI jobs fail (single-card demos, t3k demos) #11614
• Add git_branch_name field to pipeline for CI/CD once data team adds to schema #11704
• Update event data to use correct original event for a workflow_call instance rather than GITHUB_EVENT_NAME for cicd data #10533
• Add git_branch_name and github_job_run_link to environment.csv for benchmarks #10680
• De-duplicate environment CSV generation code so that we can have CI/CD information generation in one place #11134
• Align device_info fields with customer team's data #11137
• Add Blackhole as an option to infra/data_collection scripts #11238
• Cache environment for ci/cd infra workflow so it doesn't install every time #12877
• Add GitHub link to pipeline/job so it shows up in CI/CD data #12878
• Add other pipelines, such as T3K/TG/TGG, to ci/cd collection set #12879
• Extract test report files out of test container for Docker jobs so that we can upload them, as currently none are being uploaded after the Docker changes #12891
• Add handling in logs + artifacts download script for data collection for logs that don't exist #12966

[tt-rkim]

Looks like timeouts during pytests can cause no data to be loaded into the junit xml

Nahhhhh this is a different, check #12799 from now on

[tt-rkim]

Looks like re-runs don't repeat artifacts: https://github.com/tenstorrent/tt-metal/actions/runs/10908473854/

tried by: https://github.com/tenstorrent/tt-metal/actions/runs/10910636802/job/30281509160