v3.240515

Changes to REST to support use of userInputFiles configuration parameter as a "fileMask" on the input dataset

Brings in also improvements to Gitlab-CI pipeline

What's Changed

• make REST accept both inputDataset and userInputFiles at same time fix #8372 by @belforte in #8395

• handle inputdata='' in REST like if it were missing by @belforte in #8399

• allow for inputdata arg to be missing. Fix #8404 by @belforte in #8407

• Deploy both TW and Publisher with Gitlab-CI by @novicecpp in #8397

Full Changelog: v3.240510...v3.240515

v3.240510

Add automatic handling of failed DBS migration and restructuring of Publisher code to put functions common to Publisher_schedd and Publisher_rucio in Publisher(Dbs)Utils

Also a minor fix to REST

List of changes:

What's Changed

• Fix failed migration handling 8382 by @belforte in #8386
• fix "created" value in FMD retriecal. Fix #8388 by @belforte in #8390
• move common code from PubMaster* to PubUtils. Fix #8387 by @belforte in #8392

Full Changelog: v3.240508...v3.240510

Validation: #8393

v3.240508

What's Changed

Tag created mainly for new env variable in jobs for frontier monitoring and WMCore version 2.3.3

• define env.var. FRONTIER_ID Fix #8366 by @belforte in #8367
• requirements - wmcore upgraded to 2.3.3 by @mapellidario in #8383

but this tag brings also many other changes

Publisher:

• Publisher support for PyPI image by @novicecpp in #8368
• Deal with failed migrations 8244 by @belforte in #8378 (still buggy)

Corrupted files reporting (not part of CRAB services proper)

• Update ProcessBadFilesList.py require at least 3 failure reports by @belforte in #8379
• do not crash ProcessBadList.py in case of no reports by @belforte in #8365
• no need for a flag in /etc anymore. Use -p(arent) in gfal-copy by @belforte in #8369

TaskWorker

• only user reliable RSEs for tape recall. Fix #8356 by @belforte in #8381

Oracle

• Oracle - many new things cumulated over many months by @mapellidario in #8326

Full Changelog: v3.240426...v3.240508

v3.240426

• small fixes for CI/CI pipeline
• make DBSDataDiscovery test work and save output to JSON file
• add fix by David Lange to speedup parent finding (x100)

List of commits:

• attempt to streamline parent file discovery (#8352) by @davidlange6
• make test of DBSDataDiscovery work again and add printout of output (#8362) by @belforte
• allow to fall back proxy to installed cert in runner machine (#8359) by @novicecpp

List of difference: v3.240423...v3.240426

v3.240423

What's Changed

• Remove T2_DE_DESY hardcode and use generic RucioException for add_replicas() instead by @novicecpp in #8325
• Clean up cmsbot's Rucio storage space by @novicecpp in #8327
• stop using AuthenticatedSubprocess. Fix #8277 by @belforte in #8331
• PoC CI pipeline on Gitlab to verify PyPI image by @novicecpp in #8328
• use classad.quote, not HTCondorUtils. Fix #8332 by @belforte in #8346
• monit - run monit scripts inside docker containers by @mapellidario in #8194
• Fix deployTW.sh script, fix CI job tags to crab3 and crab3-shell by @novicecpp in #8348

Full Changelog: v3.240325...v3.240423

v3.240325

fix previous broken release v3.240318 via #8302

Validation
ran 3 Jenkins tests at same time

• CV + CCV + ST #8304
• CV+ CCV +ST #8305
• CCV + ST #8306

all OK.

v3.240318 - BROKEN

should work on alma9 schedulers

• use classad.parseAds - fix #8285 (#8296)
• avoid empty lines in finished_jobs/ files. For #8285 (#8287)
• make dag_bootstrap.sh support alma9. Fix #8284 (#8286)

This release is broken, see #8299 (comment)

v3.240313

Alma9 support (server), new production-ready PyPI, WMCore version to 2.3.2rc3 to support alma9 job, and small fixes.

What's Changed

• PYPI proof-of-concept by @novicecpp in #8088
• skip FileTransferEvent msgs in cache_status.py by @belforte in #8262
• correct () in logging statement. Fix #8259 by @belforte in #8260
• port JobAutoTuner to py3 and keep only overflow. by @belforte in #8268
• Make task proc wrapper alma9 compatible for 8256 by @belforte in #8271
• make dag_bootstrap_startup work on alma9. For #8256 by @belforte in #8270
• Bring back ./start.sh -g -d by @novicecpp in #8254
• Fix bug from PR#8254 by @novicecpp in #8275
• Bump cmsweb TW to 20230901-stable and WMCore to 2.3.2rc3 by @novicecpp in #8278

Full Changelog: v3.240219...v3.240313

v3.240219

fix bug in latest commit/tag

• fix name of rucio creds in config. Fix #8245

difference from lates tag: v3.240216...v3.240219

Validation: #8249

v3.240216

Small fixes and has reporting of ReadBranches based on secret file available on all Schedulers. Aimed for deployment in production.

List of commit:

• New format for monit secrets fix 8231 (#8234)
• protect against empty/malformed json reports (#8237)
• handle RSE not found in getWritePfn fix #7107 fix #8230 (#8233)
• report ReadBranches to crab-test index in MONIT. Fix #7932 (#8214)

Full differences from last version: v3.240201...v3.240216

Validation: #8239