v3.240904

What's Changed

• fix pipeline by @aspiringmind-code in #8637
• Fixing registry caching for monit image by @novicecpp in #8634
• make runContainer.sh help works by @novicecpp in #8639
• Add bash-completion to pypi container by @novicecpp in #8641
• add instuctions for TW logstash config by @belforte in #8642
• monit - set CMD to bash by @mapellidario in #8640
• monit - runcontainer improved by @mapellidario in #8643
• jenkins test - add client validation test for crab recover by @mapellidario in #8087
• add comments by @belforte in #8633
• fix .gitlab-ci.yml comment by @novicecpp in #8649
• Change CI variables name for triggering the pipelines by @novicecpp in #8653
• monitoring image - some new additions by @mapellidario in #8656
• stop using username_CRAB for myproxy. Fix #8655 by @belforte in #8657
• No locations found should result in SubmissionRefused by @belforte in #8662
• fix logging format by @belforte in #8668
• add test14 by @novicecpp in #8669
• fix test14 by @novicecpp in #8670
• Fix rucio cleanup.sh and add SUBMITREFUSED to ST by @novicecpp in #8671
• spark monitoring cronjobs - a small refactor by @mapellidario in #8672
• use CRAB_Requested* ads in resubmission. Fix #8654 by @belforte in #8674
• suspend tape recall rule before deleting. Fix #8647 by @belforte in #8652
• Refactor process controller scripts by @novicecpp in #8658
• add status subresource to RESTTask and use it in task_info.js by @belforte in #8680
• fix 8658: manage.py which cause manage.sh always run on debug mode by @novicecpp in #8682
• bump WMCore to 2.3.rc2 by @belforte in #8687
• Task scheduling by @aspiringmind-code in #8676

Full Changelog: v3.240819...v3.240904

v3.240809.p1

simply bump WMCore to 2.3.6rc2 to pick up fix to voms-proxy-info concurrency issue

What's Changed

• bump WMCore to 2.3.6rc2 (#8687) by @belforte in #8688

Full Changelog: v3.240809...v3.240809.p1

v3.240819

This is mostly a way to update crabtwfilebeat:v3.latest container to fix CheckTapeRecall.
It also brings in:

• faster container building
• changes in new v2 condor bindings usage (default is to use v1)
• small changes to TW for paving the way to v2 binding which needs more testing in preprod

What's Changed

• use TaskWorkerConfig to enable use of V2 bindings by @belforte in #8595
• improve message. Fix #7857 by @belforte in #8608
• do not use deprecated schedd.transacion. Fix #8597 by @belforte in #8610
• workaround to fix #8604 (for v2 bindings) by @belforte in #8612
• Change xquery to query 8613 by @belforte in #8615
• add SUBMITREFUSED to task statuses to monitor. Fix #8620 by @belforte in #8621
• cmsrun - adapt to deprecated -p parameter by @mapellidario in #8623
• trim DBSDataDiscovery logging and add dataset name by @belforte in #8629
• Kaniko to DinD & implement registry caching #8522 by @jannakam in #8627
• rename classad TaskType to CRAB_DAGType. Fix #8488 by @belforte in #8616
• Fix small bugs by @belforte in #8632

New Contributors

• @jannakam made their first contribution in #8627

Full Changelog: v3.240809...v3.240819

v3.240809

Bump WMCore version to 2.3.3, small fixes and upgrades in TW/Pub.

What's Changed

• publisher monit - updated grok for PUBSTART by @mapellidario in #8580
• tw filebeat - new docker image by @mapellidario in #8576
• bump wmcore to 2.3.3 in pypi images and add deprecate notice to requirements.txt by @novicecpp in #8581
• fix #8585 by @belforte in #8586
• Adding site error code 10040 as recoverable error by @aspiringmind-code in #8596
• update HTC to 23.9.6 in TW pypi image by @belforte in #8605

Full Changelog: v3.240731...v3.240809

v3.240716.patch1

Cherry pick #8581 to the v3.240716 to bump WMCore version to 2.3.3. Fixing "Unsupported architecture el9" error.

v3.240731

This tag mainly brings in :

• a new task status, to differentiate between tasks submission failures that are caused by user problems and by CRAB system problems: Add submitrefused 7989 by @belforte in #8564
• a fix about monitoring in crabserver REST: Add monitoring script back to REST's PyPI image by @novicecpp in #8569

What's Changed

Moreover, this tag brings in the new CRABUtils/TaskUtils.py module

• add CRABUtils/TaskUtils.py by @belforte in #8557
• use common functions from CRABUtils/TaskUtils fix #7671 by @belforte in #8570

an improvement of the resilience of FTS_Transfers to FTS errors

• put all calls to fts3.get_job_status in try/except Fix #8556 by @belforte in #8558

an improvement in Publisher pypi image:

• Fixing PB entrypoint by @novicecpp in #8562

and some other changes

• Small fixes for rucio publisher by @belforte in #8566
• Add vchakrav remote repository to addGH.sh by @aspiringmind-code in #8551
• More verbose for cicd/rucio/cleanup.py by @novicecpp in #8565
• Fix TW config section does not exist when hardcode some values in pdb mode by @novicecpp in #8552
• [CI] fix SUBMIT_STATUS_TRACKING by @novicecpp in #8553
• [CI] Support CRABClient GH by @novicecpp in #8554
• [CI] Add Vijay dev env by @novicecpp in #8555
• Update .gitignore by @novicecpp in #8560

Full Changelog: v3.240716...v3.240731

v3.240716

This tag brings in change to the tape recall policy, in particular, tape recall expire 4 days after recall is completed and there's a limit on how much every user can recall.

What's Changed

• Limiting Tape recall days to now+4 days by @aspiringmind-code in #8548
• add per-user tape recall policy. Fix #8354 by @belforte in #8544

Full Changelog: v3.240709...v3.240716

v3.240709

Regular update production image.
In addition, congratulations to our new colleague, Vijay Chakravarty @aspiringmind-code, for the first PR.

What's Changed

• print time zone in logs and when next test iteration will be by @belforte in #8534
• Add ChildWorker module by @novicecpp in #8527
• fix rucio.cfg . Fixes #8535 by @belforte in #8536
• Fix TRACE in script and reduce verbosity of updateDatafiles.sh by @novicecpp in #8538
• Remove HTCondorUtils.py #8339 by @aspiringmind-code in #8531
• bump HTC bindings to 23.9.0a3 by @belforte in #8547
• Use simple py script as the entrypoint of TW process by @novicecpp in #8541

New Contributors

• @aspiringmind-code made their first contribution in #8531

Full Changelog: v3.240627...v3.240709

v3.240627

this release brings

improvements to pipeline and fix to Publisher_rucio

What's Changed

• Fix various CIs by @novicecpp in #8514
• reduce to minimum taskworker dependencies. Fix #8518 by @belforte in #8520
• Remove htc from rest 8338 by @belforte in #8523
• Publisher rucio to check block completion 8491 by @belforte in #8526
• store also scope in tm_dbs_blockname. Quick fix for #8525 by @belforte in #8530
• Adatp Pub_rucio to scope:name in dbs_blockname by @belforte in #8533

Full Changelog: v3.240618...v3.240627

Validation: https://gitlab.cern.ch/crab3/CRABServer/-/pipelines/7650298

v3.240618

this is the first attempt at tagging Master branch after merging with Modernize_HTC
this TW worked in test2 using ./start.sh -g : #8513

What's Changed

• [PYPI] bump htcondor in pypi image to 23.0.12 by @novicecpp in #8492
• Merge modernize htc by @belforte in #8503
• Fix pre job by @belforte in #8507
• Fix #8509 by @belforte in #8511
• add /etc/condor/condor_config to TW Fix #8497 by @belforte in #8499
• second attempt to fix #8509 by @belforte in #8512
• monit - use taskworker pypi image as base docker image for monitoring scripts by @mapellidario in #8505

Full Changelog: v3.240607...v3.240618

Validation OK on cmsweb-testbed/crab-preprod-tw01
https://gitlab.cern.ch/crab3/CRABServer/-/jobs/40241858
#8515
#8516
#8517