forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
2194 lines (1643 loc) · 84.7 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v7r0-pre4]
FIX: removed __init__ files from test directories to avoid importing
NEW: Change the schema of the Resources description in CS, use new Resources helper to
interpret this information
NEW: Base RSS resources representation on the new CS Resources schema
*Core
FIX: multiple fixes to remove pylint complaints
CHANGE: Moved log coloring to utilities so it can be used consistently across DIRAC
FIX: Workflow package - fix wild wild card imports resulting in circular imports
NEW: MySQL - assign one MySQL connection per thread
FIX: LocalConfiguration - if short option is not set, do not print "-" in help message,
closes #1371
FIX: dirac-install - update creation of bashrc/cshrc to allow multiplatform installations
*Accounting
FIX: AccountingDB - align properly days with MySQL bucketing. Closes #1219
*Framework
FIX: ProxyDB - prevent duplicate key errors on writing VOMSProxies to DB. Closes #1228
BUGFIX: NotificationDB - missing "," in SQL, closes #1373
*Configuration
NEW: Resources helper class to work with the new /Resources structure according to RFC #5
NEW: dirac-configuration-convert-resources-schema - command to convert old /Resources schema
to the new one
*Interfaces
CHANGE: Job.py - setPlatform renamed to setSubmitPools
*DMS
CHANGE: MigrationMonitoringAgent - removed obsoleted
NEW: FileCatalogClient - getFileUserMetadata to get both file and directory metadata for a given file
CHANGE: FileCatalogClient - list meta data sets
FIX: FileCatalog - extra check while new user registration for already existing entry
BUGFIX: ReplicaManager - pass catalog argument in putAndRegister to registerFile call, closes #1369
*WMS
CHANGE: JobScheduling - is now extensible. Added unit test
NEW: SiteDirector - define which extensions pilot should install in the options, do not take from globals
*Transformation
NEW: TaskManager - if a site is specified in the job definition, it is now taken into account
when submitting the task
NEW: Speeding up the getTransformationSummary call, using an UpdateTransformationCounters agent
FIX: Multiple fixes to please pylint
FIX: TransformationDB - removed bad index on TaskID
*Resources
FIX: ComputingElement - properly check if the pilot proxy has VOMS before adding it to the
payload when updating it
NEW: StorageElement - enable Storage Element proxy configuration by protocol name
CHANGE: glexecComputingElement - updated to reflect new glexec evolution, added DIRAC_GLEXEC
to the environment when running under glexec
[v6r8p8]
*RSS
BUGFIX: dirac-rss-setup - added missing return of S_OK() result
[v6r8p7]
NEW: Applied patches from v6r7p24
*DMS
BUGFIX: LcgFileCatalogClient - bug in addFile()
*RSS
BUGFIX: fixed script dirac-rss-set-token, broken in the current release.
NEW: Statistics module - will be used in the future to provide detailed information
from the History of the elements
[v6r8p6]
NEW: Applied patches from v6r7p23
*Transformation
FIX: TaskManager - allow prepareTransformationTasks to proceed if no OutputDataModule is defined
FIX: TransformationDB - remove INDEX(TaskID) from TransformationTasks. It produces a single counter
for the whole table instead of one per TransformationID
*WMS
FIX: WMSUtilities - to allow support for EMI UI's for pilot submission we drop support for glite 3.1
[v6r8p5]
NEW: Applied patches from v6r7p22
*RSS
CHANGE: removed old tests and commented out files
*WMS
FIX: PoolXMLCatalog - proper addFile usage
*Transformation
CHANGE: TransformationAgent - clear replica cache when flushing or setting a file in the workdirectory
[v6r8p4]
*Transformation
FIX: The connection to the jobManager is done only at submission time
FIX: Jenkins complaints fixes
*WMS
BUGFIX: JobDB - CPUtime -> CPUTime
FIX: Jenkins complaints fixes
[v6r8p3]
*DMS
BUGFIX: LcgFileCatalogClient
[v6r8p2]
*DMS:
FIX: LcgFileCatalogClient - remove check for opening a session in __init__ as credentials are not yet set
*Transformation
CHANGE: reuse RPC clients in Transformation System
[v6r8p1]
*Core
FIX: dirac-deploy-scripts - restored regression w.r.t. support of scripts starting with "d"
*DMS
BUGFIX: LcgFileCatalogClient - two typos fixed
[v6r8]
CHANGE: Several fixes backported from the v7r0 integration branch
*Core
CHANGE: DictCache - uses global LockRing to avoid locks in multiprocessing
FIX: X509Chain - proxy-info showing an error when there's no CS
*DMS
FIX: TransferAgent - inside loop filter out waiting files dictionary
BUGFIX: dirac-admin-allow-se - there was a continue that was skipping the complete loop for
ARCHIVE elements
NEW: LcgFileCatalogClient - test return code in startsess lfc calls
*WMS:
FIX: OptimizerExecutor, InputData, JobScheduling - check that site candidates have all the
replicas
*RSS:
BUGFIX: ResourceStatus, RSSCacheNoThread - ensure that locks are always released
*Transformation
FIX: TaskManager - site in the job definition is taken into account when submitting
NEW: Transformation - get the allowed plugins from the CS /Operations/Transformations/AllowedPlugins
FIX: ValidateOutputDataAgent - self not needed for static methods
[v6r7p25]
*Core
CHANGE: X509Chain - in getCredentials() failure to contact CS is not fatal,
can happen when calling dirac-proxy-init -x, for example
[v6r7p24]
*DMS
NEW: FileCatalog - added getFilesByMetadataWeb() to allow pagination in the Web
catalog browser
*WMS
CHANGE: WMSAdministrator, DiracAdmin - get banned sites list by specifying the status
to the respective jobDB call
[v6r7p23]
*Transformation
BUGFIX: TransformationDB - badly formatted error log message
*RMS
CHANGE: RequestDBMySQL - speedup the lookup of requests
*WMS
BUGFIX: dirac-dms-job-delete - in job selection by group
*DMS
FIX: LcgFileCatalogClient - getDirectorySize made compatible with DFC
BUGFIX: LcgFileCatalogClient - proper call of __getClientCertInfo()
[v6r7p22]
*Transformation
CHANGE: InputDataAgent - treats only suitable transformations, e.g. not the extendable ones.
CHANGE: TransformationAgent - make some methods more public for easy overload
[v6r7p21]
*Core
FIX: Shifter - pass filePath argument when downloading proxy
[v6r7p20]
*DMS
CHANGE: StrategyHandler - move out SourceSE checking to TransferAgent
CHANGE: ReplicaManager, InputDataAgent - get active replicas
FIX: StorageElement, SRM2Storage - support for '*Access' statuses, checking results
of return structures
*RSS
NEW: set configurable email address on the CS to send the RSS emails
NEW: RSSCache without thread in background
FIX: Synchronizer - moved to ResourceManager handler
[v6r7p19]
*DMS
BUGFIX: ReplicaManager - in putAndRegister() SE.putFile() singleFile argument not used explicitly
[v6r7p18]
*WMS
FIX: StalledJobAgent - do not exit the loop over Completed jobs if accounting sending fails
NEW: dirac-wms-job-delete - allow to specify jobs to delete by job group and/or in a file
FIX: JobManifest - If CPUTime is not set, set it to MaxCPUTime value
[v6r7p17]
*Resources
FIX: SRM2Storage - treat properly "22 SRM_REQUEST_QUEUED" result code
[v6r7p16]
*DMS
FIX: StrategyHandler - do not proceed when the source SE is not valid for read
BUGFIX: StorageElement - putFile can take an optional sourceSize argument
BUGFIX: ReplicaManager - in removeFile() proper loop on failed replicas
*RSS
FIX: SpaceTokenOccupancyCommand, CacheFeederAgent - add timeout when calling lcg_util commands
*WMS
FIX: JobManifest - take all the SubmitPools defined in the TaskQueueAgent
NEW: StalledJobAgent - declare jobs stuck in Completed status as Failed
[v6r7p15]
*Core
BUGFIX: SocketInfo - in host identity evaluation
*DMS
BUGFIX: FileCatalogHandler - missing import os
*Transformation
CHANGE: JobManifest - getting allowed job types from operations() section
[v6r7p14]
*DMS
CHANGE: StorageElementProxy - removed getParameters(), closes #1280
FIX: StorageElementProxy - free the getFile space before the next file
FIX: StorageElement - added getPFNBase() to comply with the interface
*Interfaces
CHANGE: Dirac API - allow lists of LFNs in removeFile() and removeReplica()
*WMS
CHANGE: JobSchedulingAgent(Executor) - allow both BannedSite and BannedSites JDL option
*RSS
FIX: ElementInspectorAgent - should only pick elements with rss token ( rs_svc ).
FIX: TokenAgent - using 4th element instead of the 5th. Added option to set admin email on the CS.
[v6r7p13]
*Core
FIX: Resources - in getStorageElementSiteMapping() return only sites with non-empty list of SEs
*DMS
FIX: StorageElement - restored the dropped logic of using proxy SEs
FIX: FileCatalog - fix the UseProxy /LocalSite/Catalog option
*Transformation
FIX: TransformationDB - use lower() string comparison in extendTransformation()
[v6r7p12]
*WMS
BUGFIX: JobManifest - get AllowedSubmitPools from the /Systems section, not from /Operations
*Core
NEW: Resources helper - added getSites(), getStorageElementSiteMapping()
*DMS
CHANGE: StrategyHandler - use getStorageElementSiteMapping helper function
BUGFIX: ReplicaManager - do not modify the loop dictionary inside the loop
[v6r7p11]
*Core
CHANGE: Subprocess - put the use of watchdog in flagging
[v6r7p10]
*Core
NEW: Logger - added getLevel() method, closes #1292
FIX: Subprocess - returns correct structure in case of timeout, closes #1295, #1294
CHANGE: TimeOutExec - dropped unused utility
FIX: Logger - cleaned unused imports
*RSS
CHANGE: ElementInspectorAgent - do not use mangled name and removed shifterProxy agentOption
[v6r7p9]
*Core
BUGFIX: InstallTools - MySQL Port should be an integer
[v6r7p8]
*Core
FIX: Subprocess - consistent timeout error message
*DMS
NEW: RemovalTask - added bulk removal
FIX: StrategyHandler - check file source CEs
CHANGE: DataIntegrityClient - code beautification
CHANGE: ReplicaManager - do not check file existence if replica information is queried anyway,
do not fail if file to be removed does not exist already.
[v6r7p7]
FIX: Several fixes to allow automatic code documentation
*Core
NEW: InstallTools - added mysqlPort and mysqlRootUser
*DMS
CHANGE: ReplicaManager - set possibility to force the deletion of non existing files
CHANGE: StrategyHandler - better handling of checksum check during scheduling
[v6r7p6]
*Core
FIX: dirac-install - restore signal alarm if downloadable file is not found
FIX: Subprocess - using Manager proxy object to pass results from the working process
*DMS:
CHANGE: StorageElement - removed overwride mode
CHANGE: removed obsoleted dirac-dms-remove-lfn-replica, dirac-dms-remove-lfn
NEW: FTSMonitorAgent - filter out sources with checksum mismatch
FIX: FTSMonitorAgent, TransferAgent - fix the names of the RSS states
*RSS
NEW: ElementInspectorAgent runs with a variable number of threads which are automatically adjusted
NEW: Added policies to force a particular state, can be very convenient to keep something Banned for example.
NEW: policy system upgrade, added finer granularity when setting policies and actions
*WMS
NEW: SiteDirector- allow to define pilot DN/Group in the agent options
CHANGE: JobDescription, JobManifest - take values for job parameter verification from Operations CS section
[v6r7p5]
*Interfaces
BUGFIX: dirac-wms-job-get-output - properly treat the case when output directory is not specified
[v6r7p4]
*Core
FIX: Subprocess - avoid that watchdog kills the executor process before it returns itself
*Framework
BUGFIX: ProxuManagerClient - wrong time for caching proxies
*RSS
FIX: removed obsoleted methods
*DMS
NEW: FileCatalog - added findFilesByMetadataDetailed - provides detailed metadata for
selected files
[v6r7p3]
*DMS
FIX: FTSMonitorAgent - logging less verbose
*Transformation
FIX: TransformationAgent - use the new CS defaults locations
FIX: Proper agent initialization
NEW: TransformationPlaugin - in Broadcast plugin added file groupings by number of files,
make the TargetSE always defined, even if the SourceSE list contains it
*ResourceStatus
FIX: Added the shifter's proxy to several agents
*RMS
FIX: RequestContainer - the execution order was not properly set for the single files
*Framework:
BUGFIX: ProxyManagerClient - proxy time can not be shorter than what was requested
[v6r7p2]
*Core
FIX: dirac-configure - switch to use CS before checking proxy info
*Framework
NEW: dirac-sys-sendmail new command
NEW: SystemAdmininistratorCLI - added show host, uninstall, revert commands
NEW: SystemAdmininistratorHandler - added more info in getHostInfo()
NEW: SystemAdmininistratorHandler - added revertSoftware() interface
*Transformation
FIX: TransformationCleaningAgent - check the status of returned results
[v6r7p1]
*Core
FIX: Subprocess - finalize the Watchdog closing internal connections after a command execution
CHANGE: add timeout for py(shell,system)Call calls where appropriate
CHANGE: Shifter - use gProxyManager in a way that allows proxy caching
*Framework
NEW: ProxyManagerClient - allow to specify validity and caching time separately
FIX: ProxyDB - replace instead of delete+insert proxy in __storeVOMSProxy
*DMS
NEW: FTSMonitorAgent - made multithreaded for better efficiency
FIX: dirac-dms-add-file - allow LFN: prefix for lfn argument
*WMS
NEW: dirac-wms-job-get-output, dirac-wms-job-status - allow to retrieve output for a job group
FIX: TaskQueueDB - fixed selection SQL in __generateTQMatchSQL()
CHANGE: OptimizerExecutor - reduce diversity of MinorStatuses for failed executors
*Resources
FIX: CREAMComputingElement - remove temporary JDL right after the submission
[v6r6p21]
*DMS
BUGFIX: TransformationCleaningAgent - use the right signature of cleanMetadataCatalogFiles() call
[v6r6p20]
*DMS
FIX: RegistrationTask - properly escaped error messages
BUGFIX: DirectoryMetadata - use getFileMetadataFields from FileMetadata in addMetadataField()
NEW: When there is a missing source error spotted during FTS transfer, file should be reset
and rescheduled again until maxAttempt (set to 100) is reached
*WMS
FIX: JobScheduling - fix the site group logic in case of Tier0
[v6r6p19]
*DMS
BUGFIX: All DMS agents - set up agent name in the initialization
*Core
NEW: Subprocess - timeout wrapper for subprocess calls
BUGFIX: Time - proper interpreting of 0's instead of None
CHANGE: DISET - use cStringIO for ANY read that's longer than 16k (speed improvement)
+ Less mem when writing data to the net
FIX: Os.py - protection against failed "df" command execution
NEW: dirac-info prints lcg bindings versions
CHANGE: PlotBase - made a new style class
NEW: Subprocess - added debug level log message
*Framework
NEW: SystemAdministratorIntegrator client for collecting info from several hosts
NEW: SystemAdministrator - added getHostInfo()
FIX: dirac-proxy-init - always check for errors in S_OK/ERROR returned structures
CHANGE: Do not accept VOMS proxies when uploading a proxy to the proxy manager
*Configuration
FIX: CE2CSAgent - get a fresh copy of the cs data before attempting to modify it, closes #1151
FIX: Do not create useless backups due to slaves connecting and disconnecting
FIX: Refresher - prevent retrying with 'Insane environment'
*Accounting
NEW: Accounting/Job - added validation of reported values to cope with the weird Yandex case
FIX: DBUtils - take into account invalid values, closes #949
*DMS
FIX: FTSSubmitAgent - file for some reason rejected from submission should stay in 'Waiting' in
TransferDB.Channel table
FIX: FTSRequest - fix in the log printout
CHANGE: dirac-dms-add-file removed, dirac-dms-add-files renamed to dirac-dms-add-file
FIX: FileCatalogCLI - check the result of removeFile call
FIX: LcgFileCatalogClient - get rid of LHCb specific VO evaluation
NEW: New FileCatalogProxy service - a generalization of a deprecated LcgFileCatalog service
FIX: Restored StorageElementProxy functionality
CHANGE: dirac-dms-add-file - added printout
NEW: FileCatalog(Factory), StorageElement(Factory) - UseProxy flag moved to /Operations and /LocalSite sections
*RSS
NEW: general reimplementation:
New DB schema using python definition of tables, having three big blocks: Site, Resource and Node.
MySQLMonkey functionality almost fully covered by DB module, eventually will disappear.
Services updated to use new database.
Clients updated to use new database.
Synchronizer updated to fill the new database. When helpers will be ready, it will need an update.
One ElementInspectorAgent, configurable now is hardcoded.
New Generic StateMachine using OOP.
Commands and Policies simplified.
ResourceStatus using internal cache, needs to be tested with real load.
Fixes for the state machine
Replaced Bad with Degraded status ( outside RSS ).
Added "Access" to Read|Write|Check|Remove SE statuses wherever it applies.
ResourceStatus returns by default "Active" instead of "Allowed" for CS calls.
Caching parameters are defined in the CS
FIX: dirac-admin-allow/ban-se - allow a SE on Degraded ( Degraded->Active ) and ban a SE on Probing
( Probing -> Banned ). In practice, Active and Degraded are "usable" states anyway.
*WMS
FIX: OptimizerExecutor - failed optimizations will still update the job
NEW: JobWrapper - added LFNUserPrefix VO specific Operations option used for building user LFNs
CHANGE: JobDB - do not interpret SystemConfig in the WMS/JobDB
CHANGE: JobDB - Use CPUTime JDL only, keep MaxCPUTime for backward compatibility
CHANGE: JobWrapper - use CPUTime job parameter instead of MaxCPUTime
CHANGE: JobAgent - use CEType option instead of CEUniqueID
FIX: JobWrapper - do not attempt to untar directories before having checked if they are tarfiles
NEW: dirac-wms-job-status - get job statuses for jobs in a given job group
*SMS
FIX: StorageManagementDB - when removing unlinked replicas, take into account the case where a
staging request had been submitted, but failed
*Resources
NEW: glexecCE - add new possible locations of the glexec binary: OSG specific stuff and in last resort
looking in the PATH
NEW: LcgFileCatalogClient - in removeReplica() get the needed PFN inside instead of providing it as an argument
*TS
CHANGE: Transformation types definition are moved to the Operations CS section
*Interfaces
FIX: Dirac.py - CS option Scratchdir was in LocalSite/LocalSite
FIX: Dirac.py - do not define default catalog, use FileCatalog utility instead
[v6r6p19]
*DMS
BUGFIX: All DMS agents - set up agent name in the initialization
[v6r6p18]
*Transformation
CHANGE: /DIRAC/VOPolicy/OutputDataModule option moved to <Operations>/Transformations/OutputDataModule
*Resources
FIX: ComputingElement - properly check if the pilot proxy has VOMS before adding it to the payload
when updating it
*WMS
BUGFIX: JobSanity - fixed misspelled method call SetParam -> SetParameter
[v6r6p17]
*Transformation
BUGFIX: TransformationAgent - corrected __getDataReplicasRM()
[v6r6p16]
*DMS
FIX: Agents - proper __init__ implementation with arguments passing to the super class
FIX: LcgFileCatalogClient - in removeReplica() reload PFN in case it has changed
[v6r6p15]
*Framework
BUGFIX: ErrorMessageMonitor - corrected updateFields call
*DMS:
NEW: FTSMonitorAgent completely rewritten in a multithreaded way
*Transformation
FIX: InputDataAgent - proper instantiation of TransformationClient
CHANGE: Transformation - several log message promoted from info to notice level
[v6r6p14]
*Transformation
FIX: Correct instantiation of agents inside several scripts
CHANGE: TransformationCleaningAgent - added verbosity to logs
CHANGE: TransformationAgent - missingLFC to MissingInFC as it could be the DFC as well
FIX: TransformationAgent - return an entry for all LFNs in __getDataReplicasRM
*DMS
FIX: TransferAgent - fix exception reason in registerFiles()
[v6r6p13]
*DMS
CHANGE: TransferAgent - change RM call from getCatalogueReplicas to getActiveReplicas.
Lowering log printouts here and there
[v6r6p12]
*DMS
BUGFIX: RemovalTask - Replacing "'" by "" in error str set as attribute for a subRequest file.
Without that request cannot be updated when some nasty error occurs.
[v6r6p11]
*RMS:
BUGFIX: RequestClient - log string formatting
*DMS
BUGFIX: RemovalTask - handling for files not existing in the catalogue
*Transformation
FIX: TransformationManager - ignore files in NotProcessed status to get the % of processed files
*Interfaces
FIX: Fixes due to the recent changes in PromptUser utility
[v6r6p10]
*RMS
FIX: RequestDBMySQL - better escaping of queries
*WMS
FIX: SiteDirector - get compatible platforms before checking Task Queues for a site
[v6r6p9]
*Core
FIX: Utilities/PromptUser.py - better user prompt
*Accounting
NEW: Add some validation to the job records because of weird data coming from YANDEX.ru
*DMS
BUGFIX: ReplicaManager - typo errStr -> infoStr in __replicate()
FIX: FTSRequest - fixed log message
*WMS
FIX: SiteDirector - use CSGlobals.getVO() call instead of explicit CS option
[v6r6p8]
*Transformation
BUGFIX: TransformationDB - typo in getTransformationFiles(): iterValues -> itervalues
[v6r6p7]
*Resources
FIX: StorageFactory - uncommented line that was preventing the status to be returned
BUGFIX: CE remote scripts - should return status and not call exit()
BUGFIX: SSHComputingElement - wrong pilot ID reference
[v6r6p6]
*WMS
FIX: TaskQueueDB - in findOrphanJobs() retrieve orphaned jobs as list of ints instead of list of tuples
FIX: OptimizerExecutor - added import of datetime to cope with the old style optimizer parameters
*Transformation
FIX: TransformationAgent - fix finalization entering in an infinite loop
NEW: TransformationCLI - added resetProcessedFile command
FIX: TransformationCleaningAgent - treating the archiving delay
FIX: TransformationDB - fix in getTransformationFiles() in case of empty file list
[v6r6p5]
*Transformation
FIX: TransformationAgent - type( transClient -> transfClient )
FIX: TransformationAgent - self._logInfo -> self.log.info
FIX: TransformationAgent - skip if no Unused files
FIX: TransformationAgent - Use CS option for replica cache lifetime
CHANGE: TransformationAgent - accept No new Unused files every [6] hours
[v6r6p4]
*DMS
FIX: TransferAgent - protection for files that can not be scheduled
BUGFIX: TransferDB - typo (instIDList - > idList ) fixed
*Transformation
BUGFIX: TransformationAgent - typo ( loginfo -> logInfo )
[v6r6p3]
FIX: merged in patch v6r5p14
*Core
BUGFIX: X509Chain - return the right structure in getCredentials() in case of failure
FIX: dirac-deploy-scripts.py - allow short scripts starting from "d"
FIX: dirac-deploy-scripts.py - added DCOMMANDS_PPID env variable in the script wrapper
FIX: ExecutorReactor - reduced error message dropping redundant Task ID
*Interfaces
BUGFIX: Dirac.py - allow to pass LFN list to replicateFile()
*DMS
FIX: FileManager - extra check if all files are available in _findFiles()
BUGFIX: FileCatalogClientCLI - bug in DirectoryListing
[v6r6p2]
FIX: merged in patch v6r5p13
*WMS
FIX: SiteDirector - if no community set, look for DIRAC/VirtualOrganization setting
*Framework
FIX: SystemLoggingDB - LogLevel made VARCHAR in the MessageRepository table
FIX: Logging - several log messages are split in fixed and variable parts
FIX: SystemLoggingDB - in insertMessage() do not insert new records in auxiliary tables if they
are already there
[v6r6p1]
*Core:
CHANGE: PromptUser - changed log level of the printout to NOTICE
NEW: Base Client constructor arguments are passed to the RPCClient constructor
*DMS:
NEW: FTSRequest - added a prestage mechanism for source files
NEW: FileCatalogClientCLI - added -f switch to the size command to use raw faile tables
instead of storage usage tables
NEW: FileCatalog - added orphan directory repair tool
NEW: FIleCatalog - more counters to control the catalog sanity
*WMS:
FIX: SandboxStoreClient - no more kwargs tricks
FIX: SandboxStoreClient returns sandbox file name in case of upload failure to allow failover
FIX: dirac-pilot - fixed VO_%s_SW_DIR env variable in case of OSG
*TS:
FIX: TransformationManagerHandler - avoid multiple Operations() instantiation in
getTransformationSummaryWeb()
[v6r6]
*Core
CHANGE: getDNForUsername helper migrated from Core.Security.CS to Registry helper
NEW: SiteSEMapping - new utilities getSitesGroupedByTierLevel(), getTier1WithAttachedTier2(),
getTier1WithTier2
CHANGE: The DIRAC.Core.Security.CS is replaced by the Registry helper
BUGFIX: dirac-install - properly parse += in .cfg files
FIX: Graphs.Utilities - allow two lines input in makeDataFromCVS()
FIX: Graphs - allow Graphs package usage if even matplotlib is not installed
NEW: dirac-compile-externals will retrieve the Externals compilation scripts from it's new location
in github (DIRACGrid/Externals)
NEW: Possibility to define a thread-global credentials for DISET connections (for web framework)
NEW: Logger - color output ( configurable )
NEW: dirac-admin-sort-cs-sites - to sort sites in the CS
CHANGE: MessageClient(Factor) - added msgClient attribute to messages
NEW: Core.Security.Properties - added JOB_MONITOR and USER_MANAGER properties
*Configuration
NEW: Registry - added getAllGroups() method
*Framework
NEW: SystemAdministratorClientCLI - possibility to define roothPath and lcgVersion when updating software
*Accounting
NEW: JobPlotter - added Normalized CPU plots to Job accounting
FIX: DBUtils - plots going to greater granularity
*DMS
NEW: FileCatalog - storage usage info stored in all the directories, not only those with files
NEW: FileCatalog - added utility to rebuild storage usage info from scratch
FIX: FileCatalog - addMetadataField() allow generic types, e.g. string
FIX: FileCatalog - path argument is normalized before usage in multiple methods
FIX: FileCatalog - new metadata for files(directories) should not be there before for directories(files)
NEW: FileCatalog - added method for rebuilding DirectoryUsage data from scratch
NEW: FileCatalog - Use DirectoryUsage mechanism for both logical and physical storage
CHANGE: FileCatalog - forbid removing non-empty directories
BUGFIX: FileCatalogClientCLI - in do_ls() check properly the path existence
FIX: FileCatalogClientCLI - protection against non-existing getCatalogCounters method in the LFC client
FIX: DMS Agents - properly call superclass constructor with loadName argument
FIX: ReplicaManager - in removeFile() non-existent file is marked as failed
FIX: Make several classes pylint compliant: DataIntegrityHandler, DataLoggingHandler,
FileCatalogHandler, StorageElementHandler, StorageElementProxyHandler, TransferDBMonitoringHandler
FIX: LogUploadAgent - remove the OSError exception in __replicate()
FIX: FileCatalogClientCLI - multiple check of proper command inputs,
automatic completion of several commands with subcommands,
automatic completion of file names
CHANGE: FileCatalogClientCLI - reformat the output of size command
FIX: dirac-admin-ban-se - allow to go over all options read/write/check for each SE
NEW: StrategyHandler - new implementation to speed up file scheduling + better error reporting
NEW: LcgFileCatalogProxy - moved from from LHCbDirac to DIRAC
FIX: ReplicaManager - removed usage of obsolete "/Resources/StorageElements/BannedTarget"
CHANGE: removed StorageUsageClient.py
CHANGE: removed obsoleted ProcessingDBAgent.py
*WMS
CHANGE: RunNumber job parameter was removed from all the relevant places ( JDL, JobDB, etc )
NEW: dirac-pilot - add environment setting for SSH and BOINC CEs
NEW: WMSAdministrator - get output for non-grid CEs if not yet in the DB
NEW: JobAgent - job publishes BOINC parameters if any
CHANGE: Get rid of LHCbPlatform everywhere except TaskQueueDB
FIX: SiteDirector - provide list of sites to the Matcher in the initial query
FIX: SiteDirector - present a list of all groups of a community to match TQs
CHANGE: dirac-boinc-pilot dropped
CHANGE: TaskQueueDirector does not depend on /LocalSite section any more
CHANGE: reduced default delays for JobCleaningAgent
CHANGE: limit the number of jobs received by JobCleaningAgent
CHANGE: JobDB - use insertFields instead of _insert
CHANGE: Matcher, TaskQueueDB - switch to use Platform rather than LHCbPlatform retaining LHCbPlatform compatibility
BUGFIX: Matcher - proper reporting pilot site and CE
CHANGE: JobManager - improved job Killing/Deleting logic
CHANGE: dirac-pilot - treat the OSG case when jobs on the same WN all run in the same directory
NEW: JobWrapper - added more status reports on different failures
FIX: PilotStatusAgent - use getPilotProxyFromDIRACGroup() instead of getPilotProxyFromVOMSGroup()
CHANGE: JobMonitoringHandler - add cutDate and condDict parameters to getJobGroup()
NEW: JobMonitoringHandler - check access rights with JobPolicy when accessing job info from the web
NEW: JobManager,JobWrapper - report to accounting jobs in Rescheduled final state if rescheduling is successful
FIX: WMSAdministrator, SiteDirector - store only non-empty pilot output to the PilotDB
NEW: added killPilot() to the WMSAdministrator interface, DiracAdmin and dirac-admin-kill-pilot command
NEW: TimeLeft - renormalize time left using DIRAC Normalization if available
FIX: JobManager - reconnect to the OptimizationMind in background if not yet connected
CHANGE: JobManifest - use Operations helper
NEW: JobCleaningAgent - delete logging records from JobLoggingDB when deleting jobs
*RMS
FIX: RequestDBFile - better exception handling in case no JobID supplied
FIX: RequestManagerHandler - make it pylint compliant
NEW: RequestProxyHandler - is forwarding requests from voboxes to central RequestManager.
If central RequestManager is down, requests are dumped into file cache and a separate thread
running in background is trying to push them into the central.
CHANGE: Major revision of the code
CHANGE: RequestDB - added index on SubRequestID in the Files table
CHANGE: RequestClient - readRequestForJobs updated to the new RequetsClient structure
*RSS
NEW: CS.py - Space Tokens were hardcoded, now are obtained after scanning the StorageElements.
*Resources
FIX: SSHComputingElement - enabled multiple hosts in one queue, more debugging
CHANGE: SSHXXX Computing Elements - define SSH class once in the SSHComputingElement
NEW: SSHComputingElement - added option to define private key location
CHANGE: Get rid of legacy methods in ComputingElement
NEW: enable definition of ChecksumType per SE
NEW: SSHBatch, SSHCondor Computing Elements
NEW: SSHxxx Computing Elements - using remote control scripts to better capture remote command errors
CHANGE: put common functionality into SSHComputingElement base class for all SSHxxx CEs
NEW: added killJob() method tp all the CEs
NEW: FileCatalog - take the catalog information info from /Operations CS section, if defined there,
to allow specifications per VO
*Interfaces
CHANGE: Removed Script.initialize() from the API initialization
CHANGE: Some general API polishing
FIX: Dirac.py - when running in mode="local" any directory in the ISB would not get untarred,
contrary to what is done in the JobWrapper
*TS
BUGFIX: TaskManager - bug fixed in treating tasks with input data
FIX: TransformationCleaningAgent - properly call superclass constructor with loadName argument
NEW: TransformationCleaningAgent - added _addExtraDirectories() method to extend the list of
directories to clean in a subclass if needed
CHANGE: TransformationCleaningAgent - removed usage of StorageUsageClient
NEW: TransformationAgent is multithreaded now ( implementation moved from LHCbDIRAC )
NEW: added unit tests
NEW: InputDataAgent - possibility to refresh only data registered in the last predefined period of time
NEW: TransformationAgent(Client) - management of derived transformations and more ported from LHCbDIRAC
BUGFIX: TransformationDB - wrong SQL statement generation in setFileStatusForTransformation()
[v6r5p14]
*Core
NEW: Utilities - added Backports utility
*WMS
FIX: Use /Operations/JobScheduling section consistently, drop /Operations/Matching section
NEW: Allow VO specific share correction plugins from extensions
FIX: Executors - several fixes
[v6r5p13]
*WMS
FIX: Executors - VOPlugin will properly send and receive the params
NEW: Correctors can be defined in an extension
FIX: Correctors - Properly retrieve info from the CS using the ops helper
[v6r5p12]
FIX: merged in patch v6r4p34
[v6r5p11]
FIX: merged in patch v6r4p33
*Core
FIX: MySQL - added offset argument to buildConditions()
[v6r5p10]
FIX: merged in patch v6r4p32
[v6r5p9]
FIX: merged in patch v6r4p30
[v6r5p8]
FIX: merged in patch v6r4p29
[v6r5p7]
FIX: merged in patch v6r4p28
[v6r5p6]
FIX: merged in patch v6r4p27
*Transformation
BUGFIX: TransformationDB - StringType must be imported before it can be used
*RSS
NEW: CS.py - Space Tokens were hardcoded, now are obtained after scanning the StorageElements.
[v6r5p5]
FIX: merged in patch v6r4p26
[v6r5p4]
FIX: merged in patch v6r4p25
[v6r5p3]
*Transformation
FIX: merged in patch v6r4p24
[v6r5p2]
*Web
NEW: includes DIRACWeb tag web2012092101
[v6r5p1]
*Core
BUGFIX: ExecutorMindHandler - return S_OK() in the initializeHandler
FIX: OptimizationMindHandler - if the manifest is not dirty it will not be updated by the Mind
*Configuration
NEW: Resources helper - added getCompatiblePlatform(), getDIRACPlatform() methods
*Resources
FIX: SSHComputingElement - add -q option to ssh command to avoid banners in the output
FIX: BOINCComputingElement - removed debugging printout
FIX: ComputingElement - use Platform CS option which will be converted to LHCbPlatform for legacy compatibility
*DMS
FIX: RequestAgentBase - lowering loglevel from ALWAYS to INFO to avoid flooding SystemLogging
*WMS:
FIX: SiteDirector - provide CE platform parameter when interrogating the TQ
FIX: GridPilotDirector - publish pilot OwnerGroup rather than VOMS role
FIX: WMSUtilities - add new error string into the parsing of the job output retrieval
[v6r5]
NEW: Executor framework
*Core
NEW: MySQL.py - added Test case for Time.dateTime time stamps
NEW: MySQL.py - insertFields and updateFields can get values via Lists or Dicts
NEW: DataIntegrityDB - use the new methods from MySQL and add test cases
NEW: DataIntegrityHandler - check connection to DB and create tables (or update their schema)
NEW: DataLoggingDB - use the new methods from MySQL and add test cases
NEW: DataLoggingHandler - check connection to DB and create tables (or update their schema)
FIX: ProcessPool - killing stuck workers after timeout
CHANGE: DB will throw a RuntimeException instead of a sys.exit in case it can't contact the DB
CHANGE: Several improvements on DISET
CHANGE: Fixed all DOS endings to UNIX
CHANGE: Agents, Services and Executors know how to react to CSSection/Module and react accordingly
NEW: install tools are updated to deal with executors
FIX: dirac-install - add -T/--Timeout option to define timeout for distribution downloads
NEW: dirac-install - added possibility of defining dirac-install's global defaults by command line switch
BUGFIX: avoid PathFinder.getServiceURL and use Client class ( DataLoggingClient,LfcFileCatalogProxyClient )
FIX: MySQL - added TIMESTAMPADD and TIMESTAMPDIFF to special values not to be scaped by MySQL
NEW: ObjectLoader utility
CHANGE: dirac-distribution - added global defaults flag and changed the flag to -M or --defaultsURL
FIX: Convert to string before trying to escape value in MySQL
NEW: DISET Services - added PacketTimeout option
NEW: SystemLoggingDB - updated to use the renewed MySQL interface and SQL schema
NEW: Added support for multiple entries in /Registry/DefaultGroup, for multi-VO installations
CHANGE: Component installation procedure updated to cope with components inheriting Modules
CHANGE: InstallTools - use dirac- command in runit run scripts
FIX: X509Chain - avoid a return of error when the group is not valid
FIX: MySQL - reduce verbosity of log messages when high level methods are used
CHANGE: Several DB classes have been updated to use the MySQL buildCondition method
NEW: MySQL - provide support for greater and smaller arguments to all MySQL high level methods
FIX: Service.py - check all return values from all initializers
*Configuration
CHANGE: By default return option and section lists ordered as in the CS
NEW: ConfigurationClient - added function to refresh remote configuration
*Framework
FIX: Registry.findDefaultGroup will never return False
CHANGE: ProxyManager does not accept proxies without explicit group
CHANGE: SystemAdministratorHandler - force refreshing the configuration after new component setup
*RSS