-
Notifications
You must be signed in to change notification settings - Fork 10
/
CHANGES.txt
228 lines (203 loc) · 9.37 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
3.1.24
Update documentation
Fix tests
Remove dependency on python3-future
3.1.23:
Use pytest instead of nose
3.1.21:
Freeze pymongo to 3.12.3 (4.x breaks)
Change isAlive() which is deprecated in python 3.9 to is_alive
3.1.20:
Follow-up of #127 to get last release in file (refactor and bug fix)
3.1.19:
Add tgz archive support
Add log file info to production info
#126 Issue with getting last release in file
3.1.18:
Python 3 support only
If multiple files match release.file, take most recent one
If mail template not found, log and use default
3.1.17:
Fix regression when saving file with a differe,t structure such as xxx/(FASTA)/(file.txt) to save under FASTA/file.txt
Send removal mail for --remove-all option
#119 add support for custom notification emails with templates and log tail/attach options
New optional fields in global.properties (or per bank properties):
mail.body.tail=0
mail.body.attach=9000000
mail.template.subject=file_path_to_subject.jinja2
mail.template.body=file_path_to_body.jinja2
Available variables:
'log_file': path to log file
'log_tail': last lines of log file
'bank': bank name
'release': release related tooperation
'status': operation status (true/false)
'modified': did operation modified bank (true/false)
'update': was operation an update
'remove': was operation a removal
3.1.16:
Fix status check of process for --from-task postprocess
#118 Rename protocol options to options
Add more debug logging
3.1.15:
#117 Fix incorrect behavior with --post-process
3.1.14:
Add repair option
3.1.13:
Add process name and status in logs
PR #116 update to use download 3.1.0
3.1.12:
In case of multiple matches for release regexp, try to determine most recent one
#115 Correctly use save_as for release file name
3.1.11:
Increase one log level
#110 Allow ftps and directftps protocols (needs biomaj-download 3.0.26 and biomaj-core 3.0.19)
#111 locked bank after bad update command
Ignore UTF-8 errors in release file
Add plugin support via biomaj-plugins repo (https://github.com/genouest/biomaj-plugins) to get release and list of files to download from a plugin script.
Add support for protocol options in global and bank properties (options.names=x,y options.x=val options.y=val). Options may be ignored or used differently depending on used protocol.
3.1.10:
Allow to use hardlinks when reusing files from previous releases
3.1.9:
Fix remote.files recursion
3.1.8:
Fix uncompress when saved files contains subdirectory
3.1.7:
Fix utf/ascii encoding issue with python3
In case of uncompress failure, put back all compressed files to avoid redownload
3.1.6:
Fix #100 Catch error and log error if biomaj fails to connect to InfluxDB
Add history to update/remove operations
Add log in case of file deletion error during bank removal
check lock file exists when removing it
Update protobuf to work with biomaj.download 3.0.18
3.1.5:
Fix #97 Wrong offline dir checks
3.1.4:
Fix #88 Unset 'last_update_session' when found in pending sessions using --remove-pending
Add formats in bank info request
Add checks for some production fields before display
Add irods download support
3.1.3:
Remove post-install step for automatic upgrades, not supported by wheel package
3.1.2:
Fix #86 remove special character from README.md
Feature #85 SchemaVersion automatically add new property
3.1.1:
Fix #80 Check process exists with `--from-task` and `--process`
Manage old banks with no status
3.1.0:
## Needs database upgrade
If using biomaj-watcher, must use version >= 3.1.0
Feature #67,#66,#61 switch to micro service architecture. Still works in local monolithic install
Fix some configuration parameter loading when not defined in config
Fix HTTP parsing parameters loading
Fix download_or_copy to copy files in last production release if available instead of downloading files again
Manage user migration for micro services
Feature #74 add influxdb statistics
Feature #65 add a release info file at the root of the bank which can be used by other services to know the latest release available
Feature #25 experimental support of rsync protocol
Add rate limiting for download with micro services
Limit email size to 2Mb, log file may be truncated
3.0.20:
Fix #55: Added support for https and directhttps
Add possibility to define files to download from a local file with remote.list parameter
Fix visibility modification (bug deleted the bank properties field)
Fix #65 Add release file in bank dir after update
Add md5 or sha256 checksum checks if files are downloaded and available
3.0.19:
Fix missing README.md in package
Fix #53 avoid duplicates in pending databases
3.0.18:
Add migration method to update schema when needed
Manage HTTP month format to support text format (Jan, Feb, ...) and int format (01, 02, ...)
New optional bank property http.parse.file.date.format to extract date in HTTP protocol following python date regexp format (http://www.tutorialspoint.com/python/time_strptime.htm)
Example: %d-%b-%Y %H:%M
3.0.17:
Fix #47: save_as error with directhttp protocol
Fix #45: error with pending releases when release has dots in value
typo/pylint fixes
3.0.16:
Do not use config values, trust database values #39
Fix #42: Add optional release.separator to name the bank directory bankname_release (underscore as default)
3.0.15:
Fix #37: remote local files history from db and put it in cache.dir
Feature #38: add optional keep.old.sessions parameter to keep all sessions in database, even for removed releases
Feature #28: add optional release.format parameter to specify the date format of a release
3.0.14:
Fix in method set_owner
Force release to be a str
Fix #32: fix --from-task issue when calling a meta process
Fix #34: remove release from pending when doing cleanup of old sessions
Remove logs on some operations
Add --status-ko option to list bank in error state
Fix #36 manage workflows over by error or unfinished
3.0.13:
Fix #27: Thread lock issue during download
New optional attribute in bank properties: timeout.download
HTTP protocol fix (deepcopy error)
3.0.12:
Fix index deletion on bank removal
Fix lock errors on dir creation for multi-threads,
pre-create directroy structure in offline directory
Fix #26: save error when too many files in bank
3.0.11:
Fix in session management with pre and rm processes
Fix #23: Check workflow step name passed to
--stop-after/--start-after/--from-task
Fix #24: deprecated delete_by_query method in elasticsearch
Add some controls on base directories
3.0.10:
Change dir to process.dir to find processes in subdirs
If all files found in offline dir, continue workflow with no download
Remove extra log files for bank dependencies (computed banks)
Fix computed bank update when sub banks are not updated
Fix #15 when remote reverts to a previous release
Feature #16: get possibility not to download files (for computed banks for
example). Set protocol='none' in bank properties.
Fix on --check with some protocols
Fix #21 release.file not supported for directhttp protocol
Feature #22: add localrelease and remoterelease bank properties to use the
remote release as an expression in other properties
=> remote.dir = xx/yy/%(remoterelease)s/zz
Feature #17,#20: detect remote modifications even if release is the same
new parameter release.control (true, false) to force a check
even if remote release (file controlled or date) is the same.
Fix on 'multi' protocol
Fix on "save_as" regexp when remote.files starts with a ^ character.
3.0.9:
Fix thread synchro issue:
during download some download threads could be alive while main thread continues worflow
the fix prevents using Ctrl-C during download
Workflow fix:
if subtask of workflow fails, fail main task
3.0.8:
do not test index if elasticsearch is not up
minor fixes
add http proxy support
pylint fixes
retry uncompress once in case of failure (#13)
3.0.7:
Reindent code, pep8 fixes
Various fixes on var names and OrderedDict suport for Python < 2.7
Merge config files to be able to reference global.properties variables in bank
property file in format %(xx)s
Use ConfigParser instead of SafeConfigParser that will be deprecated
3.0.6:
Add option --remove-pending to remove all pending sessions and directories
Add process env variables logdir and logfile
Fix Unicode issue with old versions of PyCurl.
3.0.5:
Fix removal workflow during an update workflow, removedrelease was current
release.
Fix shebang of biomaj-cli, and python 2/3 compat issue
3.0.4:
Update code to make it Python 3 compatible
Use ldap3 library (pure Python and p2,3 compatible) instead of python-ldap
get possiblity to save downloaded files for ftp and http without keeping full
directory structure:
remote.files can include groups to save file without directory structure,
or partial directories only, examples:
remote.files = genomes/fasta/.*\.gz => save files in offline directory, keeping remote structure offlinedir/genomes/fasta/
remote.files = genomes/fasta/(.*\.gz) => save files in offline directory offlinedir/
remote.files = genomes/(fasta)/(.*\.gz) => save files in offline directory offlinedir/fasta