Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH: smtp_batch invalid row hardening #2491

Merged
merged 2 commits into from
Apr 9, 2024
Merged

ENH: smtp_batch invalid row hardening #2491

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bbbae1a
ENH: smtp_batch invalid row hardening
e3rd Apr 4, 2024
6e6c03b
[to-be-squashed]
e3rd Apr 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/user/bots.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4969,7 +4969,7 @@ Note: The field "raw" gets base64 decoded if possible. Bytes `\n` and `\r` are r

Launch it like this:
```
</usr/local/bin executable> <bot-id> cli [--tester tester's email]
</usr/local/bin executable> <bot-id> --cli [--tester tester's email]
```
Example:
```bash
Expand All @@ -4993,7 +4993,7 @@ You can schedule the batch sending easily with a cron script, I.E. put this into

```
# Send the e-mails every day at 6 AM
0 6 * * * /usr/local/bin/intelmq.bots.outputs.smtp_batch.output smtp-batch-output-cz cli --ignore-older-than-days 4 --send > /tmp/intelmq-send.log
0 6 * * * /usr/local/bin/intelmq.bots.outputs.smtp_batch.output smtp-batch-output-cz cli --ignore-older-than-days 4 --send &> /tmp/intelmq-send.log
```

**Module:** `intelmq.bots.outputs.smtp_batch.output`
Expand Down
12 changes: 7 additions & 5 deletions intelmq/bots/outputs/smtp_batch/output.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,7 @@ def prepare_mails(self):
continue
else:
# visible both warning and print
e3rd marked this conversation as resolved.
Show resolved Hide resolved
print(f"Warning: {mail_record} timeout, too big to read from redis", flush=True)
self.logger.warning(f"Warning: {mail_record} timeout, too big to read from redis")
self.logger.warning(f"Warning: %s timeout, too big to read from redis", mail_record)
e3rd marked this conversation as resolved.
Show resolved Hide resolved
self.timeout.append(mail_record)
continue

Expand All @@ -253,8 +252,11 @@ def prepare_mails(self):
fieldnames = set()
rows_output = []
for row in lines:
if threshold and row["time.observation"][:19] < threshold.isoformat()[:19]:
continue
try:
if threshold and row["time.observation"][:19] < threshold.isoformat()[:19]:
continue
except KeyError:
self.logger.warning(f"Warning: %s row skipped due to time.observation error", mail_record)
e3rd marked this conversation as resolved.
Show resolved Hide resolved
fieldnames = fieldnames | set(row.keys())
keys = set(self.allowed_fieldnames).intersection(row)
ordered_keys = []
Expand Down Expand Up @@ -290,7 +292,7 @@ def prepare_mails(self):
try:
zf.writestr(filename + ".csv", output.getvalue())
except Exception:
self.logger.error(f"Cannot zip mail {mail_record}")
self.logger.error(f"Cannot zip mail %s", mail_record)
e3rd marked this conversation as resolved.
Show resolved Hide resolved
continue

if email_to in self.alternative_mail:
Expand Down
Loading