Using django-celery-beat to manage periodic tasks

[ManofWax]

BuffaLogs is based on periodic tasks to detect anomalies. The configs for remote tasks are on django settings file.

We should use django-celery-beat to manage task scheduling configs, in this way we could allow users to edit and change task scheduling in realtime.

This implementation should also take care of loading the default django-celery-beat config when the system starts up the first time. It possible to do so using a django managment command and add it to the docker container start up script

[deep-poharkar]

can i have this? would love to contribute.

[ManofWax]

Hi, yeah sure you can implement this.

[0x0elliot]

seems like no one is helping on this issue. would you mind if i quickly wrap this up?

so would you just like me to add a django celery beat? i do need a function to add in here to run.

[0x0elliot]

a bit unclear about which function i should plugin, so i am adding the process_logs celery function here.

[ManofWax]

Hi @0x0elliot, the function you should plugin are process_logs and update_risk_level defined here: https://github.com/certego/BuffaLogs/blob/main/buffalogs/buffalogs/settings/settings.py#L194

Since the django-celery-beat will be used instead of this configuration you should also provide a way to load on the db the cron settings so when someone launches the containers the tasks will be already active.

I'm thinking on putting a command after collectstatics in run.sh: https://github.com/certego/BuffaLogs/blob/main/buffalogs/run.sh that loads the cron setting if the database is empty.

[0x0elliot]

sorry, i am still catching up here but do you want to call update_risk_level separately in the cron? i see that impossible_travel command also calls it. i think it's better to just call that so i am adding it for now.

[0x0elliot]

rabbitMQ seems to be breaking for me and not working. weird.

[Lorygold]

sorry, i am still catching up here but do you want to call update_risk_level separately in the cron? i see that impossible_travel command also calls it. i think it's better to just call that so i am adding it for now.

Hi, the impossible_travel command calls the update_risk_level function in order to check its correct operation manually, in staging.
But to ensure the automatic execution of update_risk_level, it is necessary to insert it in the cron. In this way, this task is performed every 10 minutes.

[Lorygold]

rabbitMQ seems to be breaking for me and not working. weird.

It is still working for me. Can you print here the rabbitMQ logs, please?

[0x0elliot]

It is still working for me. Can you print here the rabbitMQ logs, please?

don't worry about that. it's just the delay in it starting for the first time when celery tries to connect to it that was bothering me.

[AnkurPrabhu]

is this issue still open ?

[AnkurPrabhu]

@ManofWax can i pick this up ?