Retain generated corpus and pretranslation files for a build

[ddaspit]

Currently, Serval deletes corpus and pretranslation files once a build has finished. This makes it difficult to debug issues and to perform testing. Instead, Serval should retain the files after a build has finished. The files should be deleted after a predetermined amount of time (maybe 30 days).

[ddaspit]

This is needed to support gray box testing by the test team.

[johnml1135]

Do we want this for QA only? Do we want this controllable by a flag?

[ddaspit]

I think it would be good to do this on production as well, since it will be helpful for debugging.

[ddaspit]

We still need to implement a method for cleaning up the files.

[Enkidu93]

I know you were asking about an approach in the PR above, @ddaspit: I'm thinking I'll just add a DateFinished to the machine build model which we can set in BuildJobService.BuildJobFinishedAsync and then add another RecurrentTask similar to the ModelCleanupService to delete the build artifacts after a certain amount of time (is 30 days long enough?). Does that sound like a good approach? We could wait until 30 days after the code has been pushed to production and then push another change to delete build artifacts for translation engines which don't have a DateFinished property set to avoid indefinitely storing the build artifacts from builds build recently or we could set the existing builds to have the current date at the time of the push (probable simpler).

[johnml1135]

The longest that I could imagine that a job could take would be 1 day. If we set the expiration for 30 days and key off of DateCreated (or whatever the field is), I think it could be simpler to implement and not have an old vs. new data issue.

[Enkidu93]

The longest that I could imagine that a job could take would be 1 day. If we set the expiration for 30 days and key off of DateCreated (or whatever the field is), I think it could be simpler to implement and not have an old vs. new data issue.

I'm not sure what you mean by 'set the expiration', @johnml1135. Are you referring to an s3 object expiration?

[johnml1135]

Set the expiration -> "delete the build artifacts after a certain amount of time." Sorry for the bad English.

[Enkidu93]

Set the expiration -> "delete the build artifacts after a certain amount of time." Sorry for the bad English.

No, no, I just wanted to make sure I understood 😁 . So you're suggesting we use a property of the files themselves?

[johnml1135]

Yes. Any system should at least have a created on and last modified on date.

[ddaspit]

The files are stored in the S3 bucket. Is that correct? Does S3 have a date modified/created field?

[Enkidu93]

Yes. Any system should at least have a created on and last modified on date.

OK, so basically just the equivalent of the clean_s3 script in silnlp. I guess that makes sense.

[ddaspit]

A script like that could be run using a cron job. I think that is how Matthew is doing it.

[Enkidu93]

A script like that could be run using a cron job. I think that is how Matthew is doing it.

Right. Might it be appropriate just to hijack that script and tag on an additional pattern that covers the Serval builds?

[ddaspit]

That makes sense to me. Although, we are planning on moving silnlp data to the NAS, so production and research data will be stored in separate locations.

[johnml1135]

I am assuming that the job would be run directly within serval as a recurring task. If it were an actual cron job, the deployment would be non-obvious.

[ddaspit]

If possible, I would prefer to run this as a cron job rather than use up a thread in the Serval server. Does Kubernetes have some way to run scheduled jobs?

[johnml1135]

Another option is just to have it completely separate from Serval - running the cron job on with the SILNLP cron job for the S3 bucket data. I would make sure that there was a flag to turn on and off the serval auto-deleting at the end of jobs though. It would be a bit manual, but really only effects S3 bucket cost if the cron job does not execute.

[Enkidu93]

What's the conclusion here in regards to a strategy?

[johnml1135]

Let's talk about it at Wednesday's meeting. I am still inclined to do the "roll it into SILNLP cleanup" thing.

[johnml1135]

@mshannon-sil - If I am correct, there is an SILNLP job that runs that cleans up old data. Is that correct?

[mshannon-sil]

Yes that's right, the clean_s3 script in the SILNLP repo currently runs as a cron job on the AQuA server every Sunday.

[johnml1135]

@mshannon-sil - how hard would it be to extend that script to also clean production data?

[mshannon-sil]

Should be pretty straightforward. It would just need to check if the files are in the production folder and match the pattern for corpus and pretranslation files, and then delete any files older than 30 days. If we just want to update the clean_s3 script for this, I'd be happy to do it.

[johnml1135]

@mshannon-sil, what is the status of this issue?

[mshannon-sil]

The status is ready to be worked on. I have some time today, so I'll work on it and keep you updated.

[mshannon-sil]

@johnml1135 I just submitted a PR for this issue in SILNLP.