Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deleter: prioritise different file types #34338

Draft
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

incognitojam
Copy link
Member

@incognitojam incognitojam commented Jan 8, 2025

Deleter processes files one at a time, ranking them based on:

  • whether the segment is preserved
  • whether the file is for "dashcam" (qlog, qcamera.ts)
  • whether the file is for "training" (rlog, hevc...)

The amount of files classified as "training" is limited to 5GB. This means that only the most recent 5GB of training files are retained and the excess is de-prioritised and deleted to make space for other files.

  • deprioritise old training data over size limit
  • delete empty folders
  • deleting all of file type in the same segment? deleting one training file and keeping another would be useless...

@maxime-desroches
Copy link
Contributor

You can rebase on master to solve the Jenkins failures

@incognitojam incognitojam force-pushed the deleter-training-limit branch 4 times, most recently from 4dd828a to 5fe1bfa Compare January 9, 2025 15:00
@incognitojam incognitojam force-pushed the deleter-training-limit branch from 0841310 to 3863293 Compare January 9, 2025 21:45
Copy link
Contributor

This PR has had no activity for 9 days. It will be automatically closed in 2 days if there is no activity.

@github-actions github-actions bot added the stale label Jan 20, 2025
@github-actions github-actions bot removed the stale label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants