Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to add element to the queue? #23

Open
leekiaxnn opened this issue Aug 3, 2022 · 2 comments
Open

how to add element to the queue? #23

leekiaxnn opened this issue Aug 3, 2022 · 2 comments
Labels
question Further information is requested

Comments

@leekiaxnn
Copy link

leekiaxnn commented Aug 3, 2022

I found the following comments in the source code

No longer using BucketMonitor to add keys to Redis.

Instead, use a StaleFileBucketMonitor to prune stale files in the bucket.

But the new class does not have any operations for redis queue.
Did it move to another place?

`class StaleFileBucketMonitor(BaseBucketMonitor):
"""Watches a bucket for new uploads and adds data for each to Redis."""

def scan_bucket_for_stale_files(self,
                                prefix='uploads/',
                                threshold=7 * 24 * 60 * 60):
    """Remove stale files in bucket with the given prefix

    Args:
        prefix (str): The prefix/folder to look for stale files.
        threshold (int): The maximum allowed age of files, in seconds.
    """
    prefix = '{}/'.format(prefix) if not prefix.endswith('/') else prefix

    current_timestamp = datetime.datetime.now(pytz.UTC)

    # get references to every file starting with `prefix`
    all_files = self.get_all_files(prefix=prefix)

    for f in all_files:
        if f.name == prefix:
            continue  # no need to process the prefix directory

        age_in_seconds = (current_timestamp - f.updated).total_seconds()

        if age_in_seconds > threshold:
            self.logger.info('Found file %s which is %s seconds old.',
                             f.name, age_in_seconds)
            f.delete()  # delete the file, cannot be undone
            self.logger.info('Successfully deleted file %s.', f.name)

`

@msschwartz21
Copy link
Member

Hi Lijiaxin3,

If you are trying to add a new queue and job type to the kiosk environment I would suggest that you look at this tutorial: https://deepcell-kiosk.readthedocs.io/en/master/CUSTOM-JOB.html. In order to add a new job type, you will need to modify three helmfiles, but it shouldn't require any changes to the bucket monitor. The kiosk-redis-consumer handles most of the interactions with the redis queue.

If you are working on a different problem, please let me know and I will try to answer that question.

@msschwartz21 msschwartz21 added the question Further information is requested label Aug 3, 2022
@leekiaxnn
Copy link
Author

Thanks for your reply!
I now know how to do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants