Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

atproto_firehose: load disabled accounts into in-memory DIDs #1670

Open
snarfed opened this issue Jan 6, 2025 · 0 comments
Open

atproto_firehose: load disabled accounts into in-memory DIDs #1670

snarfed opened this issue Jan 6, 2025 · 0 comments
Labels

Comments

@snarfed
Copy link
Owner

snarfed commented Jan 6, 2025

Right now, atproto-hub keeps the DIDs of all accounts bridged into and out of Bluesky in memory, and updates them every 10s. It only adds to them, though; it doesn't remove them when they're disabled. We handle disabled accounts fine later on, in receive tasks, but this means that when we disable acounts due to volume/overuse, eg #1669, it doesn't take effect in atproto-hub until we restart it.

We should drop the status/enabled_protocols filters from these queries, check those in memory, and remove DIDs that got disabled. (Or do two queries.)

atproto_query = ATProto.query(ATProto.enabled_protocols != None,
ATProto.updated > atproto_loaded_at)
loaded_at = ATProto.query().order(-ATProto.updated).get().updated
new_atproto = [key.id() for key in atproto_query.iter(keys_only=True)]
atproto_dids.update(new_atproto)
# set *after* we populate atproto_dids so that if we crash earlier, we
# re-query from the earlier timestamp
atproto_loaded_at = loaded_at
bridged_query = AtpRepo.query(AtpRepo.status == None,
AtpRepo.created > bridged_loaded_at)

@Tamschi Tamschi added the infra label Jan 6, 2025
@snarfed snarfed added now and removed now labels Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants