enhancement: Speed up shopify scraping by pushing chunks into the queue as you go.

[cdxker]

Description

When scraping a shopify store, we shouldn't need to wait for all the items to be pulled and listed. We should auto push chunks into the queue in batches of 120.

Try out the store rugsusa .com store to test out this behavior. It takes a very long time to grab the initial chunks. This will require a bit of a refactor from how we do it currently. (Might be fun to try out a bounded channel).

Target(s)

crawl-worker

Requirement to close

• push chunks in batches of 120 as soon as they become available.

Community channels

Matrix is preferred. Reach out on discord or Matrix for further assistance.

• discord
• matrix