Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No more posts while gathering *lot* of posts based on hashtags #786

Open
mandys opened this issue Aug 31, 2022 · 0 comments
Open

No more posts while gathering *lot* of posts based on hashtags #786

mandys opened this issue Aug 31, 2022 · 0 comments

Comments

@mandys
Copy link

mandys commented Aug 31, 2022

Describe the bug
The script works fine if I am trying to pull around 500 hashtags ( most of the time ). But, if the hashtags have let's say about 1000+ results, the response is unpredictable. The script is looping fine, 30 results at a time.

For my use case: the hashtag has more than 10,000+ results. I have set number=1000.
Every single time it's looping, it stops around -
https://m.tiktok.com/api/challenge/item_list/?challengeID=&count=30&cursor=800&aid=1988&verifyFp=&_signature=

{
"statusCode": 0,
"cursor": "800",
"hasMore": false
}

I think in TikTok.js it reaches here
if ((updatedApiResponse && !result.itemList) || (!updatedApiResponse && !result.items)) {
throw new Error('No more posts');
}

In this scenario -

  1. The script should return the results since it went through 26 cycles before encountering no itemList. At least whatever it scraped should be returned ? Right now it just aborts saying "No more posts".

  2. Why is this stopping around 799 ( total ) results when the hashtag has 10,000+ results. In fact even on tiktok web interface, I wasn't able to consistently scroll past 500 odd results. They stopped loading post that. So, does tiktok not give back all the results ? Only upto a particular date ? Any idea ?

If fields below will be empty then issue will be Ignored and Closed

To Reproduce
Steps to reproduce the behavior:

  • Used OS
    [ Mac OS X ]

  • NodeJs Version (node -v)
    [ 12.16.1]

  • Scraper Version (tiktok-scraper --version)
    [1.4.36]

  • CLI or Module
    [ Module ]

  • Used method
    TikTokScraper.hashtag

  • Used options
    const posts = await TikTokScraper.hashtag(hashtag, {
    number: 700,
    sessionList: ['sid_tt=123;'],
    });

  • Do you run the scraper through Proxy, VPN, VPS?
    VPN and also through an AWS box

  • If scraping from a profile. Did you checked if profile is Public? Scraper won't work with Private profiles!
    [ Scraping hashtags which are public ]

  • Preferably provide URL to the profile,hashtag,video and etc page
    https://www.tiktok.com/tag/pandya

Screenshots
If applicable, add screenshots to help explain your problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant