Scraping without logging in + proxies

[Mdaadm]

Hi! I am trying to scrape non-private groups without logging in. Facebook allows to do so, but after ~1000 scraped posts it starts to require login, and then temporarily blocks my ip. Tried to use some paid residential proxies, but if i use them, facebook instantly redirects to login page, so i cant get any data at all.

For example, without proxies i can scroll page https://m.facebook.com/groups/1885625948323258 , but with proxies connected i get "you must log in first" page. Used set_proxy method, also checked this from browser with proxies connected, same result.

Does this mean, that proxies are on the FB black list? If so, can anyone advise good proxy service for this purpose?

[neon-ninja]

I would recommend using cookies instead of proxies

[jonathanrstern]

Good to know, thank you @neon-ninja!

A follow up: In the past, I've used Puppeteer with a residential proxy from Bright Data, to scrape FB data.

Do you think it would work to scrape public info off of Facebook pages without a proxy? I imagine Facebook would ban that IP eventually,? What have you seen in your experience?

What I'm thinking of doing would probably average 1-2 requests per minute.

[neon-ninja]

In my experience, FB doesn't ban IPs, just accounts. Some IPs (cloud providers) are "more suspicious" than others. 1-2 requests per minute should be safe. Most of the people complaining about temp bans in this issue tracker are doing a lot more than that

[jonathanrstern]

Thanks for the thoughts @neon-ninja - appreciate it!!

[boynet]

@jonathanrstern hi, is it to much to ask about brightdata residential proxy pricing , is it wrong to assume that each page is around 10mb-20mb will cost around 0.2$-0.3$ per request? or I am doing something wrong here

[jonathanrstern]

That’s a decent rough estimate for the residential proxy.

Bright Data’s custom solution is far better if your quantities are high.

[sagihgithub]

Hi
I'm facing the same outcome but with a different scenario:
I've build a function that sends request to facebook fublic groups feed (without logging in), the code works fine in development environment. but in production I'm been redirected to the "you must log in first" page. my production environment is on shared hosting (the server located on different country then development environment).
what could be the problem and how can I debug it ?
appreciate any help

[neon-ninja]

Most people don't connect to Facebook from shared hosting, so it's pretty obvious you're scraping

[Charlotte-br560]

It seems like Facebook might be detecting the proxies you're using and blocking access. This could be due to the proxies being on Facebook's blacklist. As for proxy services, I've heard good things about Crawlbase for scraping purposes. You might want to try rotating through different proxies or experimenting with different providers to see if you can bypass the block.