Same scraping script not work on VPS (test on two VPS, one centos8, one Ubuntu 20.04), but works on local machine (Arch linux).

[zw963]

This is a so strange issue, for test it, i deploy same script on a new VPS, use Ubuntu 20.04 (former use Centos 8), and both of them can reproduce this issue.

Before start, let me upload a video for describe the scraping web site.

2.mp4

If you can't see it, please download zipped mp4 video from attachment.
2.zip

As you can see, this video start with after log in success, it enter user admin management page. and then, i send a request to https://www.jin10.com and wait the following element appear.

You can see it works on my local. (Arch Linux) use ferrum + chrome, in fact, it works when use ferrum + chrome headless too.
but same code, not work on my VPS, i test on two vps, one centos 8, one Ubuntu 20.04, both of them get blocked to wait the tabs in above screenshot red box appear.

Following is my scraping code:

options = {
  pending_connection_errors: false,
  window_size: [1600, 900],
  timeout: 30,
  browser_options: { 'no-sandbox': nil, 'blink-settings' => 'imagesEnabled=false', 'start-maximized': true},
  headless: true,
  slowmo: 0.5
}

instance = Ferrum::Browser.new(options)

url = 'https://ucenter.jin10.com'

instance.goto url

# try finding login form
until (form=instance.at_css('#J_loginForm')) &&
    (login=form.at_css('#J_loginPhone')) &&
    login.focusable?

  sleep 0.5
end

# focus on it, typing username, password
login.focus.type(ENV['JIN10_USER'])
form.at_css('#J_loginPassword').focus.type(ENV['JIN10_PASS'])
instance.network.wait_for_idle

# click on login button
form.at_css('button[type="submit"]').click

# wait user admin management page is appear.
until instance.at_css('div.ucenter-menu span.ucenter-menu_title')
  instance.screenshot(path: 'pngs/ccc.png')
  sleep 3
end

# after appear, goto the homepage
url = 'https://www.jin10.com'
instance.goto url

# waiting tabs appear.
while (group_count = instance.css('ul.classify-list li').count) < 2
  sleep 5                         # <= it keep infinite loop here when run on vps.
  instance.screenshot(path: '1.png')
end

I have to admit, headless on local occasionally not work, and headless on VPS, it works several days ago too, it just very very slow when waiting the tabs in most of case if use with chrome headless.

Anyway, please guide me for how to find out where the issue come from.

I add logger to instance, like this:

  options = {
    logger: ChromeHeadlessLogger.new(Logger.new('log/chrome_headless.log', 10, 1024000)),
    pending_connection_errors: false,
    window_size: [1600, 900],
    process_timeout: 15,
    browser_options: { 'no-sandbox': nil, 'blink-settings' => 'imagesEnabled=false', 'start-maximized': true},
    headless: true,
  }

it have many logs initialily, but when code keep loop to find out following element, no log output to log/chrome_headless.log anymore.

while (group_count = instance.css('ul.classify-list li').count) < 2
  sleep 5                         # <= it keep infinite loop here when run on vps.
  instance.screenshot(path: '1.png')
end

Thank you.

[route]

Does this website has some kind of a protection? It could block ip ranges of your vpc. Did you try xvfb on the server?

[zw963]

Does this website has some kind of a protection?

Don't know yet.

It could block ip ranges of your vpc

In fact, my first VPS is IP from HongKong, and second is come from america, it maybe not IP issue.

Did you try xvfb on the server?

Not yet, i will

[route]

From the video I can see that the website is loaded so it's not IP protection, but you could try to turn images on and move mouse a bit around or scroll a little bit. Some websites have content lazy loaded, it might help

[zw963]

Hi, i don't know what's happen, i guess i almost never change anythings, it seem like works now.

After that, i do some refactor to reduce code like browser.visit('???') in my scrap script, i find those type script
very easy to cause issue, becaue the site i am scraping is a websocket live refresh site, previous solution
i login from one page, and then visit to main page, which maybe the reason of many issue.

Anyway, i change to only visit one time at first, and then check login model window, it seem like work well.
i need more times to check, if this issue happen again, i will report here.

Thanks for help.