No results returned from some React websites

[opsdemon]

Hi,

I've been using HtmlUnit for the last few weeks and it really is a great tool - very flexible with a nice intuitive API.

In the last few days I've encountered issues trying to scrape the following 2 websites:
https://www.elastic.co/blog/
https://blog.bitrise.io/

Htmlunit: 2.35.0

Both sites seem to be using React which I suspect is related to the problem.

Here is the code to reproduce the issue:

	WebClient client = new WebClient();
	client.setJavaScriptTimeout(30000);
	client.getOptions().setTimeout(60000);
	client.getOptions().setRedirectEnabled(true);
	client.getOptions().setThrowExceptionOnFailingStatusCode(false);
	client.getOptions().setThrowExceptionOnScriptError(false);
	client.getOptions().setCssEnabled(false);
	client.getOptions().setJavaScriptEnabled(true);
	client.getOptions().setPrintContentOnFailingStatusCode(false);
	client.getCookieManager().setCookiesEnabled(false);
	client.getOptions().setUseInsecureSSL(true);
	client.getCurrentWindow().setInnerHeight(60000);
	client.setAjaxController(new NicelyResynchronizingAjaxController());

	// Example 1
	String url = "https://www.elastic.co/blog/";
	String selector = ".main-wrapper .blog-column";

	// Example 2
	//String url = "https://blog.bitrise.io/";
	//String selector = ".articles .article-container";

	HtmlPage page = client.getPage(url);
	client.waitForBackgroundJavaScript(30000);

	DomNodeList<DomNode> results = page.querySelectorAll(selector);
	System.out.println("Found "+results.size()+" results");

Both examples always produce zero results even though if you click "Inspect" on the pages the CSS classes are all present.

I've tried changing the client options based on what I've read from previous issues but none of the changes made any difference.

I'd really appreciate some pointers as I've spent hours trying to crawl these pages but couldn't make it work.

Thanks,

Gerald.

[rbri]

see #755