TravisCI Time Limit

[thebigG]

Hi everyone.

Description

TravisCI has a time limit of 50 minutes for each job as per their documentation. And as you can see on my last push thebigG@2417f6e the build on TravisCI fails because one of the scrape runs hits that 50 minutes cap.

One possible solution I have thought about is adding a number_of_pages/number of jobs limit to each scraper, but this might add complexity to our settings file and Command Line Interface so I'm just posting this issue to start the discussion on possible solutions to tackle this issue.

Another thought that comes to mind is limiting the scraping we do when testing, but I don't think that's optimal because we want our CI pipeline to be reliable and trustworthy, so at the very least we should be doing doing some scraping for every provider, even if it is just one page per provider.

Like I said, just wanted to start the discussion of potential solutions.

Steps to Reproduce

1. Run TravisCI script

Expected behavior

The tests should run without TravisCI intervention.

Actual behavior

The JobFunnel tests stop and fail because of the time limit.

Environment

• Build: JobFunnel 3.0.1 on commit cf80740
• Operating system and version: Pop!_OS 20.04 LTS
• [Linux] Desktop Environment and/or Window Manager: Cinnamon

[PaulMcInnis]

Unfortunately travis CI also now uses a credits system and I don’t want to pay, so we will eventually run out of them.

[thebigG]

Oh man that makes sense, I thought that I had read something about that the other day. We'll figure something out.

[markkvdb]

Yeah I read the controversy online. Seems like we should just switch to a different provider such as Github Actions or Jenkins.

[thebigG]

I've been investigating Github Actions and it is completely free for open source projects and it is very similar to TravisCI. There is a yaml file that describes the CI process, but the syntax is just slightly different of course. I'll keep on investigating further and give you guys feedback on what I can find.

[thebigG]

UPDATE: I have Github Actions all setup! You can check out the GitHub Actions configuration on this branch:https://github.com/thebigG/JobFunnel/tree/isolate_remote.

I came across a hurdle. Which was, for some reason, JobFunnel will fail to scrape Indeed with max_listing_days set to 35. I set it to 3 days, and everything passed now. @PaulMcInnis @markkvdb Let me know if there is anything you would like changed before I make the PR.

[PaulMcInnis]

Thanks a bunch @thebigG

Lets open that PR and cut an issue for the scraping, since it sounds like this is easily replicated, will make it easier to fix.

[thebigG]

Has been resolved on #127