NEPTUN-SCRAPER

RUN

• Install dependencies

poetry install

• Run Neptun-Scraper

scrapy crawl dockerhubDockerRegistrySpider -a query="python"

• Run Neptun-Bot (WebUI, Rest-Interface, Scrapy-Daemon)

docker compose up --build

• Access the WebUI:

http://localhost:8000

• Access the DAEMON-Interface:

http://localhost:6802

• Create a Scrapyd-Job

curl http://localhost:6802/schedule.json -d project=neptunscraper -d spider=dockerhubDockerRegistrySpider