New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

`User-agent: * Disallow: /` makes yacy not index the page even if it's own user agent is allowed in `robots.txt` #636

Open

GamePlayer-8 opened this issue Apr 11, 2024 · 0 comments

Labels

GamePlayer-8 commented Apr 11, 2024

Good afternoon again,

There's an issue with YaCy robots.txt detection where if I put:

User-agent: yacybot
Allow: /

User-agent: *
Disallow: /

it will take the second definition as more important even if it's own user agent is allowed.

The text was updated successfully, but these errors were encountered:

okybaca added bug crawler labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment