Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved crawler logging #5175

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Murgeye
Copy link
Contributor

@Murgeye Murgeye commented Sep 15, 2022

When using --crawl, some logging of Connection and TLS errors lacks information (i.e., the current URL or host and port) to be useful. This pull request improves the logging with this additional information.

@stamparm
Copy link
Member

...%s" % repr(url) looks bad
...'%s'" % url looks better

@Murgeye
Copy link
Contributor Author

Murgeye commented Sep 16, 2022

I have updated the PR with the requested changes.

@stamparm
Copy link
Member

ok. now went through changes. there is lots of noise here AFAICS. lots of redundant information with URLs. i could keep those .debug(...), but everything else seems to me will bring just noise

@Murgeye
Copy link
Contributor Author

Murgeye commented Oct 21, 2022

ok. now went through changes. there is lots of noise here AFAICS. lots of redundant information with URLs. i could keep those .debug(...), but everything else seems to me will bring just noise

I can see that this adds a lot of noise for non-crawler execution and I could put this into a second debug logging call.

However, that would mean either requiring repeating the information from the current log or (due to multithreading) risking that the error information and the URL/host:port get seperated by other logging calls in between.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants