You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As I worked with a scalable web crawler (apache nutch), to scan my serverlist and outgoing links, I noticed that you didn't forbid crawler to scan the avatars.
I would suggest that you do so in robots.txt, to avoid unnecessary traffic from other crawlers (The problem is that you don't use the query syntax like ?avatar=, so the crawler thinks the avatars are individual pages).
e.g.
User-agent: *
Disallow: /avatars/*
The text was updated successfully, but these errors were encountered:
I guess the toss of up is do we see any benefit from being indexed at the image level? That's probably (?) a no, but I would be open to opinion on that.
As I worked with a scalable web crawler (apache nutch), to scan my serverlist and outgoing links, I noticed that you didn't forbid crawler to scan the avatars.
I would suggest that you do so in robots.txt, to avoid unnecessary traffic from other crawlers (The problem is that you don't use the query syntax like ?avatar=, so the crawler thinks the avatars are individual pages).
e.g.
The text was updated successfully, but these errors were encountered: