set noindex on all content outside of https://istio.io/

[craigbox]

Google is confused about our duplicate content. We think it's all in the search index in order to power the custom search engine, but it looks like at least some that content is still available even though I've delisted it from the main search index.

That, and the facets don't actually correctly list versioned content.

My suggestion at this point is that we break site search for old versions, and then we add everything back, with a new engine if needed.

So, the first thing to do is to add noindex to the meta tags of every page on preliminary and archive.

https://developers.google.com/search/docs/crawling-indexing/block-indexing

Please do this in order:

(a) preliminary.istio.io
(b) await validation
(c) archive

[milinddethe15]

Hi @craigbox , we can include the noindex meta tag globally by updating the /layouts/_default/baseof.html file. This ensures that the tag is applied across all pages that extend from this base template.
the logic that can be added to the section of baseof.html:

    {{ if .Site.Data.args.preliminary }}
        <meta name="robots" content="noindex">
    {{ else if .Params.noindex }}
        <meta name="robots" content="noindex">
    {{ end }}

Let me know if this works?

[craigbox]

You would have to tell me if it works.

https://istio.io/v1.20/ should get noindex, but it's not preliminary. How does this work in the context of our current archiving, and how will it work in a future case?

[milinddethe15]

https://istio.io/v1.20/ should get noindex, but it's not preliminary. How does this work in the context of our current archiving, and how will it work in a future case?

For previous versions, it can be re-builded with noindex tags (we have source code in respective version branches) or add manually, lots of work.
I have created PR for adding noindex tag for preliminary. Later we can add tags in older versions. once the tags works for preliminary.