In browser KWIC indexer

[DocOtak]

I'm making this issue under the github label since it is not a standard name request.

I've opened PR #228 that adds an in browser KWIC indexer. This is an attempt to solve two problems we currently have:

• The KWIC indexer is broken or not runnable, the most recent standard name table didn't have a KWIC index generated for it
• The generated KWIC index html is quite large (8+MB) and we are approaching some limits github has for repository and deployed website sizes.

Additional features made possible:

• Can generated a KIWC index for every version of the standard name table
• The ability to download the generated HTML for offline use (or hosting someplace else)
• No need to update the indexer with each new version of the standard name table, only publishing the XML.

Attn @JonathanGregory

[JonathanGregory]

Thanks for doing this useful work and opening the issue to describe it. Since at this stage it's a test of a new facility, I think it's fine to merge it and see if it works as expected! Please go ahead.

[DocOtak]

Now that I merged in the #228, the KWIC Index is available at: https://cfconventions.org/vocabularies/kwic-index.html

Because this is hosted on the CF website, versions of the standard name table do not need the full URI and can be shortened from my original discussion:

• Version 20 example: https://cfconventions.org/vocabularies/kwic-index.html?tableURI=/Data/cf-standard-names/20/src/cf-standard-name-table.xml

I find the version on the CF website to be quite performant.

@JonathanGregory Is there an idea with how this vocabularies repo might be structured? I could put together a straw man demo that combines that KWIC indexer and the stylesheet proposed in: #231 (including file downloader). I'm tempted to do this just to see what kind of file space we'd be looking at.

Even if we end up hosting static HTML, these JS/XLST implementations and static download links I think might be very useful for the workflow. Since it just needs a browser to run, it could be how all the static files are generated.

[larsbarring]

Just echoing what Jonathan writes -- this is impressive. When I load the kwik page (using the link you give), it takes about 7-8 seconds, which I think is very acceptable given the content. I have just made a comment regarding the XSLT file over in vocabularies#231.

[DocOtak]

@larsbarring I was looking into the rendering times when building it. The static KWIC indexes on the CF website take just as long to render, but can do so in a streaming way. So the top is displaying to you as the lower parts of the page are still being rendered.

I liked the idea from @JonathanGregory of having the "current" KWIC and CF standard name tables be static, and all the previous version be just the XML with the XSLT (name table) and Javascript (KWIC Index) to display them. My napkin calculations have this reducing the total amount of data in vocabularies to around 200MB (from ~800MB), but I want to actually try it. I think this would allow about 100 more standard name table version until the size limits become a problem again.

[larsbarring]

Yes, I agree 👍 with @JonathanGregory and you regarding have the current version static and the other ones generated.

[JonathanGregory]

@DocOtak, many thanks for your continuing work on this. I suggest that it would be helpful if you and the vocabularies team (Alison @japamment, Fran @feggleton and Ellie @efisher008) could discuss the appropriate setup of the vocabularies repo. Others might have views as well, such as Lars @larsbarring, arising from his previous work on the standard name table.