Overhaul of the Manual

[01mf02]

In the last days, I've completely overhauled the jq manual. Some of the most important changes:

• I've regrouped the chapters. I tried to make it such that the chapters build on each other; for example, to explain basic filters such as |, it is useful to know how to construct values first, and to understand paths such as .posts[0], it is useful to understand | first.
• To help with finding the information, the TOC can now be expanded to show subsections of sections.
• I've annotated many filters with compatibility notes that document divergent behaviour for other implementations. Currently, this covers mostly my own jq interpreter jaq, but I expect this to be useful too for gojq.
• I've reworked the visual appearance of the man page, such that longer blocks (such as examples or notes) are visible as such.
• I've corrected bugs in the manual as I went along; for example, the behaviour of arithmetic-update assignments.
• I've also added a large number of new examples.
• I significantly reworked the sections on paths, definitions, and assignments. These are nearly rewrites.

All in all, this PR adds a lot of new information to the jq manual and tries to improve its structure.

To see it in all its glory, you need Pandoc >3.0. Go to jq/docs, run pipenv sync, then

pipenv run python3 build_website.py --root /output && python3 -m http.server

and open http://0.0.0.0:8000/output/manual in your browser.

Please let me know if I can do something to get this merged.
This builds on and supersedes my previous PR (#3186).

[01mf02]

@itchyny, thanks a lot for your improvement suggestions!

Example styles are lost; the Examples link should be in var(--bs-secondary-color) and the outputs should be aligned each lines. The Run buttons lack the border and target="_blank" because they are links to the external website.

I adapted the run buttons and the examples links (although adapting the latter to precisely match the current manual would probably take quite a bit more time, just let me know if the current solution is an issue). Concerning the output alignment, I'm not sure whether I understand the issue, because outputs seem aligned to me:

Anchor link icons are lost, this is important for users to notice we have anchor links each sections and filters. Also, I want to keep the existing anchor links; #recurse to the recurse filter section.

I added the anchor link icons. I also tried to preserve more of the existing anchor links; however, this is sometimes not possible where the structure between the old and the new version has changed too much. Still, much more links should match their old IDs now. (Including your example, #recurse.)

[01mf02]

@01mf02 - A tiny enhancement request whenever you get around to it:

The very first line in the subsection on Modules currently reads:

jq has a library/module system. Modules are files whose names end in .jq.

I think it would be very helpful to add something to the effect that a module cannot have a "main" program. That should make it clear that not every .jq file is a valid jq module, while also making it clear why the word "module" is used.

Good suggestion. Let us do this once this PR is merged. I'm reluctant to make content changes at this point.

[itchyny]

How can you parse .foo?? in this definition?

[01mf02]

.foo? is a path, and .foo?? is equivalent to (.foo?)?; that is, catching all errors from the path .foo?. We can see this by the following:

$ jq -n '.[error]?'
jq: error (at <unknown>) (not a string): null
$ jq -n '.[error]??'
$ jq -n '(.[error]?)?

(Here, the last two commands do not output anything, because the second ? silences all errors of the path, including the error from error.)

[itchyny]

A path should be followed by part before ? in your definition, but .foo?? cannot be parsed because part is missing. If any path can be followed by ?, it should be path = path "?"?

[itchyny]

I think these are called string formats. They can be used like filters, but when placed before strings they can format interpolations, so different concept from functions.

[01mf02]

The jq 1.7 manual says: "The @foo syntax is used to format and escape strings [...]", but it never really gives an explicit name for @.... You are right that they are not completely like functions, because they can be used before strings (unlike regular functions), but they can be also used like functions. I added a note about this to the manual now, but left the name of the section unchanged. Please let me know if you are happy with that, otherwise I can still change it.

[01mf02]

Thanks a lot for all your comments, @itchyny!

[yochem]

From @wader:

maybe generate html directly from md?

I think this is a very good idea, and a much more elegant solution than the current manual conversion of md -> yml -> html with the Python code. You could either (1) use a static site generator, like Jekyll (GitHub pages native) or Hugo (faster, more modern, bit easier to use), or (2) since pandoc is already used, just use that to convert everything from md to html and add that to the template. I think (1) is probably the easier approach. It has builtin support for TOC creation, templates, includes and more. I could create a POC for this if you want.

Another benefit would be that you can have a website repo with just the website code, that pulls it's content (the markdown files) from this repo on changes. This separates all markup and website code from this repo, while still keeping the actual documentation close to the code.

[wader]

@yochem yeap would be great to look into such things after we get this PR merged, it's already quite large and dragged on for a long time 😅

[yochem]

Yes definitely haha! I was not suggesting to include that in this PR, just a suggestion for after that. Maybe would've been better to create a separate issue for it. Sorry for the noice!

[01mf02]

I think this is a very good idea, and a much more elegant solution than the current manual conversion of md -> yml -> html with the Python code. You could either (1) use a static site generator, like Jekyll (GitHub pages native) or Hugo (faster, more modern, bit easier to use), or (2) since pandoc is already used, just use that to convert everything from md to html and add that to the template. I think (1) is probably the easier approach. It has builtin support for TOC creation, templates, includes and more. I could create a POC for this if you want.

Another benefit would be that you can have a website repo with just the website code, that pulls it's content (the markdown files) from this repo on changes. This separates all markup and website code from this repo, while still keeping the actual documentation close to the code.

I also agree with you, but the problem is that the manual for the older jq versions also need to be generated somehow. And they are in YAML. So while the current solution in this PR is a bit clumsy, it ensures backwards compatibility. If the older manuals were not around, then generating HTML from Markdown directly would become feasible.

[itchyny]

Can you apply the small class to this details tag instead of the summary tag?