Documentation/manual/reference that unambigously defines operator precedence and associativity, operator arity etc

[amn]

As someone who's spent the past two weeks trying to decipher the fundamental pattern of behaviour behind yq (and jq, to a lesser degree) -- having grokked it now almost to my satisfaction -- I must confess I think a form of documentation appendix that functions as reference for certain things current manual rather informally (through examples) covers, would be a great addition.

For instance, one may deduce that + in yq functions much like in jq (which specifies it more formally), although nothing really explicitly refers to jq documentation so it isn't clear that the former functions as the latter unless one is aware of the relationship between the two tools.

Assuming one is not aware of the relationship, or assuming one hasn't understood what exactly jq calls a filter vs. an operator vs. one vs. multiple result -- and with yq keeping silent on these things as well -- one can't expect people's intuition to make up for this much lack of specification, in my opinion.

For example, I have deduced that select expects multiple inputs, if I define something called "multiple results" and "multiple inputs" where an input is a [YAML/JSON] node and "multiple" means "a collection of [inputs]" but specifically not a [YAML/JSON] sequence/array (here jq seems to either be stepping on own toes referring to arrays liberally, implying both a multitude of results/inputs and a JSON array). This is very confusing to someone about to strap either tool on their belt. I certainly have spent hours before I understood this crucial difference. [ "foo", "bar" ] | select(. == "foo"), for instance, does not produce two results where 'bar' is filtered from the array being piped in to select! This much may be obvious to someone who now understands yq just as I do, but to a beginner it's almost completely counter-intuitive, and they don't understand (yet) the distinction between a yq processing sequence and an array specified in the document. In the former example, one single value that is the array is indeed piped to select but the latter actually selects based on the whole array, meaning the selection is done akin to "do I allow the entire array through as one single output element in the result, based on the predicate . == 'foo'?"). That will do the wrong thing. Obviously, the right thing to do is to "splat" the array before passing it on to select, with "splat" being another alien term yq seems to borrow from Ruby or Go of which I am expert at neither, and drop straight into the manual without giving it much further thought of definition. In case of the example, the right thing to do is [ "foo", "bar" ] | .[] | select(. == "foo").

Also, one of the most important constructs in the query language used in yq, the pipe operator (is it a "filter"? a "function"?) |, is again not explained very well, in particular with regards to its precedence (other than mentioning it's "very low") and, as importantly, its associativity, a term that would for an operator define whether it "binds left" or "right". The example with Sally and Bob in yq documentation is very interesting but I still don't understand it.

To sum this argument of mine up -- it would greatly help, IMO, if yq manual went more or at least in addition to what is there today, in direction of more formally defining the elements of its potent query language. It (the latter) deserves as much, given how it's a context-free language in its own right and is thus as complex as JavaScript. It conversely does not do it justice to leave it defined through examples and very loose terms that are being used interchangeable like "operator", "function", "array", "map", etc.

Without a formal definition, I feel like I am stumbling in near-dark, forced to re-read Mike's awesome incantations that only he, the true master of yq, can design.

Another example is the tag function (operator? filter?) where it's allowed to write . tag = $value but not . tag, former apparently being its own form of assignment operator tag = of arity 2 (a binary operator like arithmetical +) and latter being a different operator called tag which has arity of 1 (an unary operator) and syntactically expects a value on the left. The documentation seems to gloss over this or at least imply it's a given or should be apparent.

I am also bringing this up because I think noone else should spend two weeks going through the same I did. Because I think in general syntactically-aware utilities like yq are a god-send vs. error-prone text-wrangling, so I'd like to make most use of yq and the like.

[mikefarah]

Yeah I agree - I've struggled to describe how it all works succinctly and clearly (and in fact I struggled to work out jq in the beginning till one day it 'clicked').

That said, from your description, I think it would help if I had some sort of terminology page with examples and diagrams. Open to suggestions / content to make it easier to learn...

[amn]

To be clear, in case it wasn't -- I really really like the idea of yq, so thank you for bringing it to implementation. Had I not cared as much, I wouldn't have made this an issue at all, I guess. It's having so much potential of expressive query power and not fully understanding it, that frustrates me.

I have since the original post above, put in more effort into "charting" the query language so it "clicks", as you put it -- writing an experimental parser and evaluator to test if my assumptions are correct. Once that's done I can either offer a third-party manual (with illustrations generated from code) or in any other form or package that may help "spread the yq gospel".

Thank you.

[jandubois]

Hi @amn, did you ever finish your work on documenting precedence, associativity, etc?

I think it would be really useful to share, even if it is not yet complete!

[amn]

Wish I could say I did, @jandubois, but I never even got started properly.

I do have the beginnings of some "YQ-compatible" tool I wrote to test things, but looking at it now I am not even sure it works, I may have gradually abandoned it at the time. It's what I was referring to with "experimental parser" earlier. I don't think it's of any use to anyone in its current state.

We're mostly using yq at our work, and by "we" I mostly mean myself -- lack of adoption hasn't exactly helped me to spend more time with the tool, as much as I myself love using it, even for what I consider all the raw edges. And lack of adoption has frankly been, in my view, due to people not wanting to pay the cost of learning YQ and its relatively complex query language, for getting semantic data analysis (a factor I personally value enough to have learned the tool). At our work, everyone just uses Jinja, wrangling YAML as plaintext, yikes. But I digress.

I am afraid you'll have to get at the precedence and associativity through sheer experimentation, but I'll let someone else correct me here.

[jandubois]

As in most other language, the way to deal with unknown precedence and associativity is to use brute force: add more parenthesis until there is no ambiguity left. It also makes your code more robust, in case the implementation changes in the future (either accidentally, or to bring it more in line with jq).

Ask your Lisp/Clojure developer friends; they will tell you it looks more beautiful anyways. They can also hook you up to buy parenthesis in bulk. 😄

[jandubois]

@amn BTW, if you don't care about maintaining comments in your transformations, then you may want to take a look at YAMLScript. It is still relatively new, but uses a more traditional/procedural language paradigm, and has more expressive control structures. You can use it similar to yq and jq; minimal example at https://yamlscript.org/doc/query.

This is Ingy's latest project. He was one of the creators of YAML back in 2001...