Compatibility between implementations

[adamretter]

I had been thinking about how we can release proposals or unleash experimental syntax features on our users without sacrificing the portability of XQuery code between implementations.

I hadn't formulated any particularly strong ideas, apart from to note the obvious, that the XQuery Version Declaration is meant for that purpose.

Previously xquery version "1.0; very much meant W3C XML Query Language version 1.0.

Today I note that BaseX has released version 9.1, and for us, it is very exciting to see that they have included XQuery syntax extensions for: ternary if, elvis operator and, if without else.

It is not clear for me from their documentation if these are enabled by default, or can be enabled and disabled by some configuration.

My concern is that BaseX users will happily use these functions under the xquery version "3.1"; declaration, but by doing so their code become incompatible with other implementations, we have many users out there that learn from each other by sharing XQuery code between implementations. Furthermore, the features that BaseX has added are not even yet accepted "Proposals" here.

Now I want to be clear, I am not criticising BaseX in any way, I have the upmost respect for the product, @ChristianGruen and the team. I congratulate them on their new release and adding these wonderful extensions.

For the sake of portability, I wonder if we need to create the notion of xpath-ng profiles, where we can say define something like:

xpath-ng profile 20181101

Syntax extensions from the following proposals:

1. https://github.com/expath/xpath-ng/blob/profile-20181101/conditional-expressions.md
2. https://github.com/expath/xpath-ng/blob/profile-20181101/ternary-if.md
3. https://github.com/expath/xpath-ng/blob/profile-20181101/if-without-else.md

Of course none of the above are at "proposal" level yet, so we would need to get there first! However the idea would then be that implementations could do something like:

xquery version "3.1+xng20181101";

Multiple profiles would be allowed if necessary, e.g.

xquery version "3.1+xng20181101+xng20190101";

Thoughts?

[ChristianGruen]

Hi @adamretter, thanks for summarizing the syntax extensions that we’ve already made available to the users of our software.

Our extensions features (including the ones for updates and full-text, which @rhdunn has summarized very nicely for the XQuery IntelliJ Plugin) are all enabled by default. However, it’s worth mentioning that the syntax of BaseX-specific extensions may change with new versions of our software. This is particularly true for features that have been added to a fairly recent version. We planned to add the three discussed extensions to our software anyway, and it was a fortunate coincidence that the xpath-ng discussion started just recently. If it turns out that some of the suggested features will be added as xpath-ng proposals and eventually accepted, and if it turns out that the syntax and semantics must be changed, we’ll update the implementation of the extension features as well.

With regards to portability, I agree with you that the XQuery Version Declaration would be the canonical place to define new versions, and your approach looks promising and flexible enough to me.

From a practical point of view, my impression is that nearly all non-trivial XQuery code relies on processor-specific features anyway, so I would already be happy if we manage to get many of the discussed proposals accepted, and if these extensions were made available by as many processors as possible.

[michaelhkay]

"From a practical point of view, my impression is that nearly all non-trivial XQuery code relies on processor-specific feature anyway"

I find it difficult to argue with this because I actually see very little non-trivial XQuery code. I would guess that 95% of Saxon use, both trivial and non-trivial, is XSLT. And the situation there is rather different, since (a) we're in a dominant market position for XSLT 3.0, but at the same time (b) people do want interoperability between our processor and Altova's. Traditionally the expectations of compatibility/interoperability in the XSLT world have been very much higher than for XQuery, but this is declining a bit now there's less choice of processors.

Our approach has been that if you want to use language extensions (beyond the scope of extensibility mechanisms defined in the spec, e.g. extension attributes in a different namespace) then you have to explicitly enable them at API level. The other advantage of that is that (as in Christian's case) it's easier to label extensions as "experimental" and allow the spec to evolve in response to feedback over a couple of releases.

[rhdunn]

I'm happy with using the version label to reflect the EXPath extensions provided that they don't get too wieldy. As such, it would be easier if they were limited to a single profile string extension (otherwise you could end up with a large number of permutations and very long and hard to read version strings, especially if the number of profiles grows over time).

Note that the version string would apply to XSLT/XPath.

Most of the XQuery code I have written is targetting the MarkLogic XQuery database, and relies heavily on their XQuery extensions. I have also written some XQuery code for other XQuery processors that does not make use of vendor syntax extensions, but does make use of vendor specific functions. So I would agree with Christian's assessment.

For my IntelliJ plugin I aim to support all known syntax in the parser and report conformance issues in static checks that get run after the AST is generated. This allows me to handle different XQuery versions, W3C extensions (update, full text, scripting), and vendor extensions using the vendor, processor version, XQuery dialect (xquery, update facility, full text, vendor extensions, etc.), and XQuery versions that all affect what syntax is valid.

[benibela]

I have been using - as separator in the version. xquery version "3.0-xidel"; or xquery version "3.0-jsoniq";. I think marklogic had - too

[liamquin]

An alternative might be a prolog option to enable or disable specific sets of extensions.
declare option syntax-extensionsenabled := (bx:elvis, bx:ternery-if);