OAD Structure and referencing in OpenAPI 3.1

[mikekistler]

We've been having a lot of discussions about document structure and how OAS and JSON Schema references work in various places, such as

• In recent TDC meetings
• In PR Fix description vs document terminology #4100, which is for 3.0 but winds up discussing how 3.1 works in places
• In Slack, particularly this looooong thread in the spec channel

I think we can continue discussing the 3.0 behavior in #4100, but (to the degree that these can be separated) thought it would be a good idea to start a discussion focused primarily on the behavior in 3.1, which I think most folks feel is what we want going forward.

[mikekistler]

I'll try to summarize my understanding of how document structure and referencing works in OAS 3.1, quoting the 3.1.1 spec or borrowing words from @handrews and others in places.

From the 3.1.0 spec, the general concept is this:

An OpenAPI document MAY be made up of a single document or be divided into multiple, connected parts at the discretion of the author.

As for how references work, the 3.1.0 spec says this:

Unless specified otherwise, all properties that are URIs MAY be relative references as defined by RFC3986.

which was also the case in OAS 3.0, but the difference in 3.1 is that URI references are resolved differently. In OAS 3.1:

Relative references [in OAS fields --not in Schema Objects] are resolved using the referring document as the Base URI according to RFC3986.

and

Relative references in Schema Objects including any that appear as $id values, use the nearest parent $id as a Base URI, as described by JSON Schema Specification Draft 2020-12.

There's more to the story but I'm going to save this and amend with more content later.

[mikekistler]

Thanks @handrews -- I think you have confirmed that my understanding is correct. Which means that in OAS 3.1 $refs outside of Schema Objects work completely different from $refs inside Schema Objects. I'm surprised by this but this whole topic has been a huge education for me.

[handrews]

@mikekistler

Which means that in OAS 3.1 $refs outside of Schema Objects work completely different from $refs inside Schema Objects

Hmm... that's not really how I think about it. We have 8 fields that use URI-references. Six of them are what I think of as "normal" URI-reference fields:

• Reference Object $ref
• Path Item Object $ref
• Schema Object $ref
• Link Object operationRef
• Example Object externalValue
• External Documentation Object url

There are then two fields that require a preprocessing step:

• Discriminator Object mapping, which has to first figure out whether the value is a component name of a URI-reference, but if it's a URI-reference it is otherwise normal
• $dynamicRef, which we've discussed elsewhere and requires a pre-processing step to determine the resource that provides the base URI — this is weird and I regret it and it has no impact on how anything else behaves at all, so I'm going to ignore it for the rest of this comment.

So let's think of our seven normal URI-reference fields (the six totally normal ones, and the URI-reference syntax of mapping). There are three aspects of how these function:

1. Determine the base URI, which is always done in accordance with RFC3986, which allows customizing the first possible base URI source (per §5.1.1) based on the resource's media type, after which §5.1.2 – 5.1.4 are the same for all media types
2. Resolve the URI-reference against the base URI to produce a full URI (if the URI-reference is not relative, this is a no-op, but technically still occurs, and again every aspect of this is defined by RFC3986 and we do not change it- none of our fields other than $dynamicRef even attempt to change this, with preprocessing steps or otherwise)
3. Handling the reference target: This is where these fields differ, in terms of how adjacent fields are handled and whether there is an inline equivalent or whether the effect is purely delegation. This is the only aspect in which the fields differ (assuming we've already determined mapping is using the URI-reference syntax)

Where you seem to be seeing a difference is in the base URI determination, but these are just determined by the (unregistered) media types (with the YAML equivalents behaving the same as the JSON ones):

• application/openapi+json, as of OAS 3.1, does not define a §5.1.1 mechanism for base URI determination, leaving it entirely determined by §5.1.2-5.1.4 (as of OAS 3.2, it will define self in the OpenAPI Object as its §5.1.1 mechanism)
• application/schema+json uses $id in a schema resource root as its §5.1.1 mechanism, and allows embedding multiple schema resources in a single document (every schema document is a schema resource, but not all schema resources are separate schema documents)

But none of that changes how any $ref or any of these other "normal" URI-reference keywords function. They all just resolve against the base URI, whatever it is. Figuring out the base URI and resolving a URI-reference against it are separate steps: the first is impacted by the media type under §5.1.1, and the second is absolutely standard in all cases for these seven fields.

[mikekistler]

Hmmm ... now I think you are disagreeing with my earlier statement. Maybe I need a better / more complete example. I'll work on that.

[mikekistler]

Okay ... maybe I don't need a more complete example, as I think I see where my confusion is.

I think the key is understanding what "the referring document" is this statement:

Relative references [in OAS fields --not in Schema Objects] are resolved using the referring document as the Base URI according to RFC3986.

"document" is unfortunately a very generic term and could mean many things.

It seems reasonable to interpret "document" to be "JSON document". And the JSON Schema 2020-12 spec does define "JSON document" and in particular says this:

In JSON Schema, the terms "JSON document", "JSON text", and "JSON value" are interchangeable because of the data model it defines.

So if the entry document referenced "otherDoc.json#/components/parameters/myParam.json", it seems reasonable to say that the value of that fragment, which is a JSON value, is "the referring document" for any "$ref" within it.

I acknowledge that there are other, maybe better interpretations, but I'd like to know if there is any reason that this interpretation should be considered invalid.

[handrews]

@mikekistler

It seems reasonable to interpret "document" to be "JSON document". And the JSON Schema 2020-12 spec does define "JSON document" and in particular says this:

In JSON Schema, the terms "JSON document", "JSON text", and "JSON value" are interchangeable because of the data model it defines.

This is about the data model, and has nothing to do with resources vs documents.

So if the entry document referenced "otherDoc.json#/components/parameters/myParam.json", it seems reasonable to say that the value of that fragment, which is a JSON value, is "the referring document" for any "$ref" within it.

A reference connects a referring document to a referenced document. The "referring document" is just the document containing the reference.

I don't know who added that term to 3.1 because Ron squashed a ton of commits when releasing the spec and obliterated a lot of the history, so I can't track down the historical context. But this is not some strange new concept. Document A refers to document B (using a $ref), therefore document A is the referring document.

@mikekistler A reference connects from a referring document/resource to a referred or reference document/resource (really, it should be resource, but in most cases they are the same, just not in JSON Schema... or in the multipart/related document structure with Content-Location that I mentioned earlier... or if the "document" is a YAML stream...)

[miqui]

tdc notes: resource , documents, and description. (rework, should be addressed before patch release?)

[mikekistler]

We had a lively discussion of this topic in the TDC meeting yesterday.

Trying to tease apart what is meant by "OpenAPI Description" vs "OpenAPI Document" has been challenging.

And connecting these to related terms used in the OAS or referenced RFCs, such as "resource", "entity", adds yet more complexity.

There were differing views on what should be considered "the referring document" in this sentence taken from the OAS 3.1.0:

Relative references [not in Schema Objects] are resolved using the referring document as the Base URI according to RFC3986.

There was the view I presented in the comment above, which is basically that the JSON value found at the fragment could be considered the referring document (ignoring the outer "document" in which it was contained).

Henry argued that this interpretation is invalid -- that the referring document must be considered to the document/resource referenced by the "absolute" part of the URI (fragment removed). As support for this view, Henry pointed out that fragment processing must be done based on the MIME type of the resource and "application/json" has no defined fragment identification syntax, citing RFC 6901 (section 5). As such, referencing a fragment in a "plain JSON" document is undefined and (as spec writers) we should disallow it.

At the end of the meeting we agreed to attempt to formalize the "description", "document", "resource" terminology and then use that as a way of clarifying this whole area.

[handrews]

@mikekistler yes, agreed, primary and secondary resources are both resources, thanks for that correction.

My definition and the JSON Schema definition are both in terms of "series of octets" and, as far as the first sentence in the JSON Schema spec definition is concerns, are consistent with each other. The second sentence is making a point about terms in the JSON RFC and is not relevant to the OAS.

which by the definitions above implies that all relevant documents are also actually resources.

I think I agree. Not 100% sure I'm not missing something, but I think I agree here.

can the document/resource/content at that reference be processed on its own, or must it be interpreted within the context of the larger primary resource in which it is contained?

You can only process a secondary resource by determining the media type of the primary resource and applying that media type's fragment rules. The fact that the OAS (and before that, JSON Reference) attempt to override that is irrelevant. RFC3986 and other approved RFCs take precedence over random things someone attempted in a decade-plus-expired draft that was never reviewed properly.

[darrelmiller]

You can only process a secondary resource by determining the media type of the primary resource and applying that media type's fragment rules.

Where does that leave us when a media type, e.g. application/json does not provide any description of how to process fragment identifiers? I think it suggests that using $ref to point to fragments of external documents is not valid. That doesn't seem like a very tenable outcome.

[handrews]

@darrelmiller I didn't make the rules, although I don't see the problem. You need to reference an application/oas+json or application/schema+json document to use the fragment syntax (and if you don't have a Content-Type header you can infer the media type from other cues including extensions and content-sniffing). For JSON Hyper-Schema, in order to use fragments with instance documents, we invented application/schema-instance+json.

[mikekistler]

I'm not sure any rules are being broken by interpreting a fragment as a JSON pointer for JSON documents.

RFC 6901 just says

In particular, the fragment identifier syntax for application/json is not JSON Pointer.

This is a true statement -- there is no defined fragment identifier syntax for application/json in the IANA Media Types registry entry for application/json.

But I found no language in RFC 3986 that prohibits applying a fragment identifier syntax in the event that the media type does not define one. I did find this statement:

The fragment's format and resolution is therefore dependent on the media type [RFC2046] of a potentially retrieved representation, even though such a retrieval is only performed if the URI is dereferenced. If no such representation exists, then the semantics of the fragment are considered unknown and are effectively unconstrained.

I'm having a little trouble parsing this but in any event it is clear that in some situations RFC 3986 allows the fragment to be interpreted in an "unconstrained" way.

tl;dr: The OAS says "here's how you handle this otherwise undefined situation" -- I don't see that as breaking a rule.

[handrews]

@mikekistler We are defining media types for OAS and JSON Schema. They are not properly registered, but conceptually it all works. We don't need to come up with novel ways of handling this.

The only thing that definitely goes against things is disregarding a media type, including application/json. Although I consider it perfectly valid to refine application/json to application/oas+json (or whatever it is supposed to be) by content-sniffing. But it's the "I'm not even going to acknowledge that this primary resource has a media type so I can do weird non-standard things with extracting fragments" that is a problem. That's not a case where things are ambiguous and you're filling in details. That's willfully ignoring the information that is there.

[darrelmiller]

@mikekistler

which (to me anyway) seems to imply that only documents that can be "referenced" are relevant,

I'm not sure you should correlate "referenced" with the use of "$ref". There are other ways to reference documents from an OpenAPI description. My interpretation is that a $id that identifies a schema in another document effectively "references" that document.

So we have to allow fragments to identify "referenced documents" within YAML files

This statement makes me very squirrely. I don't like the idea of using the word "document" for fragments of a "physical" document.

I have to conclude that the only requirement for processing the primary resource is for resolving the fragment to obtain the referenced resource, and that resource (and only that) is the "referenced document" that becomes part of the OpenAPI Description.

Is the purpose of this statement to try and clarify that the server of a referenced operation comes from the entry document and not the host document? (and other things like tags and security).

[mikekistler]

@darrelmiller

So we have to allow fragments to identify "referenced documents" within YAML files

This statement makes me very squirrely. I don't like the idea of using the word "document" for fragments of a "physical" document.

I think this is part of the terminology we need to work out. It seems that you would prefer to use the term "fragment" for "secondary resources within a primary resource" (as RFC 3986 would put it). I suppose we could do that. That might require a scrub of the spec to determine where current uses of the term "document" should instead be "fragment", though a quick scan suggests there wouldn't be many of these.

But a "fragment" would still be logically a JSON object, right? Which by the JSON Schema terminology is a "JSON Document". So we start to chase our tails a bit if we want to distinguish between "document" and "fragment".

And isn't it possible, for schema objects anyway, to convert any "fragment" into a "document" by adding a "$id" with an absolute URI to it? So again, the same thing can be either a document or fragment, depending only on how it is referenced (I think).

Is the purpose of this statement to try and clarify that the server of a referenced operation comes from the entry document and not the host document? (and other things like tags and security).

Yes but even more relevant is that if there happens to be an openapi element in the "primary resource", it should have no bearing on how a "secondary resource" -- "fragment" in your terminology -- is processed.

[handrews]

@mikekistler I'm losing track of what you're trying to do. I keep trying to offer an RFC-based, standards-compliant, coherent view of the world and it feels like you are looking for some sort of alternative.

Why? Why not just go with what fits with everything else? It's one thing if there's a point I'm making that is unclear, but it is starting to feel to me like you're hunting for places where you can find alternate interpretations. Which maybe I could help with more if I understood your goal. Right now I feel like you're tossing out a lot of speculative possibilities and I can't figure out how to respond because I don't know why.

Particularly because this is the discussion about 3.1. I feel very strongly that 3.1 works as-is. The only confusions come from 3.0 (and 2.0). There's nothing about URIs or media types or fragments that need fixing in 3.1. The whole "extract a JSON object out of an unknown document" thing will never work in 3.1. We already agreed it wouldn't. There is cleanup to do around "document" but there's no need for a new conceptual framework in 3.1.

So what is it that you're trying to find with all of this?

[handrews]

And isn't it possible, for schema objects anyway, to convert any "fragment" into a "document" by adding a "$id" with an absolute URI to it? So again, the same thing can be either a document or fragment, depending only on how it is referenced (I think).

This confuses document and resource. An $id in a subschema makes that subschema into a resource. A document is a coherent, finite sequence of octets, like a file. You don't point to a subset of a file and also call that a file no matter what keywords it is using. A file is a chunk of storage of some sort.

[handrews]

@mikekistler @darrelmiller did we sort this out sufficiently? Do we need to keep this open? (closed discussions, like closed issues, are still easily find-able).