Implement dereference functionality #2865

carlesarnal · 2022-10-07T09:28:21Z

Implement the dereference parameter support in the API. This support will be gradual since the dereference support is specific for each artifact type.

ai-zelenin · 2023-05-24T06:14:00Z

may i ask when this feature will be implemented? we plan use debezium with
apicurio-registry but without dereference we unable use avro decoder in golang/python(need both)

carlesarnal · 2023-05-26T07:27:36Z

Unfortunately, there is no timeframe for it. It's for sure on our backlog and we're working on a wide range of improvements for our artifact references support, but we have no dates yet,

EricWittmann · 2023-06-16T17:54:12Z

I would like to start working on this soon. @MikeEdgar this is the motivation for migrating/porting the dereference support in apicurio-data-models. But we will need to do the same for the other types we support (Avro, Protobuf, JSON Schema).

seb-pereira · 2024-04-03T17:41:33Z

Hi, is thedereference query parameter documented in the REST API expected to work with latest version 2.5.x with Avro schemas?

I am following the example from the user documentation and I can not find an endpoint that returns a schema with resolved references when the query parameter dereference is set to true.
I tried:

by global Id (ex: /apis/registry/v2/ids/globalIds/14?dereference=true)
by artifact Id (ex: /apis/registry/v2/groups/group/artifacts/Item?dereference=true)

davidradl · 2024-04-15T16:58:26Z

@seb-pereira I found the same , I upgraded to 2.5.9 which says it has support for dereference for Avro but this rest end point does not resolve at that level either.

@EricWittmann @carlesarnal please could you review

carlesarnal · 2024-04-16T06:58:28Z

I just re-tested this and I get the expected result.

I have a few artifacts registered, each with the expected references, for instance, I'm using the example here. If I let the maven-plugin do the registration for me, this results in 4 artifacts created:

Exchange, an enum, no references to other artifacts, globalId 1.
TradeKey, a record, with a reference to Exchange, globalId 2.
TradeValue, a record, with a reference to Exchange, globalId 3.
TradeRaw, a record, with two references, to TradeKey and to TradeValue, globalId 4.

If I get the content of TradeRaw straight, I get the expected result, the original content of the artifact in the filesystem:

With this path (artifactId, the result by globalId is the same) http://localhost:8080/apis/registry/v2/groups/avro-maven-with-references-auto/artifacts/TradeRaw:

{
  "namespace": "com.kubetrade.schema.trade",
  "type": "record",
  "name": "TradeRaw",
  "fields": [
    {
      "name": "tradeKey",
      "type": "com.kubetrade.schema.trade.TradeKey"
    },
    {
      "name": "value",
      "type": "com.kubetrade.schema.trade.TradeValue"
    },
    {
      "name": "symbol",
      "type": "string"
    },
    {
      "name": "payload",
      "type": "string"
    }
  ]
}

Now, if I try to get the content dereferenced using the same endpoint but with dereference set to true, the result is the expected one, the content dereferenced (nested records inlined):
Using the path (the result using the globalId is the same) http://localhost:8080/apis/registry/v2/groups/avro-maven-with-references-auto/artifacts/TradeRaw?dereference=true:

{
  "type": "record",
  "name": "TradeRaw",
  "namespace": "com.kubetrade.schema.trade",
  "fields": [
    {
      "name": "tradeKey",
      "type": {
        "type": "record",
        "name": "TradeKey",
        "fields": [
          {
            "name": "exchange",
            "type": {
              "type": "enum",
              "name": "Exchange",
              "namespace": "com.kubetrade.schema.common",
              "symbols": [
                "GEMINI"
              ]
            }
          },
          {
            "name": "key",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "value",
      "type": {
        "type": "record",
        "name": "TradeValue",
        "fields": [
          {
            "name": "exchange",
            "type": "com.kubetrade.schema.common.Exchange"
          },
          {
            "name": "value",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "symbol",
      "type": "string"
    },
    {
      "name": "payload",
      "type": "string"
    }
  ]
}

Note that for this to happen, all the references have to be appropriately registered in the server. In this context, appropriately registered means that, if I hit the following endpoint for the TradeRaw artifact, I will see the references to the other two artifacts: http://localhost:8080/apis/registry/v2/groups/avro-maven-with-references-auto/artifacts/TradeRaw/versions/2.0/references with result:

[
  {
    "groupId": "avro-maven-with-references-auto",
    "artifactId": "com.kubetrade.schema.trade.TradeKey",
    "version": "1",
    "name": "com.kubetrade.schema.trade.TradeKey"
  },
  {
    "groupId": "avro-maven-with-references-auto",
    "artifactId": "com.kubetrade.schema.trade.TradeValue",
    "version": "1",
    "name": "com.kubetrade.schema.trade.TradeValue"
  }
]

If you give me a more concrete example, I will happily test it, but to me it seems it's working as expected.

davidradl · 2024-04-16T08:06:04Z

@carlesarnal thanks for your quick response. Can I check what level we need to be on? I assume 2.5.9.
I tried to run the sample (I assume by running mvn install) and get:
[INFO] --- apicurio-registry-maven-plugin:2.4.3.Final:register (register-artifact) @ apicurio-registry-examples-avro-maven-with-references-auto ---
[WARNING] Error processing Avro schema with name TradeKey.avsc. This usually means that the references are not ready yet to parse it
[WARNING] Error processing Avro schema with name Exchange.avsc. This usually means that the references are not ready yet to parse it
[WARNING] Error processing Avro schema with name TradeValue.avsc. This usually means that the references are not ready yet to parse it
[ERROR] Exception while registering artifact [avro-maven-with-references-auto] / [TradeRaw]
io.apicurio.registry.rest.client.exception.RestClientException
at io.apicurio.registry.rest.client.impl.ErrorHandler.parseError (ErrorHandler.java:95)
at io.apicurio.rest.client.JdkHttpClient.sendRequest (JdkHttpClient.java:207)
at io.apicurio.registry.rest.client.impl.RegistryClientImpl.createArtifact (RegistryClientImpl.java:328)
at io.apicurio.registry.maven.AbstractDirectoryParser.registerArtifact (AbstractDirectoryParser.java:101)
at io.apicurio.registry.maven.AbstractDirectoryParser.registerNestedSchema (AbstractDirectoryParser.java:76)
at io.apicurio.registry.maven.AvroDirectoryParser.handleSchemaReferences

davidradl · 2024-04-16T08:10:32Z

@carlesarnal I notice the apicurio-registry-examples repository has been archived. Does this mean it will not be worked on going forward?

carlesarnal · 2024-04-16T08:12:24Z

@carlesarnal thanks for your quick response. Can I check what level we need to be on? I assume 2.5.9. I tried to run the sample (I assume by running mvn install) and get: [INFO] --- apicurio-registry-maven-plugin:2.4.3.Final:register (register-artifact) @ apicurio-registry-examples-avro-maven-with-references-auto --- [WARNING] Error processing Avro schema with name TradeKey.avsc. This usually means that the references are not ready yet to parse it [WARNING] Error processing Avro schema with name Exchange.avsc. This usually means that the references are not ready yet to parse it [WARNING] Error processing Avro schema with name TradeValue.avsc. This usually means that the references are not ready yet to parse it [ERROR] Exception while registering artifact [avro-maven-with-references-auto] / [TradeRaw] io.apicurio.registry.rest.client.exception.RestClientException at io.apicurio.registry.rest.client.impl.ErrorHandler.parseError (ErrorHandler.java:95) at io.apicurio.rest.client.JdkHttpClient.sendRequest (JdkHttpClient.java:207) at io.apicurio.registry.rest.client.impl.RegistryClientImpl.createArtifact (RegistryClientImpl.java:328) at io.apicurio.registry.maven.AbstractDirectoryParser.registerArtifact (AbstractDirectoryParser.java:101) at io.apicurio.registry.maven.AbstractDirectoryParser.registerNestedSchema (AbstractDirectoryParser.java:76) at io.apicurio.registry.maven.AvroDirectoryParser.handleSchemaReferences

Yes, the error makes sense since the examples repository is not updated. You can update the Apicurio Registry version there to 2.5.9.Final here.

carlesarnal · 2024-04-16T08:14:20Z

@carlesarnal I notice the apicurio-registry-examples repository has been archived. Does this mean it will not be worked on going forward?

The repository itself no, the examples will for sure continue, you got us in the middle of the Apicurio Registry 3 preparation, and one of the changes is the move of the examples from that repository to the main one, here (there is a note in the readme). I have not pointed you there because Apicurio Registry 3 has not been released yet, so the main branch is a bit unstable right now.

davidradl · 2024-04-16T08:24:09Z

@carlesarnal I tried to follow https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-managing-registry-artifacts-api.html. I then issue the rest calls:

curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14
{
"namespace":"com.example.common",
"name":"Item",
"type":"record",
"fields":[
{
"name":"itemId",
"type":"com.example.common.ItemId"
}
]
}% (base) davidradley@davids-mbp-2 bashscripts % curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true
zsh: no matches found: http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true

As @seb-pereira indicated the endpoint for dereference=true is not there for me.

I am running:docker run -it -p 8085:8080 apicurio/apicurio-registry-mem:2.5.9.Final
on a Mac

davidradl · 2024-04-16T08:37:23Z

@carlesarnal I tried Yes, the error makes sense since the examples repository is not updated. You can update the Apicurio Registry version there to 2.5.9.Final [here](https://github.com/Apicurio/apicurio-registry-examples/blob/main/pom.xml#L8).

and get
The parameters 'existingReferences' for goal io.apicurio:apicurio-registry-maven-plugin:2.5.9.Final:register are missing or invalid

carlesarnal · 2024-04-16T10:18:12Z

@carlesarnal I tried to follow https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-managing-registry-artifacts-api.html. I then issue the rest calls:

curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14 { "namespace":"com.example.common", "name":"Item", "type":"record", "fields":[ { "name":"itemId", "type":"com.example.common.ItemId" } ] }% (base) davidradley@davids-mbp-2 bashscripts % curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true zsh: no matches found: http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true

As @seb-pereira indicated the endpoint for dereference=true is not there for me.

I am running:docker run -it -p 8085:8080 apicurio/apicurio-registry-mem:2.5.9.Final on a Mac

Ok, I have figured out what's going on. The query parameter has to be escaped for it to work, so if you issue this command you should be all set -> curl http://localhost:8080/apis/registry/v2/ids/globalIds/2\?dereference\=true.

carlesarnal · 2024-04-16T10:19:09Z

@carlesarnal I tried Yes, the error makes sense since the examples repository is not updated. You can update the Apicurio Registry version there to 2.5.9.Final [here](https://github.com/Apicurio/apicurio-registry-examples/blob/main/pom.xml#L8).

and get The parameters 'existingReferences' for goal io.apicurio:apicurio-registry-maven-plugin:2.5.9.Final:register are missing or invalid

Sorry, yes, we introduced that parameter and, as I said the example is outdated (and updated on main, but pointing to 3.0). You can fix that problem by adding a line like this to the pom -> https://github.com/Apicurio/apicurio-registry/blob/main/examples/avro-maven-with-references-auto/pom.xml#L44

davidradl · 2024-04-17T09:10:36Z

@carlesarnal I tried to follow https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-managing-registry-artifacts-api.html. I then issue the rest calls:
curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14 { "namespace":"com.example.common", "name":"Item", "type":"record", "fields":[ { "name":"itemId", "type":"com.example.common.ItemId" } ] }% (base) davidradley@davids-mbp-2 bashscripts % curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true zsh: no matches found: http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true
As @seb-pereira indicated the endpoint for dereference=true is not there for me.
I am running:docker run -it -p 8085:8080 apicurio/apicurio-registry-mem:2.5.9.Final on a Mac

Ok, I have figured out what's going on. The query parameter has to be escaped for it to work, so if you issue this command you should be all set -> curl http://localhost:8080/apis/registry/v2/ids/globalIds/2?dereference=true.

@carlesarnal thankyou with escaping it works for me :-) This is a great circumvention - we need to change the API so that we do not need to escape to add query parameters. Do you know why this is happening? Do you need a new issue for this?

@seb-pereira it looks like we have a way to get the dereferenced schema.

carlesarnal · 2024-04-17T09:22:03Z

@carlesarnal I tried to follow https://www.apicur.io/registry/docs/apicurio-registry/2.5.x/getting-started/assembly-managing-registry-artifacts-api.html. I then issue the rest calls:
curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14 { "namespace":"com.example.common", "name":"Item", "type":"record", "fields":[ { "name":"itemId", "type":"com.example.common.ItemId" } ] }% (base) davidradley@davids-mbp-2 bashscripts % curl -H "Authorization: Bearer $ACCESS_TOKEN" http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true zsh: no matches found: http://localhost:8085/apis/registry/v2/ids/globalIds/14?dereference=true
As @seb-pereira indicated the endpoint for dereference=true is not there for me.
I am running:docker run -it -p 8085:8080 apicurio/apicurio-registry-mem:2.5.9.Final on a Mac

Ok, I have figured out what's going on. The query parameter has to be escaped for it to work, so if you issue this command you should be all set -> curl http://localhost:8080/apis/registry/v2/ids/globalIds/2?dereference=true.

@carlesarnal thankyou with escaping it works for me :-) This is a great circumvention - we need to change the API so that we do not need to escape to add query parameters. Do you know why this is happening? Do you need a new issue for this?

@seb-pereira it looks like we have a way to get the dereferenced schema.

Sorry, maybe my previous message was not 100% clear. I don't think the API has to change, it's a straight query parameter, it just don't work using curl if the url is not properly encoded, you can either escape it as in my previous message or you can also quote the path and it will just work curl 'http://localhost:8080/apis/registry/v2/ids/globalIds/2?dereference=true'. We're not escaping the parameter in any of our tools. We might need to change the documentation to add the quotes to the URL so it just works in the case of someone following the guide and copy-pasting the command, but not the API for sure.

davidradl · 2024-04-17T09:31:43Z

@carlesarnal even better! I agree that updating the docs would have helped me.

seb-pereira · 2024-04-29T18:57:17Z

@carlesarnal @davidradl thanks for your responses. the dereferenced query parameter is ignored with version 2.5.8-final I was using. I confirm it is working as expected with 2.5.9-final. I also managed to register a schema and its references using the ccompat API, and then get the schema with resolved references using Apicurio APIs and the dereferenced query parameter 👍

carlesarnal · 2024-05-15T11:54:43Z

With Apicurio Registry 2.5.11.Final we have introduced dereferencing support for JSON Schema, OpenAPI and AsynAPI. Now we have support for all the types we think it makes sense. Closing this as the result.

carlesarnal added the type/enhancement New feature or request label Oct 7, 2022

apicurio-bot bot added the triage/needs-triage label Oct 7, 2022

carlesarnal self-assigned this Oct 7, 2022

carlesarnal added area/rest-api area/rest-client area/serdes priority/high and removed triage/needs-triage labels Oct 7, 2022

carlesarnal mentioned this issue Oct 7, 2022

Derefence parameter not working #2863

Closed

EricWittmann mentioned this issue Jan 24, 2023

Improved Artifact References support (UI, etc) in Registry Apicurio/apicurio-epics#25

Closed

EricWittmann assigned EricWittmann and MikeEdgar Jun 16, 2023

EricWittmann mentioned this issue Jun 16, 2023

Artifact with reference not updated in apicurio-registry-maven-plugin #2888

Closed

forsberg mentioned this issue Jun 21, 2023

Documentation tab makes unauthenticated requests for referenced schemas #3369

Open

carlesarnal added this to Registry 2.5 May 15, 2024

carlesarnal closed this as completed May 15, 2024

github-project-automation bot moved this to Done in Registry 2.5 May 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement dereference functionality #2865

Implement dereference functionality #2865

carlesarnal commented Oct 7, 2022 •

edited

Loading

ai-zelenin commented May 24, 2023

carlesarnal commented May 26, 2023

EricWittmann commented Jun 16, 2023

seb-pereira commented Apr 3, 2024 •

edited

Loading

davidradl commented Apr 15, 2024 •

edited

Loading

carlesarnal commented Apr 16, 2024

davidradl commented Apr 16, 2024

davidradl commented Apr 16, 2024

carlesarnal commented Apr 16, 2024

carlesarnal commented Apr 16, 2024

davidradl commented Apr 16, 2024

davidradl commented Apr 16, 2024

carlesarnal commented Apr 16, 2024

carlesarnal commented Apr 16, 2024

davidradl commented Apr 17, 2024

carlesarnal commented Apr 17, 2024 •

edited

Loading

davidradl commented Apr 17, 2024

seb-pereira commented Apr 29, 2024 •

edited

Loading

carlesarnal commented May 15, 2024

Implement dereference functionality #2865

Implement dereference functionality #2865

Comments

carlesarnal commented Oct 7, 2022 • edited Loading

ai-zelenin commented May 24, 2023

carlesarnal commented May 26, 2023

EricWittmann commented Jun 16, 2023

seb-pereira commented Apr 3, 2024 • edited Loading

davidradl commented Apr 15, 2024 • edited Loading

carlesarnal commented Apr 16, 2024

davidradl commented Apr 16, 2024

davidradl commented Apr 16, 2024

carlesarnal commented Apr 16, 2024

carlesarnal commented Apr 16, 2024

davidradl commented Apr 16, 2024

davidradl commented Apr 16, 2024

carlesarnal commented Apr 16, 2024

carlesarnal commented Apr 16, 2024

davidradl commented Apr 17, 2024

carlesarnal commented Apr 17, 2024 • edited Loading

davidradl commented Apr 17, 2024

seb-pereira commented Apr 29, 2024 • edited Loading

carlesarnal commented May 15, 2024

carlesarnal commented Oct 7, 2022 •

edited

Loading

seb-pereira commented Apr 3, 2024 •

edited

Loading

davidradl commented Apr 15, 2024 •

edited

Loading

carlesarnal commented Apr 17, 2024 •

edited

Loading

seb-pereira commented Apr 29, 2024 •

edited

Loading