feat: allow for subjects and samples to have attached gateways where files cannot be listed

[grugna]

/file endpoint response as it currently is:

{
  "data": {
    "summary": {
      ...
    },
    "files": [
      {
        "id": {
          ...
        },
        "samples": [
          {
            ...
          }
        ],
        "metadata": {
          ...
        },
        "gateways": [
          {
            "kind": "Controlled",
            "gateway": {
              "kind": "Controlled",
              "link": {
                "kind": "Approximate",
                "url": "https://..."
              }
            }
          }
        ]
      },
      ...
}

I propose to extend the gateway functionality to subjects and samples.
In addition we should have a higher level gateway for the entire response object grouped by namespace. This way the user can have a fast way to request access where needed for all the subjects/samples/files in the same requests without the need to aggregate all the gateway from each single record.

For instance here is a proposed response for the /subject endpoint

Proposed subject endpoint:

{
  "summary": {
    ...
  },
  "data": [
    {
        "id": ...,
       "name": ...,
       "kind": ...;
       "metadata": ...,
       "gateways": [
          {
            "kind": "Controlled",
            "gateway": {
              "kind": "Controlled",
              "link": {
                "kind": "Approximate",
                "url": "https://..."
              }
            }
          }
        ]
    },
    ...
   ],
   "gateways": [
          {
            "namespace": {
               "organization": "pcdc",
               "name": "COG"
             },
            "kind": "Controlled",
            "gateway": {
              "kind": "Controlled",
              "link": {
                "kind": "Approximate",
                "url": "https://..."
              }
            }
          },
         ...
    ]
}

So more in specific the gateway inside a single subject could have the url value like https://portal.kidsfirstdrc.org/participant/PT_Z34K1EM1#summary and the root level one https://pecan.stjude.cloud/variants/oncoprint/BT where BT is Brain tumor in this example URL I got from the St Jude Cloud but in the context of subject it could be Male/M.

This structure would be the same for Sample and File.

[e-t-k]

I think adding an optional gateway list to individual sample and subject objects (ie, identical to how the file endpoint does it) is a good idea and can be useful for, eg, a site that focuses more on data exploration than on serving files like UCSC Xena.

Regarding the higher-level gateway in a sample or subject response, I'm having trouble seeing what this adds over using named gateways.

For example, if I have a response that returns subject 1 with namespace A and subject 2 with namespace B, my assumption is that the root-level gateway with namespace A would only apply to subject 1, not subject 2. Is that correct? So the user would still have to refer back to each individual subject to know which root gateway applied.

[grugna]

I see the subject/sample/file specific gateway as a link to a specific resource, one specific file/1 specific subject etc etc.

The higher level gateway is a way to share a link to a cohort of subjects or a group of samples. So for instance let's say we have a search with sex = Female, so all the female subjects in the federations members. Instead of having a gateway for each resource pointing to the general interface for PCDC and other members, we could share a link to the PCDC (or other member) exploration page with the same filter already applied, or a manifest to upload.

In my opinion this would streamline and make it easier to group all the data together and submit a single request to the namespace in question when the access is controlled.

@e-t-k not sure if this responds to your question

[e-t-k]

let's say we have a search with sex = Female... Instead of having a gateway for each resource pointing to the general interface for PCDC and other members, we could share a link to the PCDC (or other member) exploration page with the same filter already applied

Yes, I agree that's really valuable. Could this link be shared using the Named Gateway system already present in the API?
Eg, each subject would have an anonymous gateway with a link directly to its own resource; and then it would have an "Reference" gateway named eg "PCDC_sex_Female" that appears a single time in the root gateways list.

Although now that I describe that, it would mean that a Subject has a different gateway when you retrieve it with a different filter. That's undesirable. So I don't in fact see a way to use the Named Gateway system.

Because the higher level gateway is specifically for mapping a filter in the API to a filter in the external resource, it really is a gateway for the whole response rather than for any subset of samples in the response.
So maybe it would make sense to put them in the summary object, separate from any other named gateways, to indicate that they apply to all entities in the response.

  "summary": {
     "counts": { ... },
     "gateways": [
      {
      "namespace": {...},
      "kind": ...
      }
   ],
  "data": [ ... ]

[grugna]

I like the idea of including it in the "summary" to keep it more separated. However, it feels the user would be requesting access to the "data" and not the "summary" information.

Let me think a little more if I can come up with something more separated but still related to the "data". Maybe it could be on the same level as "data" and "summary" instead of inside the "data" dictionary?

[claymcleod]

I think everything here looks good. My only critique would be that I would advocate the top-level namespace not be forced to be tied to a namespace. For example, in the /file response, a top-level gateway can represent anything (and, in turn, can be named anything). When there is a top-level gateway, the individual gateway entries simply refer to the top-level gateway by name.

In other words, I don't think there is a benefit to restricting top-level gateways to only be organizable by namespaces. Instead, we should just let the implementing server choose how they want to organize their gateways.

[grugna]

This makes sense. I agree.

[claymcleod]

Just to make sure it's clear, this is how the endpoint would work for two files that refer to the same top-level gateway:

{
  "data": {
    ...,
    "files": [
      {
        "id": {
          "namespace": {
            "organization": "hello",
            "name": "World"
          },
          "name": "File001.txt"
        },
        "samples": [ ... ],
        "gateways": [
          {
            "gateway": "gateway-one",
            "kind": "Reference"
          }
        ],
        "metadata": { ... }
      },
      {
        "id": {
          "namespace": {
            "organization": "hello",
            "name": "World"
          },
          "name": "File002.txt"
        },
        "samples": [ ... ],
        "gateways": [
          {
            "gateway": "gateway-one",
            "kind": "Reference"
          }
        ],
        "metadata": { ... }
      }
    ],
    "gateways": [
      {
        "link": {
          "url": "https://hello.world/data",
          "kind": "Direct"
        },
        "kind": "Controlled",
        "name": "gateway-one"
      },
    ]
  }
}

[grugna]

@claymcleod I would like to include a way to have a overall request gateway.
I propose to add another kind for the root level gateway. Just like the object level gateway can be Anonymous or Reference, I think the top level gateway should be something like Global or Anonymous/Reference.
In this case with Global the gateway would apply to the entire request and we don't even need to add it file by file / object by object as a reference.

The main advantage of doing this in my opinion is that we can have both at the same time. We could have an object level gateway that is specific to the object, so get access to a specific subject or a specific file, and the global would be for all the subjects in the request.

In you example it would be like the following. A file could still have a Reference gateway like defined before, and/or having an object specific gateway. In addition to this if one of the top level Gateways is of kind Global the link would bring to the entire cohort.
Another specific example could be the object specific gateway point to the object, so in case of direct link it would link to the web page / location of a file or of a subject in specific, while the top level defined as Global will point to an exploration page with the same set of files/subjects/samples. So for example if a user selected all Males in the federation API, and clicked on the Global link for PCDC it would be redirected to the Exploration page on our portal with filters preselected like data contributor = COG and sex = Male. This way the user could see the same cohort / dataset requested on the federation API and start the process to request access to those data with the system/process that each member has already built.

{
  "data": {
    ...,
    "files": [
      {
        "id": {
          "namespace": {
            "organization": "hello",
            "name": "World"
          },
          "name": "File001.txt"
        },
        "samples": [ ... ],
        "gateways": [
          {
            "gateway": {
               "link": {
                  "url": "portal.pedscommons.org/files/hello/World/File001.txt",
                  "kind": "Direct"
               },
               "kind": "Open"
             },
            "kind": "Anonymous"
          }
        ],
        "metadata": { ... }
      },
      ...
    ],
    "gateways": [
      {
         {
            "link": {
               "url": "https://hello.world/data",
               "kind": "Direct"
            },
           "kind": "Controlled",
           "name": "gateway-one"
        },
        "kind": "Anonymous / Reference"
      },
      {
         {
            "link": {
               "url": "https://hello.world/data?4948jdnfaihfnojas",
               "kind": "Direct"
            },
           "kind": "Controlled",
           "name": "gateway-global-one"
        },
        "kind": "Global"
      },
    ]
  }
}

[wnkhan]

First off, just want to express gratitude for the work that went into putting this together. After reviewing the proposal with the team, we definitely see the value in having Gateways for the subject and sample resources, however we would be in support of having entity level gateways rather than incorporating a global or query-specific gateway.

We believe the global or query-specific gateway could introduce a great degree of complexity as data contributors add access to additional sources of truth through their respective API's, and that this would be better left up to clients to aggregate the respective gateways to construct their access requests considering the degree of variability that could exist amongst access requests across namespaces and/or organizations.

A thought I had after our internal meeting on this, would be that if we were to have a hierarchical gateway that encompassed resources beyond the entity level, that it could possible be related to a namespace in order to reduce processing each resource in a given dataset to identify all access control measures.

[grugna]

@wnkhan the idea would be for that field to be optional, that way if it is too complex for a member to implement for the way the governance is it could just be skipped. I would also be in favor of grouping it by namespace, but also I would be ok leaving it open to each server to define the response. Having the grouping by namespace optional as well so that it can be more flexible based on the server data governance.

[claymcleod]

I'm also of the opinion that it should not be implemented in this PR (just for v1) and that we should discuss the global gateway more. In short, my thought it that the tradeoff is essentially that (a) each entity won't need to reference a global gateway by name, which saves some space but (b) parsers will need to handle a moderately greater deal of complexity. It's true that it's optional for implementers of these servers, but the parsers would all be required to handle such cases.

[grugna]

#95