Feature/fhir json schema + Remove Elasticsearch support from Grip #318

matthewpeterkort · 2024-10-23T23:16:36Z

This PR does a few things:

Add support for storing jsonschema as a grip graph
grip schema post --graphName CALIPER --jsonSchema test.json

The protobuf for this is defined as:

  rpc AddJsonSchema(RawJson) returns (EditResult) {
    option (google.api.http) = {
        post: "/v1/graph/{graph}/jsonschema"
        body: "*"
    };
  }

Add support for loading FHIR json directly into grip via new proto method:
grip caliperload schema-test/Observation.ndjson CALIPER test-data2

message RawJson {
  string graph = 1;
  google.protobuf.Struct extra_args = 2;
  google.protobuf.Struct data = 3;
}


  rpc BulkAddRaw (stream RawJson) returns (BulkJsonEditResult) {
    option (google.api.http) = {
      post: "/v1/rawJson"
    };
  }

Note also BulkJsonEditResult is also added, adding support for returning messages back to the client so that an HTTP code can be generated from the error messages or the lack of error messages.

message BulkJsonEditResult{
    int32 insert_count = 1;
    repeated string errors = 2;
}

Reorganize schema files into schema dir. Those files were previously in gripql directory.

Also removes all instances of elasticSearch from Grip. This include the website too.

Proto spec now abstracts project_id into extraArgs

matthewpeterkort · 2024-11-18T20:53:11Z

server/metagraphs.go

+func (server *GripServer) LoadSchemas(project_id string, sch *gripql.Graph, out *graph.GraphSchema) (*graph.GraphSchema, error) {
+	schcompiler := jsonschema.NewCompiler()
+	schcompiler.ExtractAnnotations = true
+	schcompiler.RegisterExtension(compile.GraphExtensionTag, compile.GraphExtMeta, compile.GraphExtCompiler{})


Our graph extension is hard coded into the loader here. An extension argument could get abstracted out as an argument to the load schema protobuf method.

kellrott · 2024-12-02T19:49:22Z

cmd/load/main.go

@@ -171,7 +173,7 @@ var Cmd = &cobra.Command{

 		if yamlFile != "" {
 			log.Infof("Loading YAML file: %s", yamlFile)
-			graphs, err := gripql.ParseYAMLGraphsFile(yamlFile)
+			graphs, err := schema.ParseYAMLGraphsFile(yamlFile)


Why isn't this graphSchema?

kellrott · 2024-12-03T17:55:43Z

docs/categories/index.xml

@@ -2,13 +2,10 @@
 <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Categories on GRIP</title>
-    <link>https://bmeg.github.io/grip/categories/</link>
+    <link>http://localhost:1313/grip/categories/</link>


The doc build was done with the wrong hostname

addressed in newest commit

matthewpeterkort · 2024-12-03T18:36:08Z

ok localhost paths should be fixed now. When hugo --source ./website -w server is run it changes the files to point to localhost but when hugo --source ./website is run it rebuilds the site to point to the actual domain name. so just had to run hugo --source ./website again

…aphql conversion scripts into jsonschemagraph package

matthewpeterkort · 2024-12-12T23:59:57Z

Moved out graphql code. This branch should be cleaned up now and ready for merge

kellrott · 2024-12-13T20:36:12Z

conformance/tests/ot_schema.py

+
+def test_post_json_schema(man):
+    errors = []
+    G = man.setGraph("swapi")


For cases where the unit test is a write option, you get the graph using

G = man.writeTest()

kellrott · 2024-12-13T20:37:05Z

conformance/tests/ot_schema.py

+    errors = []
+    G = man.setGraph("swapi")
+    # Probably don't want to add a 2MB schema file to the repo so get it via requests instead
+    res = requests.get("https://raw.githubusercontent.com/bmeg/iceberg/f1724941fe47df24846135fb515d1b89e791cee3/schemas/graph/graph-fhir.json")


This probably isn't ideal in the long run. We can think about a paired down sample schema that can be used for unit tests in the future.

kellrott · 2024-12-19T06:07:10Z

Is there a way to load a directory of JsonSchema YAML files? I only see single file JSON post for the schema post command.

kellrott · 2024-12-19T06:08:47Z

cmd/schema/main.go

-	sflags.BoolVar(&manual, "manual", manual, "Use client side schema sampling")
-	sflags.StringSliceVar(&excludeLabels, "exclude-label", excludeLabels, "exclude vertex/edge label from schema")
+	pflags.StringVar(&jsonSchemaFile, "jsonSchema", "", "Json Schema")
+	pflags.StringVar(&graphName, "graphName", "", "Name of schemaGraph")


For the other sub commands, graphName is defined via fixed argument and not a flag.

matthewpeterkort · 2025-01-02T22:09:08Z

Is there a way to load a directory of JsonSchema YAML files? I only see single file JSON post for the schema post command.

This new feature has been added see tests, cmd line option --yamlSchemaPath supports loading individual yaml schema as files or loading entire yaml schema directories of files.

Additionally the AddSchema API has been changed from a delete and replace strategy to a upsert strategy so that schemas can now be added to / edited with multiple AddSchema calls.

matthewpeterkort · 2025-01-02T22:24:45Z

Should be ready for another review / merge now

kellrott · 2025-01-03T19:47:22Z

LGTM

matthewpeterkort added 14 commits October 22, 2024 14:08

Start JSON schema loader into grip graph

42de601

Add support for extensions

f9cf8f4

cleanup vertex generation

9a800dc

Add edge generation

b9da7be

Adds schema load command

5f507f8

start implement json load

c1ee086

working json loader

7051f51

cleanup, add support for graph and project-id customization

be854ce

Adds tests, new schema load method

1fcac9f

trim down test data size

f4ae5df

bug fix

c9f36fe

bug fix

694ef9d

cleanup

f11f79e

fix schema dataset

0e7ce90

matthewpeterkort requested a review from kellrott November 15, 2024 21:28

matthewpeterkort added 2 commits November 15, 2024 15:02

fix logs

7351ace

Bug fix auth test

14f0a58

matthewpeterkort commented Nov 18, 2024

View reviewed changes

matthewpeterkort added 4 commits November 18, 2024 14:00

Adds more tests

bb2b9dd

Add server err message response

821431f

fix tests

a7f6fdd

Remove elasticSearch from grip

b8d9a0a

matthewpeterkort changed the title ~~[WIP] Feature/fhir json schema~~ Feature/fhir json schema + Remove Elasticsearch support from Grip Nov 20, 2024

matthewpeterkort added 2 commits November 20, 2024 11:27

Bugfix protobuf definition

a3ff902

bugfix, rename cmd

515a682

matthewpeterkort mentioned this pull request Nov 22, 2024

Feature/grip validation ACED-IDP/aced_etl_pod#43

Merged

factor out project_id into an optional extra arg

67d787e

kellrott reviewed Dec 2, 2024

View reviewed changes

kellrott reviewed Dec 3, 2024

View reviewed changes

fix docs hostname and schema import name

007f2fc

matthewpeterkort added 5 commits December 4, 2024 11:37

fix double slash in paths

9537849

remove extra file

3bb0fde

Flesh out jsonschema to graphql schema command

a1646b0

fixup graphql schema generation

146c50e

fix up cmd args to work with jsonschemagraph changes

b772ede

matthewpeterkort requested a review from kellrott December 11, 2024 23:51

Make go.mod versions compatible with grip plugin versions and move gr…

1eed29a

…aphql conversion scripts into jsonschemagraph package

kellrott reviewed Dec 13, 2024

View reviewed changes

kellrott reviewed Dec 19, 2024

View reviewed changes

matthewpeterkort added 4 commits January 2, 2025 12:21

Add support for yaml schemas, change AddSchema to an upsert operation

e938d18

clean up / improve tests

f365481

cleanup

683df7a

cleanup

c2dbbc7

Add gripql query function calls

9654ae9

matthewpeterkort requested a review from kellrott January 2, 2025 22:24

kellrott merged commit a231289 into develop Jan 3, 2025
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/fhir json schema + Remove Elasticsearch support from Grip #318

Feature/fhir json schema + Remove Elasticsearch support from Grip #318

matthewpeterkort commented Oct 23, 2024 •

edited

Loading

matthewpeterkort Nov 18, 2024 •

edited

Loading

kellrott Dec 2, 2024

matthewpeterkort Dec 3, 2024

kellrott Dec 3, 2024

matthewpeterkort Dec 3, 2024

matthewpeterkort commented Dec 3, 2024 •

edited

Loading

matthewpeterkort commented Dec 12, 2024

kellrott Dec 13, 2024

kellrott Dec 13, 2024

matthewpeterkort Jan 2, 2025

kellrott commented Dec 19, 2024

kellrott Dec 19, 2024

matthewpeterkort Jan 2, 2025

matthewpeterkort commented Jan 2, 2025 •

edited

Loading

matthewpeterkort commented Jan 2, 2025

kellrott commented Jan 3, 2025

Feature/fhir json schema + Remove Elasticsearch support from Grip #318

Feature/fhir json schema + Remove Elasticsearch support from Grip #318

Conversation

matthewpeterkort commented Oct 23, 2024 • edited Loading

matthewpeterkort Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewpeterkort commented Dec 3, 2024 • edited Loading

matthewpeterkort commented Dec 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kellrott commented Dec 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewpeterkort commented Jan 2, 2025 • edited Loading

matthewpeterkort commented Jan 2, 2025

kellrott commented Jan 3, 2025

matthewpeterkort commented Oct 23, 2024 •

edited

Loading

matthewpeterkort Nov 18, 2024 •

edited

Loading

matthewpeterkort commented Dec 3, 2024 •

edited

Loading

matthewpeterkort commented Jan 2, 2025 •

edited

Loading