Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Schemas discovered via JSON configurations should be created in Schema Registry #73

Open
chipmaurer opened this issue Nov 4, 2021 · 1 comment
Assignees

Comments

@chipmaurer
Copy link
Collaborator

If the server is connected to Schema Registry at startup, then it should check Schema Registry for any 'schema' that is identified by JSON configuration files, and created in Schema Registry, just as if the user had created the schema via the CLI. For example, the schema below should check to see if the 'cartoondb' schema is configured in Schema Registry

{
"schemas": [
{
"schemaTableName": {
"schema_name": "cartoondb",
"table_name": "addressTable"
},
"s3Table": {
"name": "addressTable",
"columns": [
{
"name": "Name",
"type": "VARCHAR"
},
{
"name": "Address",
"type": "VARCHAR"
}
],
"sources": {"testbucket": ["cartoondb/cartoon_table.json"] },
"objectDataFormat": "json",
"hasHeaderRow": "false",
"recordDelimiter": "\n"
}
}
]
}

@chipmaurer
Copy link
Collaborator Author

It's actually a pain, because the server thinks the schema is defined, so create schema doesn't work.

presto:cartoondb> CREATE SCHEMA s3.cartoondb;
Query 20211104_154257_00007_cciqr failed: line 1:1: Schema 's3.cartoondb' already exists
CREATE SCHEMA s3.cartoondb

But you can't create a table, because it checks schema registry.

presto:cartoondb> create table s3.cartoondb.table1 ( id bigint, name varchar, balance double) WITH (has_header_row = 'false', FORMAT = 'CSV', external_location='s3a://testbucket/db1/' );
Query 20211104_154308_00008_cciqr failed: Group not found.

If you create a schema with a different name, and then use that, create table works

presto:cartoondb> CREATE SCHEMA s3.cartoondb1;
CREATE SCHEMA

presto:cartoondb> create table s3.cartoondb1.table1 ( id bigint, name varchar, balance double) WITH (has_header_row = 'false', FORMAT = 'CSV', external_location='s3a://testbucket/db1/' );
CREATE TABLE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants