Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trino Connector #6232

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Trino Connector #6232

wants to merge 5 commits into from

Conversation

NamanMahor
Copy link

Follow the same concept as mysql connector and create this connector for trino.

Test with trino mysql and memory catallog.

@begelundmuller
Copy link
Contributor

Hi @NamanMahor, thank you for contributing this PR. For a Trino connector, instead of deriving from the MySQL connector I would recommend looking at how the Snowflake, BigQuery or Athena drivers are implemented.

Since the connector doesn't serve dashboard data, but instead is for ETL into the project's OLAP database, they transfer data as exported Parquet or Arrow buffers instead of reading rows sequentially as the MySQL connector does.

This provides much faster ingestion for the scale of data that Trino often operates on. We previously had a POC connector for BigQuery that transferred data row-by-row, but we had to disable it because the performance was so poor that it threatened stability (since all the transferred rows are created in Go memory, which adds a lot of garbage collection overhead).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants