Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][catalog] Streampark start job ship catalogstore plugin. #3946

Closed
3 tasks done
Mrart opened this issue Aug 6, 2024 · 0 comments · Fixed by #3987
Closed
3 tasks done

[Feature][catalog] Streampark start job ship catalogstore plugin. #3946

Mrart opened this issue Aug 6, 2024 · 0 comments · Fixed by #3987

Comments

@Mrart
Copy link
Contributor

Mrart commented Aug 6, 2024

Search before asking

  • I had searched in the feature and found no similar feature requirement.

Description

Now that we have implemented the catalogstore and streampark catalog management, we need to do two more things:

  1. The flink cli copies the catalogstore into the flink runtime via yarn.provide.lib.
  2. Rebuild the catalog object through the catalog configuration and crud the managed database as follows
    a. Build a catalog-plugin that contains catalogstore, flink-connnector-jdbc, flink-connector-paimon, flink-connector-jdbc, Make sure spi can scan to the CatalogFactory.
    b. Package the catalog-plugin into the streampark/plugin directory.
    c. Load the catalog-plugin into classloader/ and catlogstore configuration, reflect Catalog, and then implement Catalog crud for database/table. public static Catalog createCatalog(String catalogName,Map<String, String> options,ReadableConfig configuration,ClassLoader classLoader)

Usage Scenario

The flink sql job starts by lazy loading the catalog and parsing it.

Related issues

#3912

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant