Update get_relation_without_caching to use show with wildcard search in place of describe extended
#501
Conversation
…in place of describe extended
| @@ -234,8 +234,7 @@ def get_relations_without_caching(self, relation: DatabricksRelation) -> Table: | |||
| kwargs = {"relation": relation} | |||
|
|
|||
| new_rows: List[Tuple[Optional[str], str, str, str]] | |||
| if relation.database is not None: | |||
| assert relation.schema is not None | |||
There was a problem hiding this comment.
assert does not do anything in production. pytest flips a debug switch when running to make the assert active, but you would never want that in production. So this actually doesn't do anything, which can be misleading as it suggests we're testing when we're not.
There was a problem hiding this comment.
My understanding is that the point of this is to detect if the assertion is violated when debugging/testing. It doesn't actually do nothing, it just doesn't execute in the deployed code.
There was a problem hiding this comment.
basically, it is there to verify what we believe to be an invariant. If it is actually invariant, then we don't need the if clause below. If clause is definitely safer though.
| @@ -256,47 +255,67 @@ def get_relations_without_caching(self, relation: DatabricksRelation) -> Table: | |||
|
|
|||
| # if there are any table types to be resolved | |||
| if any(not row[3] for row in new_rows): | |||
| # Get view names and create a dictionay of view name to materialization | |||
|
|
||
| view_names: Dict[str, bool] = { | ||
| view["viewName"]: view.get("isMaterialized", False) for view in views | ||
| } | ||
|
|
||
| def parse_type(information: str) -> str: |
There was a problem hiding this comment.
The information in this format is a \n-delimited list of attributes that are of the form <attribute_name>: <attribute_value>.
|
|
||
| # create a new collection of rows with the correct table types | ||
| new_rows = [ | ||
| ( | ||
| row[0], | ||
| row[1], | ||
| row[2], | ||
| str(row[3] if row[3] else typeFromNames(row[0], row[1], row[2])), | ||
| str(row[3] if row[3] else typeFromNames(row[0], row[2])), |
There was a problem hiding this comment.
schema is no longer needed in typeFromNames, hence remove it here.
| @@ -1,4 +1,4 @@ | |||
| databricks-sql-connector>=2.9.3, <3.0.0 | |||
| dbt-spark==1.7.1 | |||
| databricks-sdk==0.9.0 | |||
| databricks-sdk>=0.9.0 | |||
There was a problem hiding this comment.
This file did not align with setup.py.
…for table default
|
closing in favor of moving into the 1.7.0 PR. Thanks @mikealfare |
Resolves #
Description
I'm attempting to replace the
describe extendedscenario with a wildcard search inshow table extended. The current behavior is to inspect each table one at a time to determine whether it is a streaming table or a traditional table. However, I think this can be replaced:Instead of:
Do this:
This PR is meant to be for discovery. I have not tested this via unit test, integration tests, etc. But I have confirmed that the wildcard search returns all tables in the schema. It's also possible it only returns the first X number of objects; I haven't created thousands of tables to test that.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.