[Bug] Complex types are truncated during describe extended

[mikealfare]

Is this a new bug in dbt-spark?

• I believe this is a new bug in dbt-spark
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Complex types are truncated when running this macro:

dbt-spark/dbt/include/spark/macros/adapters.sql

Lines 281 to 286 in 3fc624c

	{% macro spark__get_columns_in_relation_raw(relation) -%}
	{% call statement('get_columns_in_relation_raw', fetch_result=True) %}
	describe extended {{ relation }}
	{% endcall %}
	{% do return(load_result('get_columns_in_relation_raw').table) %}
	{% endmacro %}

This happens due to DESCRIBE EXTENDED, which truncates the results before returning them.

Expected Behavior

The types should be complete.

Steps To Reproduce

• create a model (e.g. my_model) with a sufficiently complex type
• run DESCRIBE EXTENDED my_model
• look at the resulting type for the complex column

Relevant log output

No response

Environment

- OS:
- Python:
- dbt-core:
- dbt-spark:

Additional Context

No response

[amychen1776]

@benc-db This is the issue we were talking about yesterday about the issues with the Databricks Metadata API. Is this just a Databricks specific issue?

[benc-db]

It is Databricks specific, but may affect dbt-spark as well.

[benc-db]

lol, I didn't see where I was commenting. So, I do not know the extent to which describe extended is standard Spark vs Databricks, which is probably what you're asking here.

[amychen1776]

@benc-db yup :)

@mikealfare did you find this bug running on Databricks then?