Project definition v2 entity schemas: application and application package #1280

sfc-gh-gbloom · 2024-07-03T15:18:34Z

Pre-review checklist

I've confirmed that instructions included in README.md are still correct after my changes in the codebase.
I've added or updated automated unit tests to verify correctness of my new code.
I've added or updated integration tests to verify correctness of my new code.
I've confirmed that my changes are working by executing CLI's commands manually on MacOS.
I've confirmed that my changes are working by executing CLI's commands manually on Windows.
I've confirmed that my changes are up-to-date with the target branch.
I've described my changes in the release notes.
I've described my changes in the section below.

Changes description

The changes in this PR are a step towards enabling definition_version: 2 in snowflake.yml. The changes are behind the hidden snow ws validate command, which requires enabling the enable_project_definition_v2 feature flag.

Changed the v2 schema to have a proper entities field that validates the type of each entity and instantiates the matching Pydantic entity model:

definition_version: 2
entities:
  pkg:
    type: application package
    ...

And added schemas for the ApplicationPackageEntity and ApplicationEntity entities.

Infra features:

Added the EntityBase class for all entities to extend
All entities share the same MetaField
Values specified in DefaultsField will be added if the field exists on the schema class and not specified by the user
TargetField takes a generic entity type, which is verified as part of the schema validation

Test

Two manual tests that were not simple to automate:

The generic type passed to TargetField is enforced. Changed ApplicationEntity.from.target to match a different type:

class ApplicationFromField(UpdatableModel):
-    target: TargetField[ApplicationPackageEntity] = Field(
+    target: TargetField[ApplicationEntity] = Field(
        title="Reference to an application package entity",
    )

And verified that snow ws validate fails:

Target type mismatch. Expected ApplicationEntity, got ApplicationPackageEntity

Tested that template vars are correctly expanded:

definition_version: 2
entities:
  pkg:
    type: application package
    name: my_app_pkg_<% ctx.env.foo %>
    artifacts: []
    manifest: src/manifest.yml
env:
  foo: bar

Added print(cli_context.project_definition) to the validate command and verified the value of name is my_app_pkg_bar:

$ snow ws validate

definition_version='2' entities={'pkg': ApplicationPackageEntity(type_=<EntityType.APPLICATION_PACKAGE: 'application package'>, meta=None, name='my_app_pkg_bar', artifacts=[], deploy_root='output/deploy/', stage='app_src.stage', distribution='internal', manifest='src/manifest.yml')} defaults=None env=ProjectEnvironment(override_env={}, default_env={'foo': 'bar'})

Project definition is valid.

sfc-gh-cgorrie · 2024-07-08T19:36:25Z

src/snowflake/cli/api/project/schemas/entities/application_entity.py

+    )
+
+
+class ApplicationFromField(UpdatableModel):


(Non-blocking) In the future, we should pull the generic type attr up, e.g.

from_: EntityReferenceField[ApplicationPackageEntity]

src/snowflake/cli/api/project/schemas/entities/application_entity.py

tests_integration/workspaces/test_validate_schema.py

sfc-gh-cgorrie · 2024-07-08T19:40:11Z

src/snowflake/cli/api/project/schemas/entities/common.py

+    )
+
+
+class EntityBase(ABC, UpdatableModel):


(Non-blocking) I would like to see e.g. StageBasedEntity, DatabaseLevelEntity, SchemaLevelEntity, AccountLevelEntity in the future to maximize re-use and align with the func spec

sfc-gh-cgorrie · 2024-07-08T19:41:00Z

src/snowflake/cli/api/project/schemas/project_definition.py

+_v2_entity_types_map = {
+    EntityType.APPLICATION.value: ApplicationEntity,
+    EntityType.APPLICATION_PACKAGE.value: ApplicationPackageEntity,
+}


Next up for us is to figure out how these pydantic models map to classes with e.g. deploy, drop, bundle methods

sfc-gh-cgorrie · 2024-07-09T14:22:11Z

src/snowflake/cli/api/project/schemas/entities/application_package_entity.py

+from snowflake.cli.api.project.schemas.native_app.path_mapping import PathMapping
+
+
+class ApplicationPackageEntity(EntityBase):


There are some properties not accounted for here, like scratch_stage, generated_root, and bundle_root. What's the plan for those?

Added these as well. While we maintain both versions we will have to remember to update v2 schema when we make v1 changes.

src/snowflake/cli/api/project/schemas/entities/application_entity.py

sfc-gh-turbaszek · 2024-07-10T07:35:22Z

src/snowflake/cli/api/project/schemas/entities/application_package_entity.py

+        title="Distribution of the application package created by the Snowflake CLI",
+        default="internal",
+    )
+    manifest: str = Field(


Suggested change

manifest: str = Field(

manifest: Path = Field(

Or SecurePath.

sfc-gh-turbaszek · 2024-07-10T08:05:53Z

src/snowflake/cli/api/project/schemas/entities/common.py

+class EntityType(Enum):
+    APPLICATION = "application"
+    APPLICATION_PACKAGE = "application package"


We have those strings as literal on entities classes and we have them here. Should we consider using this Enum as source of truth? Or should we use entities to build Enum / something?

Taking into account that we are constructing Union[ApplicationEntity, ApplicationPackageEntity] and

_v2_entity_types_map = { EntityType.APPLICATION.value: ApplicationEntity, EntityType.APPLICATION_PACKAGE.value: ApplicationPackageEntity, }

It means that adding new entity requires it to be registered in at least two places.

it may be wise to have

KNOWN_ENTITIES = [ ApplicationEntity, ApplicationPackageEntity ] ENTITIES_TYPE = Union[*KNOWN_ENTITIES] _v2_entity_types_map = {get_args(eval(inspect.get_annotations(e)["type"]))[0]: e for e in KNOWN_ENTITIES}

The dict comprehension is a bit of a magic, but probably there's something smarter we can do.

Yes, that's what I tried at first but I couldn't make pydantic understand enums as discriminators and reverted to literal strings.

I'm not sure what I think about this dict comprehension, but I get the idea. I'll try to come up with a cleaner way to have a single source of truth.

sfc-gh-turbaszek · 2024-07-10T08:06:26Z

src/snowflake/cli/api/project/schemas/entities/common.py

+        title="Role to use when creating the entity object",
+        default=None,
+    )
+    post_deploy: Optional[List[ApplicationPostDeployHook]] = Field(


Is this common for all entities?

sfc-gh-astus · 2024-07-10T13:21:34Z

src/snowflake/cli/api/project/schemas/project_definition.py

+        if "defaults" in data:
+            if "entities" in data:


if "defaults" in data and "entities" in data:

sfc-gh-turbaszek · 2024-07-11T11:06:08Z

src/snowflake/cli/api/project/schemas/entities/entities.py

+
+ALL_ENTITIES = [*get_args(Entity)]
+
+v2_entity_types_map = {e.get_type(): e for e in ALL_ENTITIES}


sfc-gh-turbaszek · 2024-07-11T11:08:24Z

src/snowflake/cli/api/project/schemas/project_definition.py

+            # TODO Automatically detect TargetFields to validate
+            if entity.type == ApplicationEntity.get_type():
+                if isinstance(entity.from_.target, TargetField):
+                    target = str(entity.from_.target)
+                    if target not in entities:
+                        raise ValueError(f"No such target: {target}")
+                    else:
+                        # Validate the target type
+                        target_cls = entity.from_.__class__.model_fields["target"]
+                        target_type = target_cls.annotation.__args__[0]
+                        actual_target_type = entities[target].__class__
+                        if target_type and target_type is not actual_target_type:
+                            raise ValueError(
+                                f"Target type mismatch. Expected {target_type.__name__}, got {actual_target_type.__name__}"


I imagine that there will more checks on the enitites should we consider something like

Suggested change

# TODO Automatically detect TargetFields to validate

if entity.type == ApplicationEntity.get_type():

if isinstance(entity.from_.target, TargetField):

target = str(entity.from_.target)

if target not in entities:

raise ValueError(f"No such target: {target}")

else:

# Validate the target type

target_cls = entity.from_.__class__.model_fields["target"]

target_type = target_cls.annotation.__args__[0]

actual_target_type = entities[target].__class__

if target_type and target_type is not actual_target_type:

raise ValueError(

f"Target type mismatch. Expected {target_type.__name__}, got {actual_target_type.__name__}"

# TODO Automatically detect TargetFields to validate

if entity.type == ApplicationEntity.get_type():

self._validate_application_targets(entity)

WDYT?

I split it into 2 methods to have slightly better separation, but we will need to refactor deeper when we find a generic way for this validation. Will address it in a dedicated PR.

sfc-gh-turbaszek · 2024-07-11T11:09:56Z

src/snowflake/cli/api/project/schemas/entities/application_package_entity.py

+    )
+    generated_root: Optional[Path] = Field(
+        title="Subdirectory of the deploy root where files generated by the Snowflake CLI will be written.",
+        default="__generated/",


Should default be a Path or conversion happens automagicaly?

Automagically, but made it Path here too to make it clearer

…kage (#1280) * application and application package entity schemas * add integration test * simplify entity type, use discriminator * create types map dynamically * get_type method * read entity list from union

sfc-gh-gbloom mentioned this pull request Jul 4, 2024

Add future support for project definition v2 #1275

Merged

5 tasks

Base automatically changed from pdfv2-schema to main July 8, 2024 14:53

application and application package entity schemas

0eaf420

sfc-gh-gbloom force-pushed the pdfv2-app-package-entity branch from 7cafab1 to 0eaf420 Compare July 8, 2024 15:34

sfc-gh-gbloom added 6 commits July 8, 2024 11:44

switch internal entity type field names

f7dc71f

add integration test

1e806be

fix rebase

65d62f5

post deploy hook type

8126435

use env vars

1858673

rename test name

244483a

sfc-gh-gbloom force-pushed the pdfv2-app-package-entity branch from 7f38deb to 244483a Compare July 8, 2024 17:23

sfc-gh-gbloom marked this pull request as ready for review July 8, 2024 17:52

sfc-gh-gbloom requested review from a team as code owners July 8, 2024 17:52

sfc-gh-gbloom changed the title ~~Workspaces entities implementation: application package and application~~ Project definition v2 entities implementation: application package and application Jul 8, 2024

sfc-gh-gbloom changed the title ~~Project definition v2 entities implementation: application package and application~~ Project definition v2 entity schemas: application and application package Jul 8, 2024

sfc-gh-cgorrie reviewed Jul 8, 2024

View reviewed changes

src/snowflake/cli/api/project/schemas/entities/application_entity.py Outdated Show resolved Hide resolved

sfc-gh-cgorrie reviewed Jul 8, 2024

View reviewed changes

tests_integration/workspaces/test_validate_schema.py Outdated Show resolved Hide resolved

sfc-gh-cgorrie reviewed Jul 8, 2024

View reviewed changes

sfc-gh-gbloom added 4 commits July 8, 2024 16:10

entity type enum

984a046

simplify feature flag

240fb53

types

d618d8f

Merge branch 'main' into pdfv2-app-package-entity

cb7b090

sfc-gh-cgorrie reviewed Jul 9, 2024

View reviewed changes

src/snowflake/cli/api/project/schemas/entities/application_entity.py Outdated Show resolved Hide resolved

sfc-gh-gbloom added 2 commits July 9, 2024 10:29

add app package fields

35617a7

Merge branch 'main' into pdfv2-app-package-entity

df209b3

sfc-gh-cgorrie previously approved these changes Jul 9, 2024

View reviewed changes

sfc-gh-turbaszek reviewed Jul 10, 2024

View reviewed changes

sfc-gh-astus reviewed Jul 10, 2024

View reviewed changes

sfc-gh-gbloom added 2 commits July 10, 2024 10:34

revert noqa A003

1fd8975

Merge branch 'main' into pdfv2-app-package-entity

79f06fb

sfc-gh-gbloom dismissed sfc-gh-cgorrie’s stale review via 79f06fb July 10, 2024 14:41

sfc-gh-gbloom added 8 commits July 10, 2024 10:47

revert noqa A003

56b9847

update types

bd79cd0

id fields

a6c6ab2

create types map dynamically

90e6f2c

get_type method

5227eda

Merge branch 'main' into pdfv2-app-package-entity

d5188dc

revert entity type

6b3faa6

read entity list from union

c25b4fb

sfc-gh-gbloom requested a review from sfc-gh-turbaszek July 11, 2024 02:45

sfc-gh-gbloom enabled auto-merge (squash) July 11, 2024 02:46

sfc-gh-turbaszek reviewed Jul 11, 2024

View reviewed changes

sfc-gh-gbloom added 2 commits July 11, 2024 10:31

validate target field method

1beb393

Merge branch 'main' into pdfv2-app-package-entity

f4fa56a

sfc-gh-gbloom requested a review from sfc-gh-turbaszek July 11, 2024 15:21

Merge branch 'main' into pdfv2-app-package-entity

c49d6d6

sfc-gh-turbaszek approved these changes Jul 12, 2024

View reviewed changes

sfc-gh-gbloom added 2 commits July 12, 2024 08:32

Merge branch 'main' into pdfv2-app-package-entity

e0555b0

Merge branch 'main' into pdfv2-app-package-entity

9665a6e

sfc-gh-gbloom merged commit 426173a into main Jul 12, 2024
16 checks passed

sfc-gh-gbloom deleted the pdfv2-app-package-entity branch July 12, 2024 13:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Project definition v2 entity schemas: application and application package #1280

Project definition v2 entity schemas: application and application package #1280

sfc-gh-gbloom commented Jul 3, 2024 •

edited

Loading

sfc-gh-cgorrie Jul 8, 2024 •

edited

Loading

sfc-gh-cgorrie Jul 8, 2024

sfc-gh-cgorrie Jul 8, 2024

sfc-gh-cgorrie Jul 9, 2024

sfc-gh-gbloom Jul 9, 2024

sfc-gh-turbaszek Jul 10, 2024

sfc-gh-turbaszek Jul 10, 2024

sfc-gh-gbloom Jul 10, 2024

sfc-gh-turbaszek Jul 10, 2024

sfc-gh-gbloom Jul 10, 2024

sfc-gh-astus Jul 10, 2024 •

edited

Loading

sfc-gh-turbaszek Jul 11, 2024

sfc-gh-turbaszek Jul 11, 2024

sfc-gh-gbloom Jul 11, 2024

sfc-gh-turbaszek Jul 11, 2024

sfc-gh-gbloom Jul 11, 2024

		from snowflake.cli.api.project.schemas.native_app.path_mapping import PathMapping


		class ApplicationPackageEntity(EntityBase):


		ALL_ENTITIES = [*get_args(Entity)]

		v2_entity_types_map = {e.get_type(): e for e in ALL_ENTITIES}

Project definition v2 entity schemas: application and application package #1280

Project definition v2 entity schemas: application and application package #1280

Conversation

sfc-gh-gbloom commented Jul 3, 2024 • edited Loading

Pre-review checklist

Changes description

Test

sfc-gh-cgorrie Jul 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-astus Jul 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-gbloom commented Jul 3, 2024 •

edited

Loading

sfc-gh-cgorrie Jul 8, 2024 •

edited

Loading

sfc-gh-astus Jul 10, 2024 •

edited

Loading