add support for running mul invocations in parallel

[mirnawong1]

this pr clarifies that the Cloud CLI now support running multiple invocations in parallel. This is based on @dichenqiandbt 's demo.

Before that cloud CLI only supports run one invocation at one time.

This pr has grown to also address parallel execution, what it means, where it's supported, and modify the current dbt commands table to further explain this.

Resolves #4952

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-getdbt-com	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 21, 2024 7:16pm

[dichenqiandbt]

I hope to clarify a little to make sure the words here is easy for customer to understand.

Today we support run 2 invocations but it might be more in future, depends on user feedback.

[dichenqiandbt]

It's not 100% accurate, on backend we categorize dbt commands into two types, data warehouse write and data warehouse non-write.

Data warehouse write command always has 1 parallelism. E.g. build. They may cause data warehouse confliction, e.g. overwrite the same table.
Data warehouse non-write command can have x(today it's 1, but it might be more, let's say 2) parallelism. E.g. parse. They are safe to run in parallel.

I'm not sure how to phrase this as it's too complicated for customer to understand, maybe say you are not able to run data warehouse conflicted commands?

[mirnawong1]

mirna to add the following list to specify write and no nwrite commands. waiting to for more info so that i can decide to add this in the commands page (ref) or cloud cli page:

Data warehouse write commands
    "build",
    "clone",
    "retry",
    "run",
    "run-operation",
    "seed",
    "snapshot",
Non-write commands
    "clean",
    "compile",
    "debug",
    "deps",
    "docs",
    "init",
    "list",
    "parse",
    "show",
    "source",
    "test",

[dbeatty10]

Handful of suggestions.

[dbeatty10]

Three suggested changes to the table:

1. Omit the "Type" column since it carries the same info as the "Parallel execution" column
2. Omit content in the "Caveat" section if there are no tool or version restrictions. Remove "Requires". Add "only".
3. Replace "N/A" with ✅ if the command can be executed simultaneously (even if it doesn't interact with the database via a read or a write).

This table is exactly the same as the current table. It just . It also

This table is exactly the same as the current table. It just omits the "Type" column since it carries the same info as the "Parallel execution" column.

It also omits content in the "Caveat" section if there are no tool or version restrictions -- the common assumption is that a command is available in all versions and all tools unless stated otherwise.

There are two versions of the table below:

1. Both suggested changes
2. Omission of "Type" column only

Both are exactly the same as the current table, just less busy.

I didn't do a prototype of suggestion 3, but my assumption is that all the "N/A" could just be replaced with ✅.

Both suggested changes

Command	Description	Parallel execution	Caveats
build	Build and test all selected resources (models, seeds, snapshots, tests)	❌
cancel	Cancels the most recent invocation.	N/A	dbt Cloud CLI only dbt v1.6 or higher
clean	Deletes artifacts present in the dbt project	✅
clone	Clone selected models from the specified state	❌	dbt v1.6 or higher
compile	Compiles (but does not run) the models in a project	✅
debug	Debugs dbt connections and projects	✅	dbt Cloud IDE and dbt Core only
deps	Downloads dependencies for a project	✅
docs	Generates documentation for a project	✅
environment	Enables you to interact with your dbt Cloud environment.	N/A	dbt Cloud CLI only dbt v1.5 or higher
help	Displays help information for any command	N/A	dbt Core and dbt Cloud CLI only
init	Initializes a new dbt project	✅	dbt Core only
list	Lists resources defined in a dbt project	✅
parse	Parses a project and writes detailed timing info	✅
reattach	Reattaches to the most recent invocation to retrieve logs and artifacts.	N/A	dbt Cloud CLI only dbt v1.6 or higher
retry	Retry the last run dbt command from the point of failure	❌	dbt v1.6 or higher
run	Runs the models in a project	❌
run-operation	Invoke a macro, including running arbitrary maintenance SQL against the database	❌
seed	Loads CSV files into the database	❌
show	Preview table rows post-transformation	✅
snapshot	Executes "snapshot" jobs defined in a project	❌
source	Provides tools for working with source data (including validating that sources are "fresh")	✅
test	Executes tests defined in a project	✅
--version	Displays the currently installed version of dbt CLI	N/A	dbt Core and dbt Cloud CLI only

Omission of "Type" column only

Command	Description	Parallel execution	Caveats
build	Build and test all selected resources (models, seeds, snapshots, tests)	❌	All tools All supported versions
cancel	Cancels the most recent invocation.	N/A	dbt Cloud CLI Requires dbt v1.6 or higher
clean	Deletes artifacts present in the dbt project	✅	All tools All supported versions
clone	Clone selected models from the specified state	❌	All tools Requires dbt v1.6 or higher
compile	Compiles (but does not run) the models in a project	✅	All tools All supported versions
debug	Debugs dbt connections and projects	✅	dbt Cloud IDE, dbt Core All supported versions
deps	Downloads dependencies for a project	✅	All tools All supported versions
docs	Generates documentation for a project	✅	All tools All supported versions
environment	Enables you to interact with your dbt Cloud environment.	N/A	dbt Cloud CLI Requires dbt v1.5 or higher
help	Displays help information for any command	N/A	dbt Core, dbt Cloud CLI All supported versions
init	Initializes a new dbt project	✅	dbt Core All supported versions
list	Lists resources defined in a dbt project	✅	All tools All supported versions
parse	Parses a project and writes detailed timing info	✅	All tools All supported versions
reattach	Reattaches to the most recent invocation to retrieve logs and artifacts.	N/A	dbt Cloud CLI Requires dbt v1.6 or higher
retry	Retry the last run dbt command from the point of failure	❌	All tools Requires dbt v1.6 or higher
run	Runs the models in a project	❌	All tools All supported versions
run-operation	Invoke a macro, including running arbitrary maintenance SQL against the database	❌	All tools All supported versions
seed	Loads CSV files into the database	❌	All tools All supported versions
show	Preview table rows post-transformation	✅	All tools All supported versions
snapshot	Executes "snapshot" jobs defined in a project	❌	All tools All supported versions
source	Provides tools for working with source data (including validating that sources are "fresh")	✅	All tools All supported versions
test	Executes tests defined in a project	✅	All tools All supported versions
--version	Displays the currently installed version of dbt CLI	N/A	dbt Core, dbt Cloud CLI All supported versions

[mirnawong1]

thank you so much for this detailed explanation! I love your suggestions and my instincts go to having all rows under 'caveats' filled in so it's explicit what version and tool is supported ( as opposed to inferred). I'll always opt to be more explicit. I'll change this up and revert back!

[dbeatty10]

Nice work @mirnawong1 -- I feel much smarter after reading this PR 🧠

[dichenqiandbt]

LGTM, thanks! I like this PR and customer will like it too!

[dichenqiandbt]

Cloud CLI do have a parallelism limit on non-read commands, can you rephrase by adding a "ideally aren't limited to...".

Today our limit is 1 for non-write invocations but it might be increased soon, so I don't want to say a concrete limit number here, otherwise it would be updated couple of times.

LMK if my expression is clear.

[mirnawong1]

updated table to remove the write column but also added a notation to clarify what x and check mean

[runleonarun]

Just some small wording suggestions! Ship when you're ready!

[mirnawong1]

thanks for everyone's feedback, i really appreciate it! I'm merging this now!