Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Key Vault] TypeSpec for Secrets library #29249

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open

Conversation

mccoyp
Copy link
Member

@mccoyp mccoyp commented May 28, 2024

Data Plane API Specification Update Pull Request

Tip

Overwhelmed by all this guidance? See the Getting help section at the bottom of this PR description.

Based on feedback from #28708. This spec is for the KV Secrets library. This has been successfully tested end-to-end with Python (Azure/azure-sdk-for-python#36901).

PR review workflow diagram

Please understand this diagram before proceeding. It explains how to get your PR approved & merged.

spec_pr_review_workflow_diagram

API Info: The Basics

Most of the information about your service should be captured in the issue that serves as your API Spec engagement record.

  • Link to API Spec engagement record issue:

Is this review for (select one):

  • a private preview
  • a public preview
  • GA release

Change Scope

This section will help us focus on the specific parts of your API that are new or have been modified.
Please share a link to the design document for the new APIs, a link to the previous API Spec document (if applicable), and the root paths that have been updated.

  • Design Document:
  • Previous API Spec Doc:
  • Updated paths:

Viewing API changes

For convenient view of the API changes made by this PR, refer to the URLs provided in the table
in the Generated ApiView comment added to this PR. You can use ApiView to show API versions diff.

Suppressing failures

If one or multiple validation error/warning suppression(s) is detected in your PR, please follow the
Swagger-Suppression-Process
to get approval.

❔Got questions? Need additional info?? We are here to help!

Contact us!

The Azure API Review Board is dedicated to helping you create amazing APIs. You can read about our mission and learn more about our process on our wiki.

Click here for links to tools, specs, guidelines & other good stuff

Tooling

Guidelines & Specifications

Helpful Links

Getting help

  • First, please carefully read through this PR description, from top to bottom.
  • If you don't have permissions to remove or add labels to the PR, request write access per aka.ms/azsdk/access#request-access-to-rest-api-or-sdk-repositories
  • To understand what you must do next to merge this PR, see the Next Steps to Merge comment. It will appear within few minutes of submitting this PR and will continue to be up-to-date with current PR state.
  • For guidance on fixing this PR CI check failures, see the hyperlinks provided in given failure
    and https://aka.ms/ci-fix.
  • If the PR CI checks appear to be stuck in queued state, please add a comment with contents /azp run.
    This should result in a new comment denoting a PR validation pipeline has started and the checks should be updated after few minutes.
  • If the help provided by the previous points is not enough, post to https://aka.ms/azsdk/support/specreview-channel and link to this PR.

@mccoyp mccoyp added KeyVault data-plane TypeSpec Authored with TypeSpec labels May 28, 2024
Copy link

openapi-pipeline-app bot commented May 28, 2024

Next Steps to Merge

Next steps that must be taken to merge this PR:
  • ❌ This is the public specs repo main branch which is not intended for iterative development.
    You must acknowledge that you understand that after this PR is merged, you won't be able to stop your changes from being published to Azure customers.
    If this is what you intend, add PublishToCustomers label to your PR.
    Otherwise, retarget this PR onto a feature branch, i.e. with prefix release- (see aka.ms/azsdk/api-versions#release--branches).
  • ❌ This PR has at least one breaking change (label: BreakingChangeReviewRequired).
    To unblock this PR, follow the process at aka.ms/brch.
  • ❌ The required check named Swagger PrettierCheck has failed. Refer to the check in the PR's 'Checks' tab for details on how to fix it and consult the aka.ms/ci-fix guide
  • ❌ The required check named Swagger SpellCheck has failed. Refer to the check in the PR's 'Checks' tab for details on how to fix it and consult the aka.ms/ci-fix guide

Copy link

openapi-pipeline-app bot commented May 28, 2024

Swagger Validation Report

️️✔️BreakingChange succeeded [Detail] [Expand]
There are no breaking changes.
️️✔️Breaking Change(Cross-Version) succeeded [Detail] [Expand]
There are no breaking changes.
️️✔️CredScan succeeded [Detail] [Expand]
There is no credential detected.
️️✔️LintDiff succeeded [Detail] [Expand]
Validation passes for LintDiff.
Compared specs (v2.2.2) new version base version
default default(56e1117) default(main)
️️✔️Avocado succeeded [Detail] [Expand]
Validation passes for Avocado.
️️✔️SwaggerAPIView succeeded [Detail] [Expand]
️️✔️TypeSpecAPIView succeeded [Detail] [Expand]
️️✔️ModelValidation succeeded [Detail] [Expand]
Validation passes for ModelValidation.
️️✔️SemanticValidation succeeded [Detail] [Expand]
Validation passes for SemanticValidation.
️️✔️PoliCheck succeeded [Detail] [Expand]
Validation passed for PoliCheck.
️️✔️SpellCheck succeeded [Detail] [Expand]
Validation passes for SpellCheck.
️️✔️Lint(RPaaS) succeeded [Detail] [Expand]
Validation passes for Lint(RPaaS).
️️✔️PR Summary succeeded [Detail] [Expand]
Validation passes for Summary.
️️✔️Automated merging requirements met succeeded [Detail] [Expand]
Posted by Swagger Pipeline | How to fix these errors?

Copy link

openapi-pipeline-app bot commented May 28, 2024

Swagger Generation Artifacts

️️✔️ApiDocPreview succeeded [Detail] [Expand]
️⚠️ azure-sdk-for-python warning [Detail]
    For more instructions, please refer to the FAQ .
  • ⚠️Warning in generating from 4e64f3e551925ec5a52ddd927e6584675b93add9. SDK Automation 14.0.0
    command	sh scripts/automation_init.sh ../azure-sdk-for-python_tmp/initInput.json ../azure-sdk-for-python_tmp/initOutput.json
    cmderr	[automation_init.sh] W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/azure-cli.list:1 and /etc/apt/sources.list.d/azure-cli.sources:1
    cmderr	[automation_init.sh] W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/azure-cli.list:1 and /etc/apt/sources.list.d/azure-cli.sources:1
    cmderr	[automation_init.sh] W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/azure-cli.list:1 and /etc/apt/sources.list.d/azure-cli.sources:1
    cmderr	[automation_init.sh] W: Target CNF (main/cnf/Commands-amd64) is configured multiple times in /etc/apt/sources.list.d/azure-cli.list:1 and /etc/apt/sources.list.d/azure-cli.sources:1
    cmderr	[automation_init.sh] W: Target CNF (main/cnf/Commands-all) is configured multiple times in /etc/apt/sources.list.d/azure-cli.list:1 and /etc/apt/sources.list.d/azure-cli.sources:1
    cmderr	[automation_init.sh] WARNING: Skipping azure-nspkg as it is not installed.
    warn		Warning: azure-sdk-for-python cannot be found in specification/keyvault/data-plane/readme.md. This SDK will be skipped from SDK generation. Please add the right config to the readme file according to this guidance https://github.com/Azure/azure-rest-api-specs/blob/main/documentation/code-gen/configure-go-sdk.md#swagger-to-sdk.
    command	sh scripts/automation_generate.sh ../azure-sdk-for-python_tmp/generateInput.json ../azure-sdk-for-python_tmp/generateOutput.json
    warn	Warning: No file changes detected after the generation. Please refer to the generation errors to understand the reasons.
    warn	Warning: No package detected after generation. Please refer to the above logs to understand why the package hasn't been generated.
Posted by Swagger Pipeline | How to fix these errors?

@mccoyp mccoyp marked this pull request as ready for review June 3, 2024 22:47
@mccoyp mccoyp requested a review from johanste June 3, 2024 22:48
@heaths
Copy link
Member

heaths commented Jun 3, 2024

I'm going to wait for the Breaking Change validation to run. It'll be easier than having to manually diff.

Copy link
Member

@heaths heaths left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Breaking change validation didn't really find anything, but, overall, I'm concerned this doesn't use much of Azure Core's abstractions and instead uses individual ops using Foundations. I appreciate Key Vault doesn't support much of the standard traits, but we can define our own and likely use more resource collections; otherwise, I don't think these TypeSpecs really "sell" the idea and likely are more authoring than the existing swaggers. We want the service team to be able to take these over, and resource collection abstractions should reduce the onboarding and maintenance effort.

We can talk offline more but should get some others e.g., @johanste involved in the discussions. I suspect my feedback will be the same as for Keys and Certificates.

BTW: Missing Administration.

Also BTW: All these need to be in separate project directories. We need to split them based on the SDK splits now.

specification/keyvault/data-plane/TypeSpec/main.tsp Outdated Show resolved Hide resolved
specification/keyvault/data-plane/TypeSpec/main.tsp Outdated Show resolved Hide resolved
specification/keyvault/data-plane/TypeSpec/models.tsp Outdated Show resolved Hide resolved

namespace KeyVault;

alias KeyVaultOperations = StandardResourceOperations;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't find the definition of StandardResourceOperations, but I highly doubt this is correct. Key Vault does not, for example, support etags for resources which, I'm betting, is probably standard for Azure.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point; I didn't consider that Azure-standard traits could be carried over by this which wouldn't apply to Key Vault. This was done for defining resource operations (e.g. KeyVaultOperations.ResourceList).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Key Vault doesn't really support standard resource operations, though. They are close, but not standard.

specification/keyvault/data-plane/TypeSpec/routes.tsp Outdated Show resolved Hide resolved
*/
@pattern("^[0-9a-zA-Z-]+$")
@path
secretName: string;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it's just "name" for all SDKs. Should that use a decorator instead? The old swagger uses "secret-name". Same for "secret-version" below.

@summary("Get a specified secret from a given key vault.")
@route("/secrets/{secretName}/{secretVersion}")
@get
op getSecret is Azure.Core.Foundations.Operation<
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will create problems for SDKs but I'm not sure how to solve it. We went with making "version" an optional parameter across languages, which means we basically collapsed the routes into one. @lmazuel or @johanste any suggestions? Can client TSPs "fix" this, or is this a case where we're going to generate an incompatible SDK and have to either wrap it or ship a new major version?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @pallavit

* requires the secrets/list permission.
*/
@summary("List secrets in a specified key vault.")
op getSecrets is KeyVaultOperations.ResourceList<
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This uses the right pattern. Why don't the others? Do they not work? Do we need to declare our own traits so they do align? Key Vault doesn't really support many of the (now-) standard params.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answered in #29249 (comment)

/**
* The object attributes managed by the KeyVault service.
*/
model Attributes {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this used other than as a base class?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just used as a base class that comes from the original swagger.


using Azure.Core;
using Azure.Core.Traits;
using Azure.ClientGenerator.Core;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not want this in the service description. Please create a client.tsp for client codegen customizations.

/**
* The secret set parameters.
*/
model SecretSetParameters {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this just the subset of properties of a SecretBundle that are writeable? If so, why do we need two models and not one with the appropriate visibility?

/**
* The secret management attributes.
*/
model SecretAttributes extends Attributes {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this extend Attributes? Can I ever use one in place of another?

@summary("Sets a secret in a specified key vault.")
@route("/secrets/{secretName}")
@put
op setSecret is Azure.Core.Foundations.Operation<
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a resource create or replace?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #29249 (comment). Many of these operations could be standard resource operations if the service returned a resource name instead of a resource ID in the form of a full URL. I couldn't get operations to conform to using the latter as a resource key, and I'm not aware of a way to override the route otherwise.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the secret name is not returned? It is only in the path of the request?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not returned as a standalone property, no. It's part of the id but needs to be parsed manually -- here's an example from a recording.

"id": "https://vaultname.vault.azure.net/secrets/{secretName}/{secretVersion}"

@summary("Deletes a secret from a specified key vault.")
@route("/secrets/{secretName}")
@delete
op deleteSecret is Azure.Core.Foundations.Operation<
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a resource delete?

@microsoft-github-policy-service microsoft-github-policy-service bot added no-recent-activity There has been no recent activity on this issue. and removed no-recent-activity There has been no recent activity on this issue. labels Jul 8, 2024
@microsoft-github-policy-service microsoft-github-policy-service bot added no-recent-activity There has been no recent activity on this issue. and removed no-recent-activity There has been no recent activity on this issue. labels Aug 5, 2024
using TypeSpec.Rest;
using TypeSpec.Http;

namespace KeyVault;
Copy link
Member

@heaths heaths Aug 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either here or in a client TSP, this or something else should be changed so we generate a SecretClient and not a KeyVaultClient.

See Azure/autorest.rust#86 for context.

@AzureRestAPISpecReview AzureRestAPISpecReview added BreakingChangeReviewRequired <valid label in PR review process>add this label when breaking change review is required and removed VersioningReviewRequired <valid label in PR review process>add this label when versioning review is required labels Oct 31, 2024
@heaths
Copy link
Member

heaths commented Nov 5, 2024

@mccoyp
Copy link
Member Author

mccoyp commented Nov 6, 2024

Lots of breaking changes reported: https://github.com/Azure/azure-rest-api-specs/pull/29249/checks?check_run_id=32562321385

The majority of these are from path formatting changes (e.g. /secrets/{secret-name} --> /secrets/{secretName}), but some are just now being identified because of moving the TypeSpec directory from keyvault/data-plane to keyvault. Addressing them now.

@gracewilcox gracewilcox force-pushed the mccoyp/kv-secrets-tsp branch 2 times, most recently from aa125b3 to 710421c Compare November 7, 2024 23:41
* at the end of the retention interval.
*/
@visibility("read")
recoveryLevel?: string; // DeletionRecoveryLevel, but this should be treated as an opaque string.
Copy link
Member

@maorleger maorleger Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👋 I think this should be DeletionRecoveryLevel and not a string IIUC - because DeletionRecoveryLevel is not actually used in routes it is not generated (at least by the typespec-ts codegen) - can we change that?

When I change the type to DeletionRecoveryLevel I see the correct code being generated (including extensible enum patterns)

Copy link
Member Author

@mccoyp mccoyp Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to get DeletionRecoveryLevel generated by using @@access(DeletionRecoveryLevel, Access.public) in our client.tsp. For example, from another library:

@@access(DetectionModel, Access.public);

EDIT: @@access didn't actually seem to work, but local testing showed that @@usage is what we'd want -- PR has been updated!

Copy link
Member

@maorleger maorleger Nov 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But why not use DeletionRecoveryLevel here instead of adding a client.tsp customization - does it not generate the correct thing? Just trying to understand the reasoning behind not using recoveryLevel?: DeletionRecoveryLevel when DeletionRecoveryLevel is defined as a string union already

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a plain string instead of the enum was a change request from Heath; "[i]t's response-only and should not be compared, which is why none of our client libraries should ship it as an enum".

Copy link
Member

@maorleger maorleger Nov 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok, got it! Thanks for the context.

So @heaths do you consider it a bug that JS shipped KnownDeletionRecoveryLevel as part of our public API? I think what you said makes sense re: response-only just want to triple-check - if an extensible enum value can only show up in a response, never as an input, we should not be shipping the Known enum values as part of our public API is that accurate? (I know the ship has sailed on this one for JS, but mostly asking for my future self)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BreakingChangeReviewRequired <valid label in PR review process>add this label when breaking change review is required data-plane KeyVault TypeSpec Authored with TypeSpec
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

8 participants