Skip to content

Latest commit

 

History

History
546 lines (390 loc) · 31.4 KB

CONTRIBUTING.md

File metadata and controls

546 lines (390 loc) · 31.4 KB
Raw

Contributing

Component Build Status
Management Libraries Build Status

Prerequisites:

GENERAL THINGS TO KNOW:

Client Libraries are sdks used to interact with azure resources at application run time while Management Libraries are those used to manage (create/modify/delete) Azure resources.

Build Repo : To build all Client and Management libraries together. Invoke dotnet build build.proj from the root of the repo.
To scope to a single service supply a scope property e.g. dotnet build build.proj /p:Scope=servicebus. This will build both client and management projects in the specified service. If using msbuild you must run restore first. See below for how to build the client or management libraries independently.

Path Length : To account for the 260 characters Path Length Limit encountered on windows OS, file paths within the repo is keep below 210 characters. This gives you a runway of 49 characters as clone path for your repo. Paths longer that 260 characters will cause the build to break on windows OS and on CI. Assuming you clone to the default VisualStudio location such that the root of your clone is C:\Users\\**USERNAME**\Source\Repos\azure-sdk-for-net your username will have to be 9 characters long to avoid build errors caused by long paths. Consider using C:\git as you clone path.

Dependencies : To ensure that the same versions of dependencies are used for all projects in the repo, package versions are managed from a central location in eng\Packages.Data.props. When adding package references you should first ensure that an Update reference of the package with the version exist in the Packages.Data.props then Include the reference without the version in your .csproj. Contact [email protected] if you need to change versions for packages already present in Packages.Data.props

Line Endings : If working on windows OS ensure git is installed with Checkout Windows-style, commit Unix-style option or core.autocrlf set to true in git config. If working on Unix based Linux or MacOS ensure git is installed with Checkout as-is, commit Unix-style option or core.autocrlf set to input in git config

Management Libraries

TO BUILD:

  1. Open any solution, eg "SDKs\Compute\Microsoft.Azure.Management.Compute.sln"
  2. Build solution from Visual Studio

Single Service from Command Line

  1. Open Developer Command Prompt
  2. From the root directory
  3. Invoke msbuild eng\mgmt.proj /p:scope=Compute

Build without any scope will build all management SDK's.

Create single nuget package

In order to build one package and run it's test msbuild eng\mgmt.proj /t:CreateNugetPackage /p:scope=Compute Nuget package will be created in root directory under \artifacts\packages\Debug (default configuration is Debug)

TO TEST:

Using Visual Studio:

  1. Build project in Visual Studio.
  2. Test Explorer window will get populated with tests. Select test and Run or Debug

Using the command line:

Run e.g. msbuild eng\mgmt.proj /t:"Runtests" /p:Scope=Compute In the above example RunTests will build and run tests for Compute only or you can use command line CLI:

dotnet test Compute\Microsoft.Azure.Management.Compute\tests\Microsoft.Azure.Management.Tests.csproj

Non-Windows command line build

Now you can use the same command on non-windows as above for e.g. on Ubuntu you can do something like below:

  • dotnet msbuild eng\mgmt.proj /t:Build /p:scope=Compute
  • dotnet msbuild eng\mgmt.proj /t:RunTests /p:scope=Compute
  • dotnet msbuild eng\mgmt.proj /t:CreateNugetPackage /p:scope=Compute
  • dotnet msbuild build.proj /t:Util /p:UtilityName=InstallPsModules

Code Coverage

If you want to enable code coverage reporting, on the command line pass /p:CollectCoverage=true like so:

dotnet tool restore
dotnet test /p:CollectCoverage=true

On developers' machines, you can open index.html from within the TestResults directory in each of your test projects. Coverage reports can also be found in Azure Pipelines on the "Code Coverage" tab after a pull request validation build completes. All covered projects should have 70% or better test coverage.

By default, all Azure.* libraries are covered, and any project that sets the IsClientLibrary=true MSBuild property. To exclude a project, set ExcludeFromCodeCoverage=true in the project's MSBuild properties before other targets are imported.

Update build tools

Build tools are now downloaded as part of a nuget package under root\restoredPackages\microsoft.internal.netsdkbuild.mgmt.tools If for any reason there is an update to the build tools, you will then need to first delete directory root\restoredPackages\microsoft.internal.netsdkbuild.mgmt.tools and re-execute your build command. This will simply get the latest version of build tools.

TO CREATE NEW LIBRARY USING TEMPLATE

We have created a dotnet template to make creating new management SDK library easier than ever.

See (README file)[(https://github.com/Azure/azure-sdk-for-net/blob/main/eng/templates/README.md)].

Client Libraries

TO BUILD:

Single Service from Command Line

  1. Open Developer Command Prompt
  2. Navigate to service directory e.g. "sdk\eventhub"
  3. Invoke dotnet build
  4. or Build the service.proj in the repo root, passing the directory name of the specific service as a property. e.g. dotnet build eng\service.proj /p:ServiceDirectory=eventhub

Single Service from Visual Studio

  1. Open any data-plane solution e.g. "sdk\eventhub\Microsoft.Azure.EventHubs.sln"
  2. Build solution from Visual Studio

All Client Services from Command Line

  1. Open Developer Command Prompt
  2. Navigate to repository root directory
  3. Invoke dotnet build eng\service.proj

Support for Visual Studio Code & Dev Containers

This repository has been configured with support for Visual Studio Code's dev container extension to make it easier to get started working on code without needing to know about how to setup all the pre-requisites. Configuration for dev containers is contained within the .devcontainer folder off the root directory.

To get started:

  1. Install and configure Docker for your platform.
  2. Install the Remote Development extension into Visual Studio Code.
  3. Clone the repository to your local workstation.
  4. Open Visual Studio Code at the root of the reposiory.
  5. Select "Reopen in Container" when prompted.

After a few moments of initial configuration Visual Studio Code will launch the container with all dependencies (.NET SDK etc) pre-installed.

TO TEST:

Single Service from Command Line

  1. Open Developer Command Prompt
  2. Navigate to service directory e.g. "sdk\eventhub"
  3. Invoke dotnet test --filter TestCategory!=Live (Skips live tests)
  4. or run test against service.proj in the repo root, passing the directory name of the specific service as a property. e.g. dotnet test eng\service.proj /p:ServiceDirectory=eventhub --filter TestCategory!=Live

Single Service from Visual Studio

  1. Build.
  2. Test Explorer window will get populated with tests. Select test and Run or Debug

All Client Services from Command Line

  1. Open VS 2019 Command Prompt
  2. Navigate to repository root directory
  3. Invoke dotnet test eng\service.proj --filter TestCategory!=Live

Live testing

Before running or recording live tests you need to create live test resources. Many of the client libraries make use of the Azure Core Test Framework to provide the basis for the live test infrastructure, including the ability to record Live tests so that they can be run without access to Azure resources. The Test Framework documentation provides more context around test recordings and other features.

To run live tests after creating live resources:

  1. Open Developer Command Prompt
  2. Navigate to service directory e.g. "sdk\keyvault"
  3. Invoke dotnet test

Some live tests may have additional steps for setting up live testing resources. See the CONTRIBUTING.md file for the service you wish to test for additional information or instructions.

Testing Against Latest Versions of Client Libraries

In some cases, you might want to test against the latest versions of the client libraries. i.e. version not yet published to nuget. For this scenario you should make use of the UseProjectReferenceToAzureClients property which when set to true will switch all package references for client libraries present in the build to project references. This result in testing against the current version of the libraries in the repo. e.g. dotnet test eng\service.proj /p:ServiceDirectory=eventhub --filter TestCategory!=Live /p:UseProjectReferenceToAzureClients=true

Public API additions

If you make public API changes or additions, the eng\scripts\Export-API.ps1 script has to be run to update public API listings. This generates a file in the library's directory similar to the example found in sdk\template\Azure.Template\api\Azure.Template.netstandard2.0.cs.

Running the script for a project in sdk\tables would look like this:

eng\scripts\Export-API.ps1 tables

Updating Sample Snippets

If the specific client library has sample snippets in markdown format, they were most likely created with help of the eng\scripts\Update-Snippets.ps1 script. Any changes made to the snippet markdown should be done via updating the corresponding C# snippet code and subsequently running the script.

Running the script for a project, for example in sdk\keyvault, would look like this:

eng\scripts\Update-Snippets.ps1 keyvault

When run, the code regions in the format below (where <snippetName> is the name of the snippet):

#region Snippet:<snippetName>
//some sample code
string snippet = "some snippet code";

// The snippet updater defines the SNIPPET directive while parsing. You can use #if SNIPPET to filter lines in or out of the snippet.
#if SNIPPET
snippet = "value that would never pass a test but looks good in a sample!";
#else
string ignored = "this code will not appear in the snippet markdown";
#endif

#endregion

will be mapped to any markdown file with a corresponding code region in the format below where the snippet names match:

```C# Snippet:<snippetName>

```

See the following example of a snippet C# file and a snippet markdown file. Note that snippet names need to be globally unique under a given service directory.

Snippets also can be integrated into XML doc comments. For example:

/// <summary>
/// Some Property.
/// </summary>
/// <example>
/// This is an example of using a snippet in XML docs.
/// <code snippet="Snippet:<snippetName>">
/// // some sample code.
/// string snippet = "some snippet code";
/// </code>
public string SomeProperty { get; set; }

For general information about samples, see the Samples Guidelines

Updating Source on Build

You can run eng\scripts\Export-API.ps1 and eng\scripts\Update-Snippets.ps1 simultaneously as part of the build by setting as true either:

  1. The property UpdateSourceOnBuild
  2. The Environment variable AZURE_DEV_UPDATESOURCESONBUILD=true

e.g.

dotnet build eng\service.proj /p:ServiceDirectory=eventhub /p:UpdateSourceOnBuild=true

API Compatibility Verification

.NET is using the ApiCompat tool to enforce API compatibility between versions. Builds of GA'ed libraries will fail locally and in CI if there are breaking changes.

How it works

Each library needs to provide a ApiCompatVersion property which is set to the last GA'ed version of the library that will be used to compare APIs with the current to ensure no breaks have been introduced. Projects with this property set will download the specified package and the ApiCompat (Microsoft.DotNet.ApiCompat) tools package as part of the restore step of the project. Then as a post build step of the project it will run ApiCompat to verify the current APIs are compatible with the last GA'ed version of the APIs. For libraries that wish to disable the APICompat check they can remove the ApiCompatVersion property from their project. Our version bump automation will automatically add or increment the ApiCompatVersion property to the project when it detects that the version it is changing was a GA version which usually indicates that we just shipped that GA version and so it should be the new baseline for API checks.

Releasing a new version of a GA'ed libary

Since the eng/Packages.Data.props is currently maintained manually, you will need to update the version number for your library in this file when releasing a new version.

NuGet Package Dev Feed

The Azure SDK for .NET releases packages daily from our CI pipeline to our NuGet package dev feed to help developers use and test new libraries before they are officially released to NuGet.

Dev Feed Package Browser:

Dev Feed Package Source:

1. Add NuGet Package Dev Feed

You have multiple options for referencing the dev feed. You can either add it via the NuGet CLI or manually edit your NuGet.Config file.

NuGet CLI

You can add the dev feed using the NuGet CLI, which will add it to the NuGet.Config file.

nuget sources add -Name "Azure SDK for .NET Dev Feed" -Source "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json"

You can then view the list of NuGet package sources with this command:

nuget sources

NuGet Config file

You can add the dev feed to your NuGet.Config file, which can be at the Solution, User, or Computer level. See NuGet.Config file locations and uses to locate your NuGet.Config file.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="Azure SDK for .NET Dev Feed" value="https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json" />
  </packageSources>
  <disabledPackageSources>
    <clear />
  </disabledPackageSources>
</configuration>

You can place a NuGet.Config file in the root of your solution. Projects within the solution will use the feed defined in that file.

2. Find NuGet Package

You can use the following options to find the available dev feed packages:

  1. Search the Azure SDK for .NET Dev Feed: https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-net
  2. In Visual Studio, use the Package Manager UI, be sure to check "Include prerelease".
  3. Use the NuGet CLI, for example nuget list azure.identity -Prerelease -Allversions

3. Reference NuGet Package

Now that you have found the package you want to use, it is time to add it to your project file.

As you can see in the example below, we want to use the Azure.Data.Tables version 3.0.0-alpha.*. By using the * in the version number each dotnet restore will pull the latest version from the dev feed.

<Project Sdk="Microsoft.NET.Sdk">
  <ItemGroup>
    <PackageReference Include="Azure.Data.Tables" Version="3.0.0-alpha.*" />
  </ItemGroup>
</Project>

Preparing a new library release

To prepare a package for release you should make use of .\eng\common\scripts\Prepare-Release.ps1 script passing it appropriate arguments for the package intended for release. This script will correctly update the package version and changelog in the repo as well as update the DevOps release work items for that release.

.\eng\common\scripts\Prepare-Release.ps1 <PackageName> [<ServiceDirectory>] [<ReleaseDate>] [-ReleaseTrackingOnly]
  • <PackageName> - Should match the full exact package name for the given ecosystem (i.e. "Azure.Core", "azure-core", "@azure/core", etc).
  • <SerivceDirectory> - Optional: Should be the exact directory name where the package resides in the repo. This is usually the same as the service name in most cases (i.e. "sdk<service_directory>" e.g. "core"). The parameter is optional and if provided will help speed-up the number of projects we have to parse to find the matching package project.
  • <ReleaseDate> - Optional: provide a specific date for when you plan to release the package. If one isn't given then one will be calculated based on the normal monthly shipping schedule.
  • <ReleaseTrackingOnly> - Optional: Switch that if passed will only update the release tracking data in DevOps and not update any versioning info or do validation in the local repo.

On-boarding New Libraries

Project Structure

In sdk\< Service Name >, you will find projects for services that have already been implemented

  1. Client library projects needs to use the $(RequiredTargetFrameworks) (defined in eng/Directory.Build.Data.props) property in its TargetFramework while management library projects should use $(SdkTargetFx) (defined in AzSdk.reference.props)
  2. Projects of related packages are grouped together in a folder following the structure specified in Repo Structure
    • Client library packages are in a folder name like Microsoft.Azure.< ServiceName >
    • Management library packages are in a folder named like Microsoft.Azure.Management.< Resource Provider Name >
  3. Each shipping package contains a project for their generated and /or Customization code
    • The folder 'Generated' contains the generated code
    • The folder 'Customizations' contains additions to the generated code - this can include additions to the generated partial classes, or additional classes that augment the SDK or call the generated code
    • The file generate.cmd, used to generate library code for the given package, can also be found in this project

Standard Process

  1. Create fork of Azure REST API Specs
  2. Create fork of Azure SDK for .NET
  3. Create your Swagger specification for your HTTP API. For more information see Introduction to Swagger - The World's Most Popular Framework for APIs
  4. Install the latest version of AutoRest and use it to generate your C# client. For more info on getting started with AutoRest, see the AutoRest repository
  5. Create a branch in your fork of Azure SDK for .NET and add your newly generated code to your project. If you don't have a project in the SDK yet, look at some of the existing projects and build one like the others.
  6. MANDATORY: Add or update tests for the newly generated code.
  7. Once added to the Azure SDK for .NET, build your local package using client or management library instructions shown in the above sections.
  8. For management libraries run eng\scripts\Update-Mgmt-Yml.ps1 to update PR include paths in eng\pipelines\mgmt.yml
  9. A Pull request of your Azure SDK for .NET changes against master branch of the Azure SDK for .NET
  10. The pull requests will be reviewed and merged by the Azure SDK team

New Resource Provider

  1. If you have never created an SDK for your service before, you will need the following things to get your SDK in the repo
  2. Follow the standard process described above.
  3. Project names helps in using basic heuristics in finding projects as well it's associated test projects during CI process.
  4. Create a new directory using the name of your service as specified in azure-rest-api-specs/specification Repo
  5. Follow the the directory structure below
sdk\<service name>\<package name>\README.md
sdk\<service name>\<package name>\*src*
sdk\<service name>\<package name>\*tests*
sdk\<service name>\<package name>\*samples*

e.g.

sdk\eventgrid\Microsoft.Azure.EventGrid\src\Microsoft.Azure.EventGrid.csproj
sdk\eventgrid\Microsoft.Azure.EventGrid\tests\Microsoft.Azure.EventGrid.Tests.csproj
sdk\eventgrid\Microsoft.Azure.Management.EventGrid\src\Microsoft.Azure.Management.EventGrid.csproj
sdk\eventgrid\Microsoft.Azure.Management.EventGrid\tests\Microsoft.Azure.Management.EventGrid.Tests.csproj

Ensure that your service name is the same as it is specified in the azure-rest-api-specs/specification Repo, that your csproj files starts with Microsoft.Azure , that test files end with .Tests and that management plane project files contain .Management. If you are adding a new service directory, ensure that it is mapped to a friendly name at ServiceMapping

  1. Copy .csproj from any other .csproj and update the following information in the new .csproj

    Project Properties
    AssemblyTitle
    Description
    VersionPrefix
    PackageTags
    PackageReleaseNotes

PackageReleaseNotes are important because this information is displayed on https://www.nuget.org when your nuget package is published

  1. Copy existing generate.ps1 file from another service and update the ResourceProvider name that is applicable to your SDK. Resource provider refers to the relative path of your REST spec directory in Azure-Rest-Api-Specs repository During SDK generation, this path helps to locate the REST API spec from the https://github.com/Azure/azure-rest-api-specs

On-boarding New generated code library

  1. Install templates for both data-plane and management-plane (control-plan) SDKs:

    # Data-plane SDK
dotnet new --install sdk/template
dotnet new azuresdk --name Azure.MyService --output sdk/myservice --ServiceDirectory myservice --ProjectName Azure.MyService

# Management-plane SDK
dotnet new --install eng/templates/Azure.ResourceManager.Template
dotnet new azuremgmt --help

    There are several options available for management-plane SDKs. You can see all those available with --help as shown above, or read about them in our documentation.

    This will perform most of the renames, namespace fix-ups, etc., for you automatically; though, be sure to check all files - especially the README.md file(s) - for required manual changes. If the template is already installed, this same command will upgrade it.

  2. Modify autorest.md to point to you Swagger file or central README.md file. E.g.

    input-file:
    - https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/specification/storage/resource-manager/Microsoft.Storage/stable/2019-06-01/blob.json
    - https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/specification/storage/resource-manager/Microsoft.Storage/stable/2019-06-01/file.json
    - https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/specification/storage/resource-manager/Microsoft.Storage/stable/2019-06-01/storage.json
    require: https://github.com/Azure/azure-rest-api-specs/blob/49fc16354df7211f8392c56884a3437138317d1f/specification/azsadmin/resource-manager/storage/readme.md

  3. Run dotnet build /t:GenerateCode in src directory of the project (e.g. net\sdk\storage\Azure.Management.Storage\src). This would run AutoRest and generate the code. (NOTE: this step requires Node 14).

  4. For management plan libraries add azure-arm: true setting to autorest.md client constructors and options would be auto-generated. For data-plane libraries follow the next two steps.

  5. Add a *ClientOptions type that inherits from ClientOptions and has a service version enum:

    namespace Azure.Management.Storage
{
    public class StorageManagementClientOptions: ClientOptions
    {
        private const ServiceVersion Latest = ServiceVersion.V2019_06_01;
        internal static StorageManagementClientOptions Default { get; } = new StorageManagementClientOptions();

        public StorageManagementClientOptions(ServiceVersion serviceVersion = Latest)
        {
            VersionString = serviceVersion switch
            {
                ServiceVersion.V2019_06_01 => "2019-06-01",
                _ => throw new ArgumentOutOfRangeException(nameof(serviceVersion))
            };
        }

        internal string VersionString { get; }

        public enum ServiceVersion
        {
#pragma warning disable CA1707 // Identifiers should not contain underscores
            V2019_06_01 = 1
#pragma warning restore CA1707
        }
    }
}

  6. Add public constructors to all the clients using a partial class.

     public partial class FileSharesClient
    {
        public FileSharesClient(string subscriptionId, TokenCredential tokenCredential): this(subscriptionId, tokenCredential, StorageManagementClientOptions.Default)
        {
        }

        public FileSharesClient(string subscriptionId, TokenCredential tokenCredential, StorageManagementClientOptions options):
            this(new ClientDiagnostics(options), ManagementClientPipeline.Build(options, tokenCredential), subscriptionId, apiVersion: options.VersionString)
        {
        }
    }

Code Review Process

Before a pull request will be considered by the Azure SDK team, the following requirements must be met:

  • Prior to issuing the pull request:
    • All code must have completed any necessary legal signoff for being publicly viewable (Patent review, JSR review, etc.)
    • The changes cannot break any existing functional/unit tests that are part of the central repository.
      • This includes all tests, even those not associated with the given feature area.
    • Code submitted must have basic unit test coverage, and have all the unit tests pass. Testing is the full responsibility of the service team
      • Functional tests are encouraged, and provide teams with a way to mitigate regressions caused by other code contributions.
    • Code should be commented.
    • Code should be fully code reviewed.
    • Code should be able to merge without any conflicts into the dev branch being targeted.
    • Code should pass all relevant static checks and coding guidelines set forth by the specific repository.
    • All build warnings and code analysis warnings should be fixed prior to submission.
  • As part of the pull request (aka, in the text box on GitHub as part of submitting the pull request):
    • Proof of completion of the code review and test passes requirements above.
    • Identity of QA responsible for feature testing (can be conducted post-merging of the pull request).
    • Short description of the payload of pull request.
  • After the pull request is submitted:
    • Our SLA is 48 hours. When your PR is submitted someone on our team will be auto assigned the PR for review. No need to email us
    • MS internal folks, please reach out to us via our Teams channel or
    • Send an email to the Azure Developer Platform team [email protected] alias.
      • Include all interested parties from your team as well.
      • In the message, make sure to acknowledge that the legal signoff process is complete.

Once all of the above steps are met, the following process will be followed:

  • A member of the Azure SDK team will review the pull request on GitHub.
  • If the pull request meets the repository's requirements, the individual will approve the pull request, merging the code into the dev branch of the source repository.
    • The owner will then respond to the email sent as part of the pull request, informing the group of the completion of the request.
  • If the request does not meet any of the requirements, the pull request will not be merged, and the necessary fixes for acceptance will be communicated back to the partner team.

Client Library Tested OSs and .NET Versions

Linux (Ubuntu 16.04) MacOS 10.13 Windows Server 2016
.NET Core 2.1 x x x

Issues with Generated Code

Much of the management plane SDK code is generated from metadata specs about the REST APIs. Do not submit PRs that modify generated code. Instead,

  • File an issue describing the problem,
  • Refer to the the AutoRest project to view and modify the generator, or
  • Add additional methods, properties, and overloads to the SDK by adding classes in the 'Customizations' folder of a project

Versioning

For more information on how we version see Versioning

Breaking Changes

For information about breaking changes see Breaking Change Rules