From 1757325aaac8c03fec2e5797b97532f7e53aa288 Mon Sep 17 00:00:00 2001 From: Mariana Dematte Date: Mon, 25 Nov 2024 18:41:12 +0100 Subject: [PATCH 1/8] Rough notes from meetings --- documentation/specs/proposed/Cache EVAL.md | 46 ++++++++++++ documentation/specs/proposed/VS-SDK thingy.md | 62 ++++++++++++++++ documentation/specs/proposed/parfStar.md | 72 +++++++++++++++++++ 3 files changed, 180 insertions(+) create mode 100644 documentation/specs/proposed/Cache EVAL.md create mode 100644 documentation/specs/proposed/VS-SDK thingy.md create mode 100644 documentation/specs/proposed/parfStar.md diff --git a/documentation/specs/proposed/Cache EVAL.md b/documentation/specs/proposed/Cache EVAL.md new file mode 100644 index 00000000000..50bbf27c4a3 --- /dev/null +++ b/documentation/specs/proposed/Cache EVAL.md @@ -0,0 +1,46 @@ +# Evaluation goes fast +*The objective of this one-pager is to provide clear and concise information about a feature or project to foster inclusion and collaboration by enabling anyone who is interested to learn the essential information, such as the project's goals, motivation, impact, and risks. The points and questions in each section are illustrative, they are not intended as a definite list of questions to be answered.* + +## Description + - Short summary of the feature or effort. + + - We could do profiling + - Jit compilation o MSBuild itself. + - We could cache at eval + + Constraint - needs to work as it does today, but fast. We can break some edge cases. + +## Goals and Motivation + - What are we trying to achieve and why? + + We evaluate all the time, a lot of times we don't need to be doing that - like up-to-date builds. It would increase the dev loop performance. Evaluate go fast. + +## Impact +Guidance: This document should not be an OKR. Let's keep the impact general here. +Questions to consider: + - What is the impact? + - Who will benefit from the feature? + - How does success look like? + - Are there any metrics or key results that could be quantified? + +## Stakeholders +Questions to consider: + - Who are the stakeholders? + - For projects with concrete stakeholders, once the project is done how does a successful handover to the stakeholder look like? + +## Risks +Questions to consider: + - Will the effort or feature cause breaking changes for existing consumers? + - Are there any unknowns that might derail the effort? + - Are there any dependencies we don’t have control over? + - Is there a hard deadline that needs to be met? + - What threatens completion of the feature? + - What is the impact of failing to deliver the feature? + +## Cost +Questions to consider: + - What is the estimated cost of the end-to-end effort? + - How accurate is the cost? + +## Plan + - High-level milestones, with estimate effort for tasks. \ No newline at end of file diff --git a/documentation/specs/proposed/VS-SDK thingy.md b/documentation/specs/proposed/VS-SDK thingy.md new file mode 100644 index 00000000000..5c3e6a13ad4 --- /dev/null +++ b/documentation/specs/proposed/VS-SDK thingy.md @@ -0,0 +1,62 @@ +# Decoupling VS builds of SDK projects +*The objective of this one-pager is to provide clear and concise information about a feature or project to foster inclusion and collaboration by enabling anyone who is interested to learn the essential information, such as the project's goals, motivation, impact, and risks. The points and questions in each section are illustrative, they are not intended as a definite list of questions to be answered.* + +Ensure that all the logic from a build, from an SDK project comes from the SDK independently of where it is being built. + +## Description + - Short summary of the feature or effort. + +## Goals and Motivation + - What are we trying to achieve and why? + +Consistency of end-user build experience +Decoupling the SDK from VS +Isolate the SDK builds to use only their components. + +Why: +Experience of tooling authors + - Roslyn analyzer authors + - MSBuild Task authors + +experience of end users + - Anyone who uses the analyzers or source egenrators + +Tooling authors need to target NetStandard2.0, or multi target and dependencies in multitargets is annoying. If you don't match the roslyn version for VS the analyzers and generators don't work. + +## Impact +Multiple layer of impact: + - Project construction + - reduce cost of development for internal teams that contribute to Roslyn analyzers / source build generators and MSBuild Tasks. + + End users will not experience mismatch between analyzer versions. And they will be sure that the build will be the same as the command line invocation. + +## Stakeholders +internal folks are the ones that will continue the work to fully complete the feature. + - VS Perf team: + - Project System team: + - Roslyn team: Handover - once we're sending the environmental variable, they can enable the use of the core compiler in VS. The second handover is the same as the rest: you can write only .net code tasks and the sdk projects will build successfully. + +These are the folks that will benefit of these changes: + - Analyzer / source generator author + - MSBuild Task authors + +## Risks + - There might be a performance hit on VS depending on how many non-framework tasks the project needs to load. As we can't do some pre-loading. + - There should be no breaking from SDK only users. The IDE tooling might have a different version, which leads to discrepancy on partial builds. + - We are early in the development effort, so if later there is a larger impact on perf or other issues, the effort in general might be delayed(?), but our part would already have been completed. + - Deadline: no concrete deadline, but early in the preview cycle (preview 4-5) to get a sense of consequences of this change. + - If we don't do this: Worse experience for devs that work in the area. + +## Cost +1. Dev week's time +2. Dev 1-2 months time. +3. Dev 1 week if things do not go wrong at all. + +## Plan + 1. Ensure that MSBuild.exe provides the same execution state as the dotnet command line invocation. MSExtensionPath (cant do that), DotNetHostPath, MSSDKsPath (cant do that). + - Low effort, should be done first. + 2. Implement .NET core task host, so we can execute core version of tasks. + - Get Rainer feedback, seems like a medium sized. + 3. Load common targets from the SDK and not .NetFramework (the VS version of it). This might be out of scope for .NET 10 + - Medium effort, can have behavioral changes. + diff --git a/documentation/specs/proposed/parfStar.md b/documentation/specs/proposed/parfStar.md new file mode 100644 index 00000000000..72a46e2accb --- /dev/null +++ b/documentation/specs/proposed/parfStar.md @@ -0,0 +1,72 @@ +# One Pager Template +*The objective of this one-pager is to provide clear and concise information about a feature or project to foster inclusion and collaboration by enabling anyone who is interested to learn the essential information, such as the project's goals, motivation, impact, and risks. The points and questions in each section are illustrative, they are not intended as a definite list of questions to be answered.* + +## Description + - Short summary of the feature or effort. + +New features and fixes for .NET 10 investment. + - Running it nicely [1 dev week] + - Report is working [2 dev weeks] + - Performance tests for new features - making it easier to make these / docs. [3 dev days per feature] + - BuildCheck perf tests + - Check Stability of perf tests / fixing those - so there is not a lot of noise around tests [2 dev weeks] + - Should be done through multiple iterations on a test + - Report needs to be reworked + - We can maybe have more `msbuild.exe`. but not VS api [1 dev week] + - Timeboxed collect and act on feedback from our team. Includes perf investigations that needs to be conducted. [1-2 dev months depending on dev feedback / requests] + - Add more tests: check logger performance with different verbosities, buildCheck. [1 dev week] + +possible directions for perfStar / next versions: + - More measurements. Like, dotnet counters tool. [3 dev weeks] + - Trace collection, when something is on automatically turn it on. [2-3 dev weeks] + - Report improvements: + - Comparison between different versions that is not main [2 dev weeks] + - Automatic detection of perf issues. Don't have someone look at the report to have news about regression. [1 dev month] + - Run VS API tests? There are some problems that we would need for VS specific tests [4 dev weeks - optional feature] + +## Goals and Motivation + - What are we trying to achieve and why? +Go fast + +## Impact +Guidance: This document should not be an OKR. Let's keep the impact general here. +Questions to consider: + - What is the impact? + - Who will benefit from the feature? + - How does success look like? + - Are there any metrics or key results that could be quantified? + +We can be sure to go faster +Enable MSBuild team to track performance, identify low hanging fruits to improve performance, and watch new feature peformance. + +## Stakeholders +Questions to consider: + - Who are the stakeholders? + - For projects with concrete stakeholders, once the project is done how does a successful handover to the stakeholder look like? +Us. + + +## Risks +Questions to consider: + - Will the effort or feature cause breaking changes for existing consumers? + - Are there any unknowns that might derail the effort? + - Are there any dependencies we don’t have control over? + - Is there a hard deadline that needs to be met? + - What threatens completion of the feature? + - What is the impact of failing to deliver the feature? + +Dependencies: Crank, which is owned by ASP.NET team. This can cause problems with the machine set-up since they are responsible for this. + +Deadlines:No hard deadlines. Just nice to have for the team. +Threat: Randomization for the team. Security issues that come up. + +Impact of delivery failure. Less numbers in performance improvement for MSBuild features. + + +## Cost +Questions to consider: + - What is the estimated cost of the end-to-end effort? + - How accurate is the cost? + +## Plan + - High-level milestones, with estimate effort for tasks. \ No newline at end of file From a0a1c8b9489d874da2b13724043878f91de355a4 Mon Sep 17 00:00:00 2001 From: Mariana Dematte Date: Wed, 27 Nov 2024 10:44:31 +0100 Subject: [PATCH 2/8] decoupling write-up complete --- documentation/specs/proposed/VS-SDK thingy.md | 62 ------------------- .../specs/proposed/decoupling-vs-sdk.md | 43 +++++++++++++ 2 files changed, 43 insertions(+), 62 deletions(-) delete mode 100644 documentation/specs/proposed/VS-SDK thingy.md create mode 100644 documentation/specs/proposed/decoupling-vs-sdk.md diff --git a/documentation/specs/proposed/VS-SDK thingy.md b/documentation/specs/proposed/VS-SDK thingy.md deleted file mode 100644 index 5c3e6a13ad4..00000000000 --- a/documentation/specs/proposed/VS-SDK thingy.md +++ /dev/null @@ -1,62 +0,0 @@ -# Decoupling VS builds of SDK projects -*The objective of this one-pager is to provide clear and concise information about a feature or project to foster inclusion and collaboration by enabling anyone who is interested to learn the essential information, such as the project's goals, motivation, impact, and risks. The points and questions in each section are illustrative, they are not intended as a definite list of questions to be answered.* - -Ensure that all the logic from a build, from an SDK project comes from the SDK independently of where it is being built. - -## Description - - Short summary of the feature or effort. - -## Goals and Motivation - - What are we trying to achieve and why? - -Consistency of end-user build experience -Decoupling the SDK from VS -Isolate the SDK builds to use only their components. - -Why: -Experience of tooling authors - - Roslyn analyzer authors - - MSBuild Task authors - -experience of end users - - Anyone who uses the analyzers or source egenrators - -Tooling authors need to target NetStandard2.0, or multi target and dependencies in multitargets is annoying. If you don't match the roslyn version for VS the analyzers and generators don't work. - -## Impact -Multiple layer of impact: - - Project construction - - reduce cost of development for internal teams that contribute to Roslyn analyzers / source build generators and MSBuild Tasks. - - End users will not experience mismatch between analyzer versions. And they will be sure that the build will be the same as the command line invocation. - -## Stakeholders -internal folks are the ones that will continue the work to fully complete the feature. - - VS Perf team: - - Project System team: - - Roslyn team: Handover - once we're sending the environmental variable, they can enable the use of the core compiler in VS. The second handover is the same as the rest: you can write only .net code tasks and the sdk projects will build successfully. - -These are the folks that will benefit of these changes: - - Analyzer / source generator author - - MSBuild Task authors - -## Risks - - There might be a performance hit on VS depending on how many non-framework tasks the project needs to load. As we can't do some pre-loading. - - There should be no breaking from SDK only users. The IDE tooling might have a different version, which leads to discrepancy on partial builds. - - We are early in the development effort, so if later there is a larger impact on perf or other issues, the effort in general might be delayed(?), but our part would already have been completed. - - Deadline: no concrete deadline, but early in the preview cycle (preview 4-5) to get a sense of consequences of this change. - - If we don't do this: Worse experience for devs that work in the area. - -## Cost -1. Dev week's time -2. Dev 1-2 months time. -3. Dev 1 week if things do not go wrong at all. - -## Plan - 1. Ensure that MSBuild.exe provides the same execution state as the dotnet command line invocation. MSExtensionPath (cant do that), DotNetHostPath, MSSDKsPath (cant do that). - - Low effort, should be done first. - 2. Implement .NET core task host, so we can execute core version of tasks. - - Get Rainer feedback, seems like a medium sized. - 3. Load common targets from the SDK and not .NetFramework (the VS version of it). This might be out of scope for .NET 10 - - Medium effort, can have behavioral changes. - diff --git a/documentation/specs/proposed/decoupling-vs-sdk.md b/documentation/specs/proposed/decoupling-vs-sdk.md new file mode 100644 index 00000000000..0e93de9ea6c --- /dev/null +++ b/documentation/specs/proposed/decoupling-vs-sdk.md @@ -0,0 +1,43 @@ +# Decoupling VS builds of SDK projects +The behaviour of an SDK project, when built in Visual Studio and when built in the DotNet CLI can vary, as different pieces of imports can be fetched from either build agent. This situation is not great specially for some developers as it increases their coding workload. To solve this, we want to ensure that all logic from a build from an *SDK project* comes from the SDK, independently of where the build is processed. + +## Goals and Motivation + +We are aiming for: + - Consistent end-user eperience for build in either DotNet CLI or Visual Studio. + - Isolating Dotnet SDK build to only their components. + - Decoupling the Dotnet SDK from VS. + + +The reason we are persuing this is for a better experience when using or writting Roslyn analyzers and MSBuild Tasks. Currently tooling authors need to target NetStandard2.0 for their projects to be recognized by VS. Another options is to multi-target but it takes a lot more effort and time spent on that. Another aspect is the user experience, if the Roslyn version for VS analyzers and generators doesn't match the one in Visual Studio, they do not work. + + +## Impact +There are a few area of impact: + - SDK style project builds will be more stable between VS and CLI builds, as the tooling will not be devided between different versions. + - Reduced cost of development for external and internal teams that contribute to Roslyn Analyzers, SourceBuild generators, or MSBuild Tasks. + - End-user will not experience mismatch between analyzer versions, and confirmation that their SDK style builds will behave the same way in VS and in the command line. + +## Stakeholders +For the internal stakeholder, we have the teams that will continue the work to fully complete the VS and SDK decoupling feature after our base work is done. There are two handovers in this project: + +1. Enabling the MSBuild.exe execution state to be the same as DotNet command line invocation so the Roslyn team can enable the use of their Core compiler in VS. +2. Tasks and other projects can be written in .NET core and the SDK projects will build successuflly in VS. This enables other teams to migrate their tasks to .NET core instead of keeping them targeting .NET Framework. + +The handovers should allow other teams to proceed with their work smoothly and no change in build behaviour should be present within MSBuild. + +## Risks +A few risks associated with this feature: + - There might be a performance hit on VS. It would depending on the amount of non-framwork tasks that the project will need to load when opening it in VS. The performance gain from pre-loading will not be available in this scenario. + - The VS tooling might present have a different version than the SDK installed, which might lead to discrepancy in partial builds. + - Our work is early in the development effort. If this feature is discovered to have too large of an impact in experience of performance the work might be delayed or discarded. + - There are no concrete deadlines for our part of the feature, but we are aiming for an early preview cycle, so we have a chance to measure the consequences and fix any issues that arise. + +## Plan + 1. Ensure that MSBuild.exe provides the same execution state as the dotnet command line invocation. + - This is should take around 1 dev week to complete, and will be handed over to Roslyn team. + 2. Implement .NET core task host, tasks can be executed on the .NET Core vresion instead of .NET framework. + - This should take 1 to 2 dev months to complete, including extensive testing. This would be handed over to internal teams that have custom tasks so they can be updated and tested. + 3. Load common targets from the .NET SDK and not from .NET NetFramework on VS. This work depends on other team's finilizing their part of the feature and might not be in scope for .NET 10. + - This should take a dev week, if there are no major bugs or regressions. + From 8a796957b44853c454a7b82dd6c9c0c8aa748552 Mon Sep 17 00:00:00 2001 From: Mariana Dematte Date: Wed, 27 Nov 2024 14:17:26 +0100 Subject: [PATCH 3/8] evaluation performance investigation --- documentation/specs/proposed/Cache EVAL.md | 46 ------------------- .../specs/proposed/evaluation-perf.md | 24 ++++++++++ 2 files changed, 24 insertions(+), 46 deletions(-) delete mode 100644 documentation/specs/proposed/Cache EVAL.md create mode 100644 documentation/specs/proposed/evaluation-perf.md diff --git a/documentation/specs/proposed/Cache EVAL.md b/documentation/specs/proposed/Cache EVAL.md deleted file mode 100644 index 50bbf27c4a3..00000000000 --- a/documentation/specs/proposed/Cache EVAL.md +++ /dev/null @@ -1,46 +0,0 @@ -# Evaluation goes fast -*The objective of this one-pager is to provide clear and concise information about a feature or project to foster inclusion and collaboration by enabling anyone who is interested to learn the essential information, such as the project's goals, motivation, impact, and risks. The points and questions in each section are illustrative, they are not intended as a definite list of questions to be answered.* - -## Description - - Short summary of the feature or effort. - - - We could do profiling - - Jit compilation o MSBuild itself. - - We could cache at eval - - Constraint - needs to work as it does today, but fast. We can break some edge cases. - -## Goals and Motivation - - What are we trying to achieve and why? - - We evaluate all the time, a lot of times we don't need to be doing that - like up-to-date builds. It would increase the dev loop performance. Evaluate go fast. - -## Impact -Guidance: This document should not be an OKR. Let's keep the impact general here. -Questions to consider: - - What is the impact? - - Who will benefit from the feature? - - How does success look like? - - Are there any metrics or key results that could be quantified? - -## Stakeholders -Questions to consider: - - Who are the stakeholders? - - For projects with concrete stakeholders, once the project is done how does a successful handover to the stakeholder look like? - -## Risks -Questions to consider: - - Will the effort or feature cause breaking changes for existing consumers? - - Are there any unknowns that might derail the effort? - - Are there any dependencies we don’t have control over? - - Is there a hard deadline that needs to be met? - - What threatens completion of the feature? - - What is the impact of failing to deliver the feature? - -## Cost -Questions to consider: - - What is the estimated cost of the end-to-end effort? - - How accurate is the cost? - -## Plan - - High-level milestones, with estimate effort for tasks. \ No newline at end of file diff --git a/documentation/specs/proposed/evaluation-perf.md b/documentation/specs/proposed/evaluation-perf.md new file mode 100644 index 00000000000..66fd2561bad --- /dev/null +++ b/documentation/specs/proposed/evaluation-perf.md @@ -0,0 +1,24 @@ +# Evaluation performance investigations +In the current effort to improve performance of MSBuild, we ideantified the evaluation as one of the focus areas of this effort. Evalution is the ifrst step when building, and it determines references, how projects are connected and what needs to be build. Because of this it runs in every single build, be it Design-time builds in Visual Studio, up-to-date builds or full builds. + +## Description +Current performance state of evaluation is mostly unkown, as it is not measured in any ways by the team. As such, we are unsure which specific areas can be improve. The investigation about this is necessary so we can identify weaknesses, and possible fixes. + + - We could do profiling + - Jit compilation o MSBuild itself. + - We could cache at eval + + Constraint - needs to work as it does today, but fast. We can break some edge cases. + +## Goals and Motivation +We are trying to make evaluation phase of the build more performant, since it is almost always executed any performance gain becomes noticeable. A performant evaluation phase would decrease build times in general, in CI cases it frees up resources, and in individual cases it can increase dev-loop performance by making up-to-date and incremental builds go faster. + +In this moment we are still in investigation phase, the obejective is to add code markers, so we can idetentify low hanging fixes, and improvement areas when testing builds within PerfStar. + +## Risks +One of the big risks is accidentally changing the current behaviour of evaluation. One of the constraints of improvement is that evaluation has the same behavior, with the exception of edge cases where we can sometimes change it. + +## Plan +The plan for evaluation at the moment is to add more code markers during execution so we can use PerfStar to have a detailed view of how long each part of evaluation phase takes. + +Larger changes to the evaluation are possible and under consideration for future iterations, like trying to cache the evaluation result in MSBuild. However we are focusing on investigation and performance gains with less work at the moment. \ No newline at end of file From 73ca8c325f9b6e0acd63ee94a3d73ce4b88c8f7e Mon Sep 17 00:00:00 2001 From: Mariana Dematte Date: Wed, 27 Nov 2024 15:41:34 +0100 Subject: [PATCH 4/8] perfStar pager --- documentation/specs/proposed/parfStar.md | 94 ++++++++---------------- 1 file changed, 30 insertions(+), 64 deletions(-) diff --git a/documentation/specs/proposed/parfStar.md b/documentation/specs/proposed/parfStar.md index 72a46e2accb..5d9a6a24d6e 100644 --- a/documentation/specs/proposed/parfStar.md +++ b/documentation/specs/proposed/parfStar.md @@ -1,72 +1,38 @@ -# One Pager Template -*The objective of this one-pager is to provide clear and concise information about a feature or project to foster inclusion and collaboration by enabling anyone who is interested to learn the essential information, such as the project's goals, motivation, impact, and risks. The points and questions in each section are illustrative, they are not intended as a definite list of questions to be answered.* - -## Description - - Short summary of the feature or effort. - -New features and fixes for .NET 10 investment. - - Running it nicely [1 dev week] - - Report is working [2 dev weeks] - - Performance tests for new features - making it easier to make these / docs. [3 dev days per feature] - - BuildCheck perf tests - - Check Stability of perf tests / fixing those - so there is not a lot of noise around tests [2 dev weeks] - - Should be done through multiple iterations on a test - - Report needs to be reworked - - We can maybe have more `msbuild.exe`. but not VS api [1 dev week] - - Timeboxed collect and act on feedback from our team. Includes perf investigations that needs to be conducted. [1-2 dev months depending on dev feedback / requests] - - Add more tests: check logger performance with different verbosities, buildCheck. [1 dev week] - -possible directions for perfStar / next versions: - - More measurements. Like, dotnet counters tool. [3 dev weeks] - - Trace collection, when something is on automatically turn it on. [2-3 dev weeks] - - Report improvements: - - Comparison between different versions that is not main [2 dev weeks] - - Automatic detection of perf issues. Don't have someone look at the report to have news about regression. [1 dev month] - - Run VS API tests? There are some problems that we would need for VS specific tests [4 dev weeks - optional feature] +# PerfStar +PerfStar is a performance tracking and investigation tool for MSBuild. PerfStar infrastructure captures performance measurements of main MSBuild branch on schedule and allows us to request experimental runs and collect performance of the new changes. The first version of this project is being finalized, with some fixes necessary to run it automatically and according to prerequisites. ## Goals and Motivation - - What are we trying to achieve and why? -Go fast +MSBuild currently does not have a lot of performance data outside of Visual Studio performance tracking, which has a lot of variables that are beyond the team's control. PerfStar enables us to measure our performance without interference of elements that the team does not own. As such, we can measure the performance of in-development features and how it will impact build times, as well as have concrete numbers when working on performance improvement tasks. ## Impact -Guidance: This document should not be an OKR. Let's keep the impact general here. -Questions to consider: - - What is the impact? - - Who will benefit from the feature? - - How does success look like? - - Are there any metrics or key results that could be quantified? - -We can be sure to go faster -Enable MSBuild team to track performance, identify low hanging fruits to improve performance, and watch new feature peformance. - -## Stakeholders -Questions to consider: - - Who are the stakeholders? - - For projects with concrete stakeholders, once the project is done how does a successful handover to the stakeholder look like? -Us. - +Perfstar's impact is focused on the team. We will be able to track performance with concrete numbers. Because of that the team will be able to take more informed decisions about taking performance improvement work, as well as future and implementation of new features. ## Risks -Questions to consider: - - Will the effort or feature cause breaking changes for existing consumers? - - Are there any unknowns that might derail the effort? - - Are there any dependencies we don’t have control over? - - Is there a hard deadline that needs to be met? - - What threatens completion of the feature? - - What is the impact of failing to deliver the feature? - -Dependencies: Crank, which is owned by ASP.NET team. This can cause problems with the machine set-up since they are responsible for this. - -Deadlines:No hard deadlines. Just nice to have for the team. -Threat: Randomization for the team. Security issues that come up. - -Impact of delivery failure. Less numbers in performance improvement for MSBuild features. - - -## Cost -Questions to consider: - - What is the estimated cost of the end-to-end effort? - - How accurate is the cost? +The risks associated with our dependencies is about Crank, which is owned by the ASP.NET team and we use it to help us with machine setup to run the performance tests. ## Plan - - High-level milestones, with estimate effort for tasks. \ No newline at end of file +Investiment for .NET 10: + 1. Making PerfStar execute automatically the way the design doc indicates + - Around 1 dev week. +2. The PowerBI reporting is working and updating the new information + - Around 2 dev weeks. +3. New performance tests for new features, and writting docs on how to write those tests. Next feature planned for tests: BuildCheck. + - Around 3 dev days per feature. +4. Analyze stability of performance tests, and fix any noise found. This will be done through multiple iterations of the same test in the same machine, as well as updating the PowerBI report to handle the new data. + - Around 2 dev weeks. +5. Add more tests using `msbuild.exe` for build instead of `dotnet build`. + - Around 1 dev week. +6. Timeboxed collection of feedback from our team, as well as performance investigations that can derive from those. + - 1 - 2 dev month depending on feedback and requests for improvement from the team. +7. Add more test cases. For example, build time with different verbosity levels. + - Around 1 dev week. + +There are more improvements form PerfStar, but these are not established for .NET 10 yet as they depend on the team's feedback to PerfStar. +1. Add more measurements, like dotnet counter tool. + - Around 3 dev weeks. +2. Trace collection when specific features are turned on for the test. + - Around 2 - 3 dev weeks. +3. Report improvements: + - Compare performance numbers between two different iterations that are not from `main` branch. Around 2 dev weeks. + - Automatic detection of performance issues, so we don't need to check the reports to see regressions. Around 1 dev month. + - Run MSBuild API tests, so we can check performance of calls relating to Visual Studio builds. Around 1 dev month. \ No newline at end of file From 2cc45bb3adfc7f998a7416542e16c1d0819501ee Mon Sep 17 00:00:00 2001 From: Chet Husk Date: Wed, 4 Dec 2024 08:35:17 -0600 Subject: [PATCH 5/8] Update documentation/specs/proposed/decoupling-vs-sdk.md Co-authored-by: Jared Parsons --- documentation/specs/proposed/decoupling-vs-sdk.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/proposed/decoupling-vs-sdk.md b/documentation/specs/proposed/decoupling-vs-sdk.md index 0e93de9ea6c..1ac2b7dc0d8 100644 --- a/documentation/specs/proposed/decoupling-vs-sdk.md +++ b/documentation/specs/proposed/decoupling-vs-sdk.md @@ -5,7 +5,7 @@ The behaviour of an SDK project, when built in Visual Studio and when built in t We are aiming for: - Consistent end-user eperience for build in either DotNet CLI or Visual Studio. - - Isolating Dotnet SDK build to only their components. + - Decoupling the .NET SDK experience from Visual Studio - Decoupling the Dotnet SDK from VS. From 929788ccfe69e26a506e0b21d17645a90f41725f Mon Sep 17 00:00:00 2001 From: Chet Husk Date: Wed, 4 Dec 2024 08:36:39 -0600 Subject: [PATCH 6/8] Update documentation/specs/proposed/evaluation-perf.md Co-authored-by: Jared Parsons --- documentation/specs/proposed/evaluation-perf.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/proposed/evaluation-perf.md b/documentation/specs/proposed/evaluation-perf.md index 66fd2561bad..af287dfeead 100644 --- a/documentation/specs/proposed/evaluation-perf.md +++ b/documentation/specs/proposed/evaluation-perf.md @@ -5,7 +5,7 @@ In the current effort to improve performance of MSBuild, we ideantified the eval Current performance state of evaluation is mostly unkown, as it is not measured in any ways by the team. As such, we are unsure which specific areas can be improve. The investigation about this is necessary so we can identify weaknesses, and possible fixes. - We could do profiling - - Jit compilation o MSBuild itself. + - Jit compilation of MSBuild itself. - We could cache at eval Constraint - needs to work as it does today, but fast. We can break some edge cases. From d2f54fa0b09291d0485b13ab4a063a23b714cefb Mon Sep 17 00:00:00 2001 From: Mariana Dematte Date: Wed, 11 Dec 2024 11:42:11 +0100 Subject: [PATCH 7/8] Some PR comment fiexes for decoupling paper --- .../specs/proposed/decoupling-vs-sdk.md | 23 ++++++++----------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/documentation/specs/proposed/decoupling-vs-sdk.md b/documentation/specs/proposed/decoupling-vs-sdk.md index 1ac2b7dc0d8..201235fc4be 100644 --- a/documentation/specs/proposed/decoupling-vs-sdk.md +++ b/documentation/specs/proposed/decoupling-vs-sdk.md @@ -1,42 +1,39 @@ -# Decoupling VS builds of SDK projects -The behaviour of an SDK project, when built in Visual Studio and when built in the DotNet CLI can vary, as different pieces of imports can be fetched from either build agent. This situation is not great specially for some developers as it increases their coding workload. To solve this, we want to ensure that all logic from a build from an *SDK project* comes from the SDK, independently of where the build is processed. +# Decoupling VS builds of .NET SDK projects +The experience of building a .NET SDK project can differ significantly depending if the project was built using Visla Studio / MSBuild or `dotnet build`. The build can produce different diagnostics, use different language rules, etc... and that is because building a .NET SDK project from Visual Studio mixes components from MSBuild and the .NET SDK. This means core tooling, like the compiler, can be substantially different between the two types of build. This leads to customer confusion and hard to diagnose problems, as well as increased code workload. To solve this want to ensure that when building a .NET SDK project we use the components from the .NET SDK to do so. ## Goals and Motivation We are aiming for: - Consistent end-user eperience for build in either DotNet CLI or Visual Studio. - Decoupling the .NET SDK experience from Visual Studio - - Decoupling the Dotnet SDK from VS. - + - Decoupling the .NET SDK from VS. The reason we are persuing this is for a better experience when using or writting Roslyn analyzers and MSBuild Tasks. Currently tooling authors need to target NetStandard2.0 for their projects to be recognized by VS. Another options is to multi-target but it takes a lot more effort and time spent on that. Another aspect is the user experience, if the Roslyn version for VS analyzers and generators doesn't match the one in Visual Studio, they do not work. - ## Impact There are a few area of impact: - - SDK style project builds will be more stable between VS and CLI builds, as the tooling will not be devided between different versions. + - .NET SDK style project builds will be more stable between VS and CLI builds, as the tooling will not be devided between different versions. - Reduced cost of development for external and internal teams that contribute to Roslyn Analyzers, SourceBuild generators, or MSBuild Tasks. - - End-user will not experience mismatch between analyzer versions, and confirmation that their SDK style builds will behave the same way in VS and in the command line. + - End-user will not experience mismatch between analyzer versions, and confirmation that their .NET SDK style builds will behave the same way in VS and in the command line. ## Stakeholders -For the internal stakeholder, we have the teams that will continue the work to fully complete the VS and SDK decoupling feature after our base work is done. There are two handovers in this project: +For the internal stakeholder, we have the teams that will continue the work to fully complete the VS and .NET SDK decoupling feature after our base work is done. There are two handovers in this project: -1. Enabling the MSBuild.exe execution state to be the same as DotNet command line invocation so the Roslyn team can enable the use of their Core compiler in VS. -2. Tasks and other projects can be written in .NET core and the SDK projects will build successuflly in VS. This enables other teams to migrate their tasks to .NET core instead of keeping them targeting .NET Framework. +1. Enabling the MSBuild.exe execution state to be the same as DotNet command line invocation so the Roslyn team can enable the use of their core compiler in VS. +2. Tasks and other projects can be written in .NET Core and the .NET SDK projects will build successuflly in VS. This enables other teams to migrate their tasks to .NET Core instead of keeping them targeting .NET Framework. The handovers should allow other teams to proceed with their work smoothly and no change in build behaviour should be present within MSBuild. ## Risks A few risks associated with this feature: - - There might be a performance hit on VS. It would depending on the amount of non-framwork tasks that the project will need to load when opening it in VS. The performance gain from pre-loading will not be available in this scenario. - - The VS tooling might present have a different version than the SDK installed, which might lead to discrepancy in partial builds. - Our work is early in the development effort. If this feature is discovered to have too large of an impact in experience of performance the work might be delayed or discarded. + - There might be a performance hit on VS once we start running tasks on .NET Core. It would depending on the amount of non-framwork tasks that the project will need to load when opening it in VS. The performance gain from pre-loading will not be available in this scenario. - There are no concrete deadlines for our part of the feature, but we are aiming for an early preview cycle, so we have a chance to measure the consequences and fix any issues that arise. ## Plan 1. Ensure that MSBuild.exe provides the same execution state as the dotnet command line invocation. - This is should take around 1 dev week to complete, and will be handed over to Roslyn team. - 2. Implement .NET core task host, tasks can be executed on the .NET Core vresion instead of .NET framework. + 2. Implement .NET Core task host, tasks can be executed on the .NET Core vresion instead of .NET framework. - This should take 1 to 2 dev months to complete, including extensive testing. This would be handed over to internal teams that have custom tasks so they can be updated and tested. 3. Load common targets from the .NET SDK and not from .NET NetFramework on VS. This work depends on other team's finilizing their part of the feature and might not be in scope for .NET 10. - This should take a dev week, if there are no major bugs or regressions. From 9e3e2c8affab4329777f06780b0c99ae84626755 Mon Sep 17 00:00:00 2001 From: Chet Husk Date: Wed, 8 Jan 2025 14:38:58 -0600 Subject: [PATCH 8/8] Rename parfStar.md to perfStar.md --- documentation/specs/proposed/{parfStar.md => perfStar.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename documentation/specs/proposed/{parfStar.md => perfStar.md} (100%) diff --git a/documentation/specs/proposed/parfStar.md b/documentation/specs/proposed/perfStar.md similarity index 100% rename from documentation/specs/proposed/parfStar.md rename to documentation/specs/proposed/perfStar.md