Skip to content

Build Performance Ideas

Jump to bottom
Jonathan Peppers edited this page Jan 24, 2019 · 92 revisions

Xamarin.Android build times are a key pain point for developers.

Parts of the build Xamarin.Android does not have control over:

  • aapt to process Android resources
  • javac to compile Java code to *.class files
  • dx (or soon d8) to convert compiled Java code to Android dex format

But there are still quite a few places we can improve, so we should do that!

Where are we today?

For an in-depth comparison between Visual Studio 2017 15.8.4 and what will ship in 15.9 or 16.0, see results here.

What can developers do today?

The SmartHotel360 app was originally using Xamarin.Forms 2.5.x and Xamarin.Build.Download 0.4.7.

We have MSBuild improvements in both of these packages. After the changes here, I saw drastic improvements to incremental build times:

Build Before Logs (Before) After Logs (After)
First build (fresh) 01:24.93 binlog 01:11.05 binlog
First package 00:28.81 binlog 00:10.47 binlog
First install 00:34.09 binlog 00:15.64 binlog
Second build (no changes) 00:22.20 binlog 00:03.41 binlog
Second package 00:28.70 binlog 00:03.42 binlog
Second install 00:34.27 binlog 00:03.42 binlog
Third build (change XAML) 00:34.45 binlog 00:11.05 binlog
Third package 00:33.12 binlog 00:07.91 binlog
Third install 00:40.62 binlog 00:08.29 binlog

Changes that made this possible:

Update your NuGet packages!

In-Depth Measurements

We are working on a CI setup. More info to come when that is available.

The idea being:

  • On dedicated hardware benchmark builds on a few apps.
  • We generate some nice-looking graphs in PowerBI.
  • We can have a benchmark of where things stand across different Visual Studio versions.
  • Eventually this could run for CI, PR builds, etc.

@pjcollins is working on this. @jonathanpeppers to assist with getting additional data from MSBuild when we get there.

Until then, we will continuing using our Jenkins Plots and do custom measurements locally.

Examples of "slow" MSBuild Tasks in Xamarin.Android

Some of the well-known targets that take up time are:

These are purely under our control, and we should improve them!

Improvements In Progress

More coming soon!

Improvements Released

15.9 P1/P2:

  • PR 1957: Design-time builds were causing full builds to "always build" and not building incrementally.
  • PR 1938: ResolveSdks caches the output of java -version and javac -version in memory, speeding up builds with multiple Xamarin.Android projects.

15.9 P3:

  • PR 2088: Fix incremental builds for Xamarin.Forms projects.
  • PR 2093: Improve LINQ usage in ConvertResourcesCases
  • PR 2130: Move inline C# MSBuild task to a compiled assembly
  • PR 2131: Remove unnecessary MSBuild target, simplify inputs to _CompileToDalvik
  • PR 2132: The _BuildLibraryImportsCache target was always running
  • PR 2140 Leave classes.zip uncompressed, to speed up javac and dx
  • PR 2105: Java.Interop is no longer a PCL.

16.0 P1:

  • PR 2128: _CopyIntermediateAssemblies improvements
  • PR 2129: Split up the work in ConvertResourcesCases, so some can be skipped
  • PR 2150: More cleanup in ConvertResourcesCases
  • PR 2148: Improve Mono.Cecil usage in BuildApk task
  • PR 2162: Consolidate StripEmbeddedLibraries task with the linker, helps release builds
  • PR 2174: Merge the CheckTargetFrameworks task into ResolveAssemblies
  • PR 2223: Optimize MSBuild $(AssemblySearchPaths)
  • PR 2309: Remove unused Inputs and Outputs from MSBuild targets

16.0 P2 (future):

  • PR 2019: D8/R8 integration
  • PR 2328: ConvertResourcesCases task, RegexOptions.Compiled and removed more LINQ usage
  • PR 2348: Whitelist support libraries, so ConvertResourcesCases will not run against them
  • PR 2367: Fix an issue on first build when enabling $(AndroidUseAapt2)
  • PR 2535: Remove all usage of temp files in GenerateJavaStubs
  • PR 2540: Use less temp files in ResolveLibraryProjectImports

Release Notes for Xamarin.Android 9.1

16.1 (future):

  • PR 2590: Linker improvements in Debug mode.
  • PR 2612: Use System.Reflection.Metadata in the <ResolveAssemblies/> MSBuild task
  • PR 2624: Use System.Reflection.Metadata in the <GetAdditionalResourcesFromAssemblies/> MSBuild task.
  • PR 2626: Install can skip Build inside of IDEs.
  • PR 2643: <GenerateJavaStubs/> should only load/evaluate TargetFrameworkIdentifier=MonoAndroid assemblies.

Ideas for Future Releases

New D8/R8 Android Dex Compilers

Java compilation steps in all Android apps do the following:

  1. javac compiles to *.class files
  2. *.class files go to a single *.jar (or in Xamarin.Android's case classes.zip)
  3. dx.jar converts compiled Java bytecode to Android "dex" format

Google has released a new tool to handle step No. 3, d8.

It should give us the following benefits:

  • d8 should have some minor performance improvements over dx
  • r8 (you could think of as an extension of d8) can perform code shrinking and dexing at the same time. This gives apps that currently use ProGuard a performance boost.
  • Google will likely drop support for dx at some point in the future.

We have merged support for d8/r8 in Xamarin.Android here.

Aapt2

We should investigate using a system-wide Aapt2 cache on NuGet packages. This would speed up <Aapt /> build times across projects.

See the Github issue for details.

There is a general perf issue, posted here.

ConvertResourcesCases [DONE]

This task runs twice during the build:

  • Once before <Aapt /> to fix casing
  • Once after <GenerateJavaStubs /> to replace custom view package names

  1. The work in this task doesn't need to happen in both places. So we should split this into two MSBuild tasks.

  2. The current task uses 2x sets of temp files. When refactoring we should avoid this being needed.

See the Github issue for more detail.

ConvertResourcesCases [DONE]

Implemented in (PR 2348)[https://github.com/xamarin/xamarin-android/pull/2348]

How about we make sure that Library projects and Bindings package the already converted assets in the .zip files. Then we can skip these in the main app as they were already converted.

We might need to figure out how to deal with Custom Views. But support libraries should not include those afaik

GenerateJavaStubs

  1. A list of assemblies is passed to JavaTypeScanner in Java.Interop. We should investigate if Parallel.ForEach would help here.

PR 365 here

It turns out it is not possible to implement this with Mono.Cecil not being thread-safe. See discussion here.

  1. We should investigate if we can pass in less assemblies. For example, we don't need to look at netstandard, PCLs, etc.

After investigation, I determined most all of these assemblies were TargetFrameworkIdentifier=MonoAndroid anyway. No. 2 did not appear to help any.

  1. We should optimize <GenerateJavaStubs/> so that it takes assemblies into consideration, so that it skips processing of assemblies which have not changed.

    For example, perhaps we should have a per-assembly output directory + .stamp file, and only update that directory if the assembly has changed.

    We should also skip processing of assemblies which are not MonoAndroid-profile assemblies, e.g. Xamarin.Forms .NETStandard assemblies.

Direct .dex generation

Various bits of our build system produce .java files which are then compiled with javac, then "re-compiled" into .dex files. (Such parts include <GenerateJavaStubs/> and <GeneratePackageManagerJava/>, among others.)

It should be possible to directly emit .dex files within some contexts. (Not easy, mind, but possible.) This would avoid the overhead of invoking javac and dx.

@(AndroidJavaSource) will still require the javac and dx invocations, and the @(AndroidJavaLibrary)/@(EmbeddedJar)/@(EmbeddedReferenceJar) build actions will still require dx invocations

Mono.Cecil usage

Many times throughout the build, we use DirectoryAssemblyResolver to traverse through all the resolved assemblies.

  1. We should do an experiment to be sure if we are using Mono.Cecil most efficiently. If we are just finding assembly references, is Mono.Cecil doing too much? Could a simpler approach find assembly references, etc.?

  2. Can we cache a readonly, InMemory instance of DirectoryAssemblyResolver with everything loaded up? So we don't go find all the assemblies over and over.

In theory, multithreaded Cecil usage is possible, but would require that we have a per-thread DirectoryAssemblyResolver instance, each instance of which loads assemblies with FileShare.Read, so that each thread can separately load assemblies. You can then imagine splitting up the User assemblies to process across N threads, each thread processing its set of assemblies and separately loading all dependencies. Dependencies would thus be "duplicated" across threads, but things should actually work safely.

StripEmbeddedLibraries

Related to Mono.Cecil usage, the StripEmbeddedLibraries task can take 1-2 seconds, and it mainly removes __AndroidLibraryProjects__.zip from assemblies.

  1. Can this work be moved to the linker? Github issue

PR 2162

  1. Should this work also be done for debug builds?

When building a default Xamarin.Forms project template, I noticed there are ~9MB of support library assemblies! I suspect a reasonable portion of the size could be AndroidResource files (we should look). Stripping these files will slow down the build, but speed up deployment and assembly loading at runtime. The tradeoff could be worth it.

Another thing to look into is Xamarin.Form's XamlC MSBuild task. Xamarin.Forms apps using shared projects are considerably slower running XamlC, because they have to load the ~9MB of support libraries. If we end up stripping the assemblies during debug builds, we should make sure XamlC can take advantage of this as well.

CreateMultiDexMainDexClassList

Currently, CreateMultiDexMainDexClassList invokes proguard.bat to generate this class. If a user is also using Proguard, it seems we are calling into proguard.bat twice. This might be unavoidable and could potentially be improved with R8 support.

4>     2760 ms  CreateMultiDexMainDexClassList             1 calls
4>     2866 ms  Proguard                                   1 calls

ResolveLibraryProjectImports and GetAdditionalResourcesFromAssemblies

The ResolveLibraryProjectImports MSBuild task runs before Compile, and its main job is to extract __AndroidLibraryProjects__.zip for all referenced assemblies.

GetAdditionalResourcesFromAssemblies runs after ResolveLibraryProjectImports, and mostly provides functionality for downloading files for older versions of the support libraries (24.x and older). Newer versions of the support library use Xamarin.Build.Download in its place.

Both of these MSBuild tasks uses Mono.Cecil to open every assembly.

Some thoughts to speed things up:

  • Could we run a task at the beginning of the build, in which its main job is to look at incoming assemblies and "classify" them with existing metadata?
  • Could these tasks be skipped entirely in some cases?

Classifications for each assembly such as:

  • Does the assembly need GetAdditionalResourcesFromAssemblies?
  • Does the assembly need ResolveLibraryProjectImports?
  • Does the assembly have native libraries? (To be used in BuildApks)

BuildApks

This MSBuild task's main job is to create the final APK file.

One of the parts that could use improvement, is that BuildApk uses Mono.Cecil to look at every referenced assembly and copy native libraries (so files) directory from EmbeddedResource to the APK file.

Some ideas:

  • Could this happen in an earlier task?
  • Could we extract these files to disk so this work can be done incrementally?

The goal here would be that we shouldn't need to look at every assembly each time an APK is built, but only during the first build.

General Ideas

  • Are there any MSBuild tasks that can run in the background? So other tasks can run in parallel while the work is done?
  • One to mention is GetPrimaryCpuAbi, which is in the proprietary source of Xamarin.Android.

Embrace Reference Assemblies

MSBuild and Roslyn support generating reference assemblies by setting the $(ProduceReferenceAssembly) MSBuild property to True.

Why do we care? What's this mean?

Assume you have a solution with two projects. Project Referenced.csproj has no further references. Project Referencer.csproj has a @(ProjectReference) to Referenced.csproj.

The developer makes a change to something within Referenced.csproj.

Question: Does Referencer.csproj need to be rebuilt?

In the "original" MSBuild world -- the world that Xamarin.Android still lives in -- the answer is yes, Referencer.csproj must always be rebuilt, because the change to Referenced.csproj may contain an API breaking change which would prevent Referencer.csproj from building.

In the new $(ProduceReferenceAssembly)=True world order, the answer is instead maybe: Referencer.csproj only needs to be built if the reference assembly produced as part of the Referenced.csproj build is updated, which in turn only happens when the public API changes. Meaning if a change doesn't alter the public API -- adding comments, fixing a method implementation, adding private/internal members, etc. -- then Referencer.csproj need not be rebuilt at all.

In more concrete terms, assume you have a Xamarin.Forms solution containing a Xamarin.Forms PCL project and a referencing Android App project. Currently, whenever the PCL project is changed, the App project must always be rebuilt. In a $(ProduceReferenceAssembly)=True order, the App project would need to be rebuilt less often.

So let's just export $(ProduceReferenceAssembly)=True! What's holding us back?

The problem is that our current build model is that a .csproj has only one output: the assembly. The assembly contains everything useful: native libraries (embedded .zip resource), Android Resources (embedded .zip resource), environment files, etc. This means that a reference assembly would be unusable: everything requires that the assembly be a full assembly, not some stubbed out reference assembly. Updates to Android Resources would thus be ignored, etc.

Thus, to support $(ProduceReferenceAssembly)=True, we need to change our build system's view of .csproj files. They can no longer "just" produce a .dll. Instead, they need to produce lots of things: native libraries, Android resources, etc.

Then we can update our build system to use reference assemblies for compilation, and go "around" the reference assemblies to pull in referenced assets, as needed.

Update for Reference Assemblies

@jonathanpeppers found out that the following scenario wasn't working as expected:

  1. Create a Xamarin.Forms project + NetStandard
  2. Add <ProduceReferenceAssembly>True</ProduceReferenceAssembly> to the NetStandard library
  3. Build
  4. Modify XAML, Build Again

It turns out that the reference assembly contains EmbeddedResource inside it! So modifying XAML causes the Xamarin.Android head to rebuild! This prevents the feature from helping us at all...

Issue is being fixed in the next Dev16 release: https://github.com/dotnet/roslyn/issues/31197

Whitelist the Support Libraries?

The Android support libraries that Xamarin provides on NuGet are generally in every app. They are also quite large if using all of them: totaling around 9MB of assemblies.

The thought here is if we could "skip" looking at these assemblies in various places during the build. Because we know for certain they may not need to be looked at.

Examples are:

  • ConvertResourcesCases
  • ConvertCustomView
  • LinkAssemblies in debug mode. Can we skip the FixAbstractMethodsStep step?

Running Tasks in Parallel

One of the idea we have had is to kick off some long running tasks in the background while the build is running.

Examples might be

  • Fast Deployment of items to device
  • Patching of Resources.
  • Parallelise Building of the Base apk and compilation of Java code.

The problem is MSBuild does not really support building Tasks or Targets in Parallel only projects. So we need a way to kick off a background task and wait for it to complete later in the build process. The idea is to use GetRegisteredTaskObject to register a global TaskManager which can be used to register background tasks with. We can then use the AsyncTask in conjunction with Task.WhenAll to wait later in the build for those tasks to complete.

public class TaskManager : IDisposable {

	SynchronizedCollection<TPL.Task> tasks = new SynchronizedCollection<TPL.Task> ();
        CancellationTokenSource tcs = new CancellationTokenSource ();

	public void RegisterTask (TPL.Task task)
	{
                lock (tasks.SyncRoot)
		    tasks.Add (task);
	}

	public TPL.Task [] Tasks {
		get {
			return tasks.ToArray ();
		}
	}
   
        public void Dispose ()
        {
            tcs.Cancel ();
        }

        public CancellationToken Token { get { return tcs.Token; } }

	public int Count => tasks.Count; 
}

We can then use code like this within a MSBuild Task to run a background task and instantly return back to MSBuild.

var manager = (TaskManager)BuildEngine4.GetRegisteredTaskObject ("TaskManager", RegisteredTaskObjectLifetime.Build);
if (manager == null) {
	manager = new TaskManager ();
	BuildEngine4.RegisterTaskObject ("TaskManager", manager, RegisteredTaskObjectLifetime.Build, allowEarlyCollection: false);
}
var task =  TPL.Task.Run (async () => {
	await TPL.Task.Delay (10000);	
});
manager.RegisterTask (task);

The in the Execute method of an AsyncTask derived task we can do something like

manager = (TaskManager)BuildEngine4.GetRegisteredTaskObject ("TaskManager", RegisteredTaskObjectLifetime.Build);
TPL.Task task;
if (manager == null || manager.Count == 0) {
	task = TPL.Task.CompletedTask;
} else {
       task = TPL.Task.WhenAll (manager.Tasks);
}
task.ContinueWith(Complete);
base.Execute ();

This can either be in a specific task, say WhenAll or bolted into an existing task like InstallPackagedAssemblies.

One this we need to figure out is how to deal with Logging. Since its a background task we cannot access the normal MSBuild Log.LogXXXX methods. So we will need some way to collect all the logging from the task and then emit the messages, warnings and errors when the task completes.

One of the other problems we thought of was "What if the user cancels the build when a Task is not running". Or if the build is NOT in the WaitAll task.. The solution there is to implement IDisposable on the TaskManager. Because we are registering it as part of RegisteredTaskObjectLifetime.Build, it should be disposed of if the user cancels or when the build completes. If the build completes then we should have got through the Wait Task already so all the registered tasks will be complete. On Cancellation, the CancellationTokenSource on the TaskManager should be Canceled.

Allow users to fast dev custom debug files

Currently the fast dev .__override__ directory can only be used by our build system. Should we look at expanding it to allow users to fast deploy custom files to the fast deployment directory. Files such as test sqlite databases, game textures and models or Json files. This could be handled via the @(Content) item group or via a new @(FastDevUserFiles) item group.

We could then perhaps provide support for loading such files by overriding the AppContext.BaseDirectory to point at the .__override__ directory so users can use a normal File.Open.

Silly Ideas:

Should we move the linker to a separate process?

Can we support a "daemon" mode?

On Windows, dotnet build, etc. long-lived processes are used to improve build performance. MSBuild.exe and VBCSCompiler.exe (Roslyn) stay running, so their startup time is avoided during incremental builds. Can we use this idea in Xamarin.Android?

aapt2

aapt2 has a new "daemon mode" feature in the latest 2.19 release.

In order to use this for aapt2 we would need to:

  • Download the latest bits for aapt2 from Maven, and ship them (their license should be OK). The Android SDK ships outdated versions of aapt2, and customers can use different versions of the Android SDK anyway...
  • Add daemon-mode support for <Aapt2Compile/> and <Aapt2Link/> MSBuild tasks

Java

Can we keep a Java daemon running all the time?

We call java.exe to run various jar files:

  • apksigner.jar
  • dx.jar (or r8.jar)
  • proguard.jar

Can we use a daemon to reduce JVM startup time? gradle does this. Can we actually use gradle (or something else) to make this work?

General

Contributing

Tips & tricks

Specifications

Release Notes

Clone this wiki locally