Proposal: Add support for loading application assemblies in memory

[davidfowl]

The premise of this issue is driven by this dotnet/aspnetcore#3793. TL;DR when you run your ASP.NET Core applications in IIS, the files are locked. This means to deploy a new application, the site needs to be stopped first, then files need to be updated in place (which results in downtime) then the site can be started again.

IIS supports dropping a file called appoffline.htm, the IIS module watches for this file and kills the process and serves this file as content during deployment. This stops the native IIS module from loading managed code while deploying new files. After the deployment, the appoffline.htm file should be deleted and the site will load the new binaries. The only tool that automates this deployment is webdeploy, and when doing ftp or normal xcopy deployment this process can be cumbersome. Things that make it a bit tricky are handling things like dropping appoffline.htm then sleeping until IIS actually picked up these changes so the tool can know when to start the deployment operation.

To mitigate some of this, we want to add a new native hosting API that allows loading application local assemblies in framework dependent deployments without locking the file. This allows the deployment process to look more like this:

1. The deployment tool can copy files over the existing deployment (adding and removing files)
2. The deployment tool will then create a file that signals to the IIS module that the deployment is complete so it should restart.
3. The IIS module would remove the file after successfully restarting.

The constraints:

• This only works for Framework Dependent Deployments
• This only works for non-R2R images

In addition to the hosting API, we should consider adding a new API on AssemblyLoadContext that has the same semantics:

public class AssemblyLoadContext
{
    public Assembly LoadFromAssemblyPath(string path, bool eagerLoad);
}

I don't like the bool but it signifies that the file will be eagerly loaded into memory instead of being memory mapped. This allows the runtime to preserve all of the file based properties of the the RuntimeAssembly implementation but without the file being mapped/locked on windows

cc @vitek-karas @richlander @jkotas

[jkotas]

This stops the native IIS module from loading managed code while deploying new files.

Is this alternative scheme going to be reliable enough without stopping the process? Every once in a while, patching of the app will collide with the app loading files that will produce assorted crashes. It sounds like a factory for Watson dumps to me. There is also non-zero chance that it may not crash and do some actual damage (security risk).

a new native hosting API that allows loading application local assemblies in framework dependent deployments

My proposal would be to add a new Windows-specific key/value pair: NO_FILE_LOCK_PATH. When the assembly loader sees attempt to load a file under this directory path, it will creates a local in-memory copy of the file before loading it.

Note that other types of files such as unmanaged .dlls under this path are still going to be locked. The runtime has no control over loading them.

adding a new API on AssemblyLoadContext

I do not think it deserves an public API. It is very Windows & IIS specific quirk.

[GrabYourPitchforks]

Can somebody weigh the pros and cons of this specific feature request against something akin to the shadow copy feature offered by Full Framework?

[davidfowl]

@jkotas

Is this alternative scheme going to be reliable enough without stopping the process? Every once in a while, patching of the app will collide with the app loading files that will produce assorted crashes. It sounds like a factory for Watson dumps to me. There is also non-zero chance that it may not crash and do some actual damage (security risk).

That’s a reasonable concern. I’ll enumerate some of the differences with shadow copying that @GrabYourPitchforks asked for below but the biggest risk would be the fact that you could be in a torn state for assemblies that haven’t loaded yet. The app could see both old and new until the restart file was dropped.

Some System.Web applications might have been able to mitigate this because I believe we basically scanned all assemblies at app start which I believe resulted in loading all of them.

I can think of ways to mitigate this that matches the pattern we have before but with less downtime.

My proposal would be to add a new Windows-specific key/value pair: NO_FILE_LOCK_PATH. When the assembly loader sees attempt to load a file under this directory path, it will creates a local in-memory copy of the file before loading it.

Sounds like a good strawman. I’ll clarify some of the constraints (non native etc). Can you specify multiple directories or just one?

@GrabYourPitchforks I’ll give it a try

• Shadow copying requires writable storage (can be temporary)
• the storage where files are copied needs to be per application “version” (can’t stomp on files from previous recycle)
• Native binaries were never supported
• There’s no working set change because the shadow copied files are still memory mapped
• Shadow copying was lazy. Files were copied before they were loaded (need a fact check on this). This means it would still be possible to get into a torn state where files that aren’t yet loaded weren’t shadow copied

[onchuner]

Would it be possible to have two copies of app? Willie first one is running the second can be redeployed Upon completion IIS would switch to the second one From: David Fowler Sent: Saturday, 8 February 2020 23:10 To: dotnet/runtime Cc: Subscribed Subject: Re: [dotnet/runtime] Proposal: Add support for loading applicationassemblies in memory (#31958) @jkotas Is this alternative scheme going to be reliable enough without stopping the process? Every once in a while, patching of the app will collide with the app loading files that will produce assorted crashes. It sounds like a factory for Watson dumps to me. There is also non-zero chance that it may not crash and do some actual damage (security risk). That’s a reasonable concern. I’ll enumerate some of the differences with shadow copying that @GrabYourPitchforks asked for below but the biggest risk would be the fact that you could be in a torn state for assemblies that haven’t loaded yet. The app could see both old and new until the restart file was dropped. Some System.Web applications might have been able to mitigate this because I believe we basically scanned all assemblies at app start which I believe resulted in loading all of them. I can think of ways to mitigate this that matches the pattern we have before but with less downtime. My proposal would be to add a new Windows-specific key/value pair: NO_FILE_LOCK_PATH. When the assembly loader sees attempt to load a file under this directory path, it will creates a local in-memory copy of the file before loading it. Sounds like a good strawman. I’ll clarify some of the constraints (non native etc). Can you specify multiple directories or just one? @GrabYourPitchforks I’ll give it a try • Shadow copying requires writable storage (can be temporary) • the storage where files are copied needs to be per application “version” (can’t stomp on files from previous recycle) • Native binaries were never supported • There’s no working set change because the shadow copied files are still memory mapped • Shadow copying was lazy. Files were copied before they were loaded (need a fact check on this). This means it would still be possible to get into a torn state where files that aren’t yet loaded weren’t shadow copied — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[jkotas]

Would it be possible to have two copies of app?

Yes, it would be a better way to update an app on a live server.

It is dilemma that we often face: Do we make customers happy by providing a convenient solution that has reliability or security issues; or do we make customer successful by providing less convenient solution that is reliable and secure by construction?

[davidfowl]

Part of the issue is that we’re coming from a model that “just worked” albeit with this caveats. People have been doing this for a long time, more mature processes can be layered on top (load balancer to rotate out instances that are being deployed to).

When running on IIS, there’s a single physical target configure in IIS. That can’t and won’t easily be changed to have a second path so the alternative is to find ways to do in place deployments.

I think it’s also important to remember that certain types of applications don’t need reliable deployments and that more mature processes can be layered on top. We’re just trying to remove some of the hurdles here

[jkotas]

How do people tell whether they do not need reliable deployment? Could you please propose the paragraph that we will include in the documentation once this is implemented? It should explain the risks and provide recommendation for when it is acceptable to use it.

[davidfowl]

How do people tell whether they do not need reliable deployment? Could you please propose the paragraph that we will include in the documentation once this is implemented? It should explain the risks and provide recommendation for when it is acceptable to use it.

Not sure what you’re looking for. When we implemented shadow copying in .NET framework, I imagine the same problems existed? Unless I misunderstood how it works.

The default set of tools we have still use the pattern where the file is dropped before and removed after the files are copied. We can absolutely document what a recommended deployment pattern and explain both the caveats (as mentioned above) and the risks like the fact that you can end up in a torn state. Just as you could with .NET Framwork.

Feels like we’re off track here. Can we get back to designing the feature? Is it just that folder path that would need to be added? I’m also thinking of a flag in the IIS module where the appoffline file isn’t served but acts like a signal to queue requests until it’s deleted. This would make requests take longer but not necessarily result in downtime.

cc @jkotalik

[jkotas]

Not sure what you’re looking for.

Paragraph we copy and paste into documentation once this is done. I want to make sure that we still like this feature once we explain the caveats in customer facing way.

shadow copying in .NET framework, I imagine the same problems existed?

Probably. The security and reliability standards that we are building for today are different than when shadow copying was built for .NET Framework 10+ years ago.

[davidfowl]

Paragraph we copy and paste into documentation once this is done. I want to make sure that we still like this feature once we explain the caveats in customer facing way.

I'll take a stab at listing out the caveats with the approach. The docs team that can use that as input to a final document. Of course, we as the design evolves, well need to keep this list up to date 😄 .

Here's the current documentation on locked files:

• https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/?view=aspnetcore-3.1#locked-deployment-files
• https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-3.1#app_offlinehtm

Assuming we do this feature, we would describe that files are no longer locked during deployment but:

Torn Deployments

In .NET 5, changes were made to reduce file locking while the application is running. The following files will still be locked:

• Ready to run assemblies
• Native libraries
• Pdbs? This one we'll need to discuss...

A torn deployment is a mixture of old and new files and can be a results of doing uncoordinated in-place updates. If old files are replaced with new files while the application is running, there is a window where the running application loads a mix of new and old binaries. This can result in an undefined application behavior (assembly load failures, security issues etc).

Best practice is to use app_offline.htm at the start of the deployment, copy all files then remove the app_offline.htm. This turns the deployment into an atomic operation and avoids the torn state.

Using app_offline.htm results in downtime, we may be able to improve that with the request queuing mentioned earlier.

Probably. The security and reliability standards that we are building for today are different than when shadow copying was built for .NET Framework 10+ years.

The standards aren't different but some of the tech is. Keep in mind we're discussing deploying to a stack that hasn't change and is built into Windows. Here's an example of what people do today dotnet/aspnetcore#3793 (comment).

Deployment recommendations also vary depending on what the target platform supports. As an example, here is our Host and Deploy table of contents on the docs:

I don't expect a one size fits all solution.

[jkotas]

Thanks! This will help us to get to the details.

In .NET 5, changes were made to reduce file locking while the application is running

This change will increase startup time and private working set. Do you expect this this be on by default? I think it should be opt-in under a config switch. I do not think it makes sense for everybody for pay for this.

Ready to run assemblies

My assumption is that we won't even try to load the files under the NO_FILE_LOCK_PATH as R2R (ie R2R payload in assemblies under NO_FILE_LOCK_PATH will be ignored).

Native images

I assume that you meant native libraries, such as native libraries in RID specific deployments. For example, applications using System.Data.SqlClient NuGet package are going to have sni.dll locked. (sni.dll is native library that is part System.Data.SqlClient NuGet package.)

Also, any other files that the app opens exclusively will be locked.

Keep in mind we're discussing deploying to a stack that hasn't change and is built into Windows.

I understand that this is deploying to a stack that was set in stone long time ago. My observation is that we tend to burn a lot of energy trying to invent tricks to make these legacy + modern combinations work great, but the results are still mediocre at best.

[rynowak]

@davidfowl - do we need to do something to address pdbs for this case? I've seen cases where users report this issue for pbds that have been locked as well. I suppose you have the option not to deploy those too...

[davidfowl]

Yea the pdb thing give me pause on the approach. I fixed an issue in 3.0 wrt pdbs not being renamable, which hopefully mitigated then web deploy issues.

Those can be loaded in memory as well and we may have to apply that here.

[616E64726173]

This is also applicable to hot-fixes in console/desktop applications where you would like to have the ability to unload and replace referenced assemblies with new versions pulled down from a server.

[dotnet-policy-service]

Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.

This process is part of our issue cleanup automation.