-
Notifications
You must be signed in to change notification settings - Fork 652
WEBSITE_RUN_FROM_PACKAGE and WEBSITE_CONTENTAZUREFILECONNECTIONSTRING Best Practices
Often times, users reported the deployment either DevOps or ARM Template/EV2 with RunFromPackage failed intermittently or why my app didn't find the expected deployed content after a successful deployment or my function returned NotFound.
Below should help isolate one common problem.
App Service provides a file system (back by storage) out of the box for user's web apps. It is a network shared file system allowing multiple instances (in case more than one instances in an App Service Plan) to access the same content.
There are 2 AppSettings to customize the File System behavior.
Instead of typical file layout, all artifacts are kept in one zip file. This zip file is virtualized as a typical file system to the app. This setting impacts both deployment (ZipDeploy) behavior to keep the deployed zip file as is (no extracting) and runtime (expecting and virtualizing the deployed zip file). The advantage is a faster, more reliable and more atomic deployment (no partial files being changed, no file locked or in-use issue during deployment while the app is running).
This switches from App Service file system to user's provided Azure Files - common scenario for Azure Functions. The advantage is there is no downtime during App Service infrastructure upgrade. One can also achieve better performance with higher SKU Azure Files.
It is strongly recommended that this setting to be set at app creation time and immutable thru out the lifetime of an app. In fact, there is no scenario that one would need to change this setting occasionally or especially during deployment. Modifying this will cause the deployment and app runtime to behavior incorrectly, guaranteed downtime and data loss perception (deployed files are not where they are expected) - imagine swapping a hard drive while installing a software to a computer.
Most of the times, the users don't switch this intentionally but rather by mistake due to the nature of AppSettings api. The api requires the payload to contains all necessary settings which will replace the existing one entirely. There is no partial remove or patch support with existing AppSettings. For instance, if one want to add a new setting, one needs to get existing set, add to that collection and update as a whole.
When setup DevOps pipeline with RunFromPackage option, for every run, DevOps will automatically add WEBSITE_RUN_FROM_PACKAGE=1 to the current AppSettings collection. As part of the pipeline, user often provides script/template to create/update the site and its AppSettings. If user misses to include this as well, it creates a situation as if the setting is removed and re-added back for every run. This leads to file system being changed during deployment and a common source of transient issue if not missing deployed content altogether.
Mitigation: Make sure the template or script include WEBSITE_RUN_FROM_PACKAGE=1 if DevOps's RunFromPackage is selected and vice versa.
The most common issue is an incorrect ARM resource dependency - the deployment (such as ZipDeploy) should happen when all the config especially these settings have been set (dependsOn). If deployment happens before or at the same time (lacking dependency), content may be deployed one way (before AppSettings update) while the runtime expects the other (after AppSettings update) given the perception as if deployment never occurs.
Mitigation: Ensure ZipDeploy resource depend on Site, Sites/Config and Sites/AppSettings resources especially if either settings exists.
Less common issue is due to an inconsistent AppSettings defined in ARM template. There are two ways to provide AppSettings - in Site and SiteConfig resources. If AppSettings in one place contains WEBSITE_RUN_FROM_PACKAGE or WEBSITE_CONTENTAZUREFILECONNECTIONSTRING but not the others. When ARM template run, it creates a situation as if the setting is removed and re-added back (or vice versa). Again this is undesirable and may lead to issues.
In a rare occasion only happening at site creation on very first template run, even if the user always define AppSettings in SiteConfig resource only, since no AppSettings is provided in Site resource, the site is created with default file system. Then SiteConfig adds those AppSettings and create a situation of the file system switching. This can also lead to one-time issue for the first template run which is usually negligible and success afterward.
Mitigation: Include AppSettings as part of Site resource - see this simple example especially if these setting exist.