Developer inner loop testing where signed packages and consistent package identity are required

[dkbennett]

From #444

Can we avoid committing in the .msix files? Git doesn't like binaries.

The basic testing strategy of the installer and as a checkin test is:

1. Build test packages
2. Build the installer with test packages
3. Test installer successfully deployed those test packages.

This issue stems from needing to include packages in the installer in order to test it, and to then test against those packages. Only signed packages can be installed successfully to verify, and the signature is tied to the publisher, and the publisher hash is part of the package identity, meaning using a different publisher (for whoever clones the repository and builds packages) would need to change the tests to match the new publisher hash.

The original simple solution to this was to sign with a well known and consistent publisher identity, the MSTest cert, which was checked into /build. This solution worked great, but has its own problems (see test cert discussion in The problem now is that the MSTest cert is being moved out and replaced with Azure KeyVault (see #478 ), which will work for people that can run the pipeline task and have access to the test cert, but it fails for inner loop testing and F5'ing the solution.

I could update this to get the cert from Azure every time and build the packages with every installer build, but that would add significantly to build time and break F5 of the installer. The current solution should work for everyone to easily do inner loop testing. In the case of the installer, these packages are empty and do not require any compiled code every time it is built, so the packages are checked in as test collateral, but that is not desired per the referenced comment in #444.

What is the right answer here for inner loop testing and accessibility to the broader developer community?

[andrewleader]

@BenJKuhn are you on point to discuss and answer this?

[DrusTheAxe]

DevCheck.cmd is your friend

DynDep's PR just merged and (among other things) includes tools\DevCheck.cmd (and tools\DevCheck.ps1).

Run tools\DevCheck.cmd when you setup your dev environment (or now) and it will (among other things) create temp\MSTest.pfx and register build\MSTest.cer in your certificate store.

Build your MSIX packages via test projects and sign them with temp\MSTest.pfx.
As the matching .cer will be known to your certificate store you can install your signed test packages.
Low friction inner-loop testing without checked in binaries

DevCheck has some internal enhancements on the radar (e.g. play nicer with Azure Key Vault) but that's all inside baseball. It won't change the developer experience, where 'developer' is you and I, github's CI/Pipeline, Helix, etc.