failing boot retry count requires hard flash

[tbottomparallel]

Hello,

I've been updating our xavier platform to use the 35.1 using an AGX Xavier devkit and the new uefi bootloader. We are running a yocto meta-tegra kirkstone image, which by default does not come with a recovery partition. The head of kirkstone on meta-tegra right now also does not run nvbootctrl verify on boot and I ran into the following issue:

Once the retry count for slot A reached 0 the platform will no longer boot. Since there is no recovery partition it drops to the UEFI Shell. There does not appear to be any way from UEFI shell to reset the retry count, nor to force the platform to boot from slot A anyway, nor to place the device in forced-recovery mode to allow flashing over USB.

The only way I am aware of to get out of this state is to have someone physically present place the device into forced recovery mode, after which USB-C flashing works fine. This is very inconvenient for remote developers though.

Things I have tried:

Manually setting the retry count on the RootfsInfo variable

Shell> setvar RootfsInfo -guid 781E084C-A330-417C-B678-38E696380CB9 -bs -rt -nv
781E084C-A330-417C-B678-38E696380CB9 - RootfsInfo - 0004 Bytes
0C C3 00 00

Shell> setvar RootfsInfo -guid 781E084C-A330-417C-B678-38E696380CB9 -bs -rt -nv
=0x0030C30C

Shell> setvar RootfsInfo -guid 781E084C-A330-417C-B678-38E696380CB9 -bs -rt -nv
781E084C-A330-417C-B678-38E696380CB9 - RootfsInfo - 0004 Bytes
0C C3 30 00

Manually writing to the Tegra Rootfs Status Register Address T194

Shell> mm 0x0C39041C -w 4 -n
MEM  0x000000000C39041C : 0x000CFACE
Shell> mm 0x0C39041C -w 4 -n =0x003CFACE
MEM  0x000000000C39041C : 0x003CFACE
Shell> mm 0x0C39041C -w 4 -n
MEM  0x000000000C39041C : 0x003CFACE

neither reset -c nor reset -w following either of the above changes had any effect.

Attempting to boot always results in

Jetson UEFI firmware (version 1.0-d7fb19b built on 2022-08-10T20:18:13-07:00)
Press ESCAPE for boot options **  WARNING: Test Key is used.  **
......
      L4TLauncher: Attempting Recovery Boot
Header not seen at either offset 0 or offset 0x1000
Failed to boot recovery:0 partition
...
Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell> 

Based on my understanding of nvidia's documentation, these are what the reboot logic should be basing the decision to not boot to slot A on.

trying to find a way to enter forced-recovery RCM mode

based on how "forced-recovery" reboots are handled by drivers/soc/tegra/pmc.c I tried to read 0xc362000 with the intention of setting the PCM_SCRATCH0_MODE_RCM bit, but this resulted in an exception and reset of the bootloader:

Shell> mm 0xC362000 -w 4 -n
MEM

Synchronous Exception at 0x0000000857C88874


Synchronous Exception at 0x0000000857C88874
ASSERT [ArmCpuDxe] /dvs/git/dirty/git-master_linux/out/nvidia/bootloader/uefi/Jetson_RELEASE/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333): ((BOOLEAN)(0==1))

Resetting the system in 5 seconds.