M-mode interrupt virtualization

[CharlyCst]

Following our discussion with @Wesdcv I am opening this issue to track the progress on M-mode interrupt virtualization.

Context

Non-delegated interrupts need to be fully virtualized. A non-delegated interrupt will necessarily trap to Miralis because Miralis is the M-mode software. If the corresponding interrupt bit is not set to 0 returning to the firmware or the payload will trap back to Miralis right away, effectively causing an infinite loop on the hart. Disabling the interrupt in mie is not an option, because there would be no trap for the next interrupt which would prevent Miralis from injecting the interrupt back into the firmware.

Delegated interrupts do not have this issue, because the OS can handle them without trapping to Miralis. According to the specification all interrupt delegated bits in mideleg can be hardwired to 1 except the M-mode interrupt bits. The M-mode interrupts are MEI, MTI, and MSI.
Virtualizing interrupts requires virtualizing the device emitting the interrupt (e.g. the PLIC and CLINT for M-mode interrupts), but we want to limit the virtualization of devices as much as possible. As a result we should force delegation of all non M-mode interrupts.

Life cycle of virtualized interrupts

Consider an interrupt MXI that is virtualized.

• The interrupt is not delegated: mideleg.MXI is set to 0.
• The interrupt is enabled when it is enabled by the firmware mie.MXI == vmie.MXI.
• When Miralis gets the interrupt (it traps with mip.MXI == 1) Miralis sets vmip.MXI to 1 and deactivates the interrupt. This requires to write to a memory mapped device (e.g. CLINT or PLIC). Note that this device must be virtualized, and thus is not directly accessible by the firmware or the payload. If the interrupt is enabled for the firmware it must be injected, otherwise it must be injected whenever the interrupts become enabled again.
• When the firmware clears the MXI interrupt pending bit by writing to the corresponding (virtualized) memory mapped device Miralis intercepts the write and sets vmip.MXI to 0.
• For interrupts that can be received multiple times, such as MIE, Miralis disables the corresponding mie.MXI bit if vmip.MXI is already 1. This prevent Miralis from trapping again if another interrupts arrives but the previous one has not yet be handled by the firmware. Miralis restores mie.MXI to 1 once the firmware disables vmip.MXI by writing to the virtualized device.
• Before each re-entry into the firmware Miralis should check if an interrupt should be injected (vmip.MXI == 1, vmie.MXI == 1, and vmstatus.MIE == 1), and inject one if that is the case.

M-mode interrupt virtualization TODO list:

• Hardwire all the non M-mode interrupt delegate bits to 1 in vmideleg.
• Virtualize MEI
• Virtualize MTI
• Virtualize MSI
• Update the documentation (explain the rationale for virtualizing interrupts and how it works).

[CharlyCst]

@Wesdcv does that make sense to you? I tried to summarize my understanding of our discussion, but let me know if you think I forgot anything/something seems off.

[Wesdcv]

Yes, great analysis, thanks!

I think there is one thing that worries me somewhat:

Assuming the firmware runs unvirtualized. It gets a trap - goes into trap handler, disables further interrupts, deals with trap cause (writes to mm register) - that's a "good" scenario. The firmware is theoretically under no obligation to disable interrupts (which Miralis reflects in physical registers) in the trap handler, or to clear the interrupt bit - so if the bit wasn't cleared, it should trap again.
In our current implementation of MTI virtualization, this doesn't happen - we clear the pMTIP bit on behalf of the firmware, inject the interrupt once, and then just assume that the firmware also clears the vMTIP bit. But the firmware won't trap again even if vMTIP=1, so the virtualization path here differs from the non-virtualized setup (yes, I agree this is an extremely unlikely scenario). Maybe in case pMXI is 0 and vMXI is 1, we want to artificially generate an interrupt (either by writing to mm register or setting trap_info directly)?

Upd:
Harts are alowed to write each others MSIP, but i'm not sure yet if that's something that should be accounted for in the virtualization (need more time to think about it).
Could you also please point me to where Miralis disables the interrupts during it's own execution? I remember seeing the comment somewhere, but cannot find it.

[CharlyCst]

Yes you are right, before returning to the firmware we might need to check if an interrupt needs to be injected (interrupts are enabled + vmip.MXI == 1. I updated the top comment with this additional step.

And regarding Miralis, it simply never enables them, so they stay disabled forever^^