Skip to content

Commit

Permalink
move xctrctl.DEPTH to new sctrdepth CSR
Browse files Browse the repository at this point in the history
  • Loading branch information
@bcstrongx
bcstrongx committed Feb 27, 2024
1 parent a3c4233 commit 77bbe80
Showing 1 changed file with 60 additions and 43 deletions.
103 changes: 60 additions & 43 deletions body.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@ The `mctrctl` register is a 64-bit read/write register that enables and configur
{bits: 1, name: '<i>WPRI</i>'},
{bits: 1, name: 'BPFRZ'},
{bits: 1, name: 'LCOFIFRZ'},
{bits: 3, name: 'DEPTH'},
{bits: 17, name: '<i>WPRI</i>'},
{bits: 20, name: '<i>WPRI</i>'},
{bits: 1, name: 'EXCINH'},
{bits: 1, name: 'INTRINH'},
{bits: 1, name: 'TRETINH'},
Expand Down Expand Up @@ -64,27 +63,6 @@ The `mctrctl` register is a 64-bit read/write register that enables and configur

|LCOFIFRZ |Set `sctrstatus`.FROZEN on local counter overflow interrupt (LCOFI) that traps to M-mode or S-mode. See <<Freeze>>.

|DEPTH[2:0] a|
WARL field that selects the depth of the CTR buffer. Encodings:

‘000 - 16

‘001 - 32

‘010 - 64

‘011 - 128

‘100 - 256

'11x - reserved

The depth of the CTR buffer dictates the number of entries to which the
hardware records transfers. For a depth of N, the hardware
records transfers to entries 0..N-1. All <<_entry_registers, Entry Registers>> read as '0' and are read-only when the selected entry is in the range N to 255. When the depth is increased, the newly accessible entries contain unspecified but legal values.

It is implementation-specific which DEPTH value(s) are supported.

|EXCINH |Inhibit recording of exceptions. See <<Transfer Type Filtering>>.

|INTRINH |Inhibit recording of interrupts. See <<Transfer Type Filtering>>.
Expand Down Expand Up @@ -115,15 +93,9 @@ It is implementation-specific which DEPTH value(s) are supported.

[%unbreakable]
--
All fields are optional except for M, BPFRZ, and DEPTH. All unimplemented fields are read-only 0, while all implemented fields are writable. S must be writable if S-mode is implemented, and U must be writable if U-mode is implemented. If the Smcofpmf/Sscofpmf extension is implemented, LCOFIFRZ must be writable.
All fields are optional except for M, S, and BPFRZ. All unimplemented fields are read-only 0, while all implemented fields are writable. U must be writable if U-mode is implemented. If the Smcofpmf/Sscofpmf extension is implemented, LCOFIFRZ must be writable.
--

[NOTE]
[%unbreakable]
====
_To avoid the use of CTR for side and/or covert channels, context switching routines should first update the CTR depth value to match that of the context being switched to, and then either clear the CTR entries or restore them with the values they previously held for that context. If the context being switched to can change the depth value then all CTR entries should be cleared unless all entries for the maximal depth value are being restored. CTR depth can be modified only by M-mode and, depending on the <<_state_enable_access_control, state enable>> configuration, S/HS-mode._
====

=== Supervisor Control Transfer Records Control Register (`sctrctl`)

The `sctrctl` register provides supervisor mode access to a subset of <<_machine_control_transfer_records_control_mctrctl, `mctrctl`>>.
Expand All @@ -147,8 +119,7 @@ If the H extension is implemented, the `vsctrctl` register is a 64-bit read/writ
{bits: 2, name: '<i>WPRI</i>'},
{bits: 1, name: 'BPFRZ'},
{bits: 1, name: 'LCOFIFRZ'},
{bits: 3, name: 'DEPTH'},
{bits: 17, name: '<i>WPRI</i>'},
{bits: 20, name: '<i>WPRI</i>'},
{bits: 1, name: 'EXCINH'},
{bits: 1, name: 'INTRINH'},
{bits: 1, name: 'TRETINH'},
Expand Down Expand Up @@ -178,20 +149,64 @@ If the H extension is implemented, the `vsctrctl` register is a 64-bit read/writ
|STE |Enables recording of traps to VS-mode when S=0. See <<External Traps>>.
|BPFRZ |Set `sctrstatus`.FROZEN on a breakpoint exception that traps to VS-mode. See <<Freeze>>.
|LCOFIFRZ |Set `sctrstatus`.FROZEN on local counter overflow interrupt (LCOFI) that traps to VS-mode. See <<Freeze>>.
|DEPTH[2:0] |Provides read-only access to the `sctrctl`.DEPTH field
2+|Other field definitions match those of <<_supervisor_control_transfer_records_control_sctrctl, `sctrctl`>>. The optional fields implemented in `vsctrctl` should match those implemented in `sctrctl`.
|===

[NOTE]
[%unbreakable]
====
`__vsctrctl__`.__DEPTH__ _is a read-only copy of `sctrctl`.DEPTH in order to allow a hypervisor to dictate the depth used by a guest. This simplifies VM (guest) migration, by providing the hypervisor a means to require the guest to use a depth supported across all systems in the datacenter._
_Unlike the CTR status register or the CTR entry registers, the CTR control register has a VS-mode version. This allows a guest to manage the CTR configuration directly, without requiring traps to HS-mode, while ensuring that the guest configuration (most notably the privilege mode enable bits) do not impact CTR behavior when V=0._
====

=== Supervisor Control Transfer Records Depth Register (`sctrdepth`)

The 32-bit `sctrdepth` register specifies the depth of the CTR buffer.

.Supervisor Control Transfer Records Depth Register Format
[%unbreakable]
[wavedrom, , svg]
....
{reg: [
{bits: 3, name: 'DEPTH'},
{bits: 29, name: '<i>WPRI</i>'},
], config:{lanes: 2, hspace:1024}}
....

.Supervisor Control Transfer Records Depth Register Field Definitions
[%unbreakable]
[width="100%",cols="15%,75%,10%",options="header",]
|===
|Field |Description |Access
|DEPTH |Selects the depth of the CTR buffer. Encodings:

‘000 - 16

‘001 - 32

‘010 - 64

‘011 - 128

‘100 - 256

'11x - reserved

The depth of the CTR buffer dictates the number of entries to which the hardware records transfers. For a depth of N, the hardware records transfers to entries 0..N-1. All <<_entry_registers, Entry Registers>> read as '0' and are read-only when the selected entry is in the range N to 255. When the depth is increased, the newly accessible entries contain unspecified but legal values.

It is implementation-specific which DEPTH value(s) are supported.
|WARL
|===

Attempts to access `sctrdepth` from VS-mode raise a virtual-instruction exception, unless CTR state enable access restrictions apply. See <<State Enable Access Control>>.

[NOTE]
[%unbreakable]
====
_Unlike the CTR status register or the CTR entry registers, the CTR control register has a VS-mode version. This allows a guest to manage the CTR configuration directly, without requiring traps to HS-mode, while ensuring that the guest configuration (most notably the privilege mode enable bits) do not impact CTR behavior when V=0._
_It is expected that kernels will access `sctrdepth` only at boot, to select the maximum supported depth value. More frequent accesses may result in reduced performance in virtualization scenarios, as a result of traps from VS-mode incurred._
_There may be scenarios where software chooses to operate on only a subset of the entries, to reduce overhead. In such cases tools may choose to read only the lower entries, and kernels may choose to save/restore only on the lower entries while using SCTRCLR to clear the others._
_The value in configurable depth lies in supporting VM migration. It is expected that a platform spec may specify that one or more CTR depth values must be supported. A hypervisor may wish to restrict guests to using one of these required depths, in order to ensure that such guests can be migrated to any system that complies with the platform spec. The trapping behavior specified for VS-mode accesses to `sctrdepth` ensures that the hypervisor can impose such restrictions._
====

=== Supervisor Control Transfer Records Status Register (`sctrstatus`)
Expand Down Expand Up @@ -233,11 +248,6 @@ _Because the `sctrstatus` register is updated by hardware, writes should be perf
_When restoring CTR state, `sctrstatus` should be written before CTR entry state is restored. This ensures that the software writes to logical CTR entries modify the proper physical entries._
====
[NOTE]
[%unbreakable]
====
_This register is a supervisor CSR because all of the state it contains can be exposed to both machine and supervisor mode. However, this means that Smctr/Ssctr depends upon implementation of S-mode. If, in the future, it becomes desirable to remove this dependency, a simple extension could add an `mctrstatus` CSR that reflects the same state as `sctrstatus`._
====

[NOTE]
[%unbreakable]
Expand All @@ -247,7 +257,6 @@ _Exposing the WRPTR provides a more efficient means for synthesizing CTR entries
_Exposing the WRPTR may also allow support for Linux perf's https://lwn.net/Articles/802821[[.underline]#stack stitching#] capability._
====

<<<
=== CSR Listing

.CTR CSR List
Expand All @@ -257,6 +266,7 @@ _Exposing the WRPTR may also allow support for Linux perf's https://lwn.net/Arti
| CSR Number | Name | Description
| 0x14E | `sctrctl` | Supervisor Control Transfer Records Control Register
| 0x14F | `sctrstatus` | Supervisor Control Transfer Records Status Register
| 0x15F | `sctrdepth` | Supervisor Control Transfer Records Depth Register
| 0x24E | `vsctrctl` | Virtual Supervisor Control Transfer Records Control Register
| 0x34E | `mctrctl` | Machine Control Transfer Records Control Register
|===
Expand All @@ -266,6 +276,13 @@ _Exposing the WRPTR may also allow support for Linux perf's https://lwn.net/Arti
_Because the ROI of CTR is perceived to be low for RV32 implementations, CTR does not fully support RV32. While control flow transfers in RV32 can be recorded, RV32 cannot access_ `x__ctrctl__` _bits 63:32. A future extension could add support for RV32 by adding 3 new CSRs (`mctrctlh`, `sctrctlh`, and `vsctrctlh`) to provide this access._
====

[NOTE]
[%unbreakable]
====
_The `sctrdepth` and `sctrstatus` registers are supervisor CSRs because all of the state they contain can be exposed to both machine and supervisor mode. As a result, Smctr/Ssctr depends upon implementation of S-mode. If, in the future, it becomes desirable to remove this dependency, a simple extension could add `mctrdepth` and `mctrstatus` CSRs that reflect the same state as `sctrdepth` and `sctrstatus`, respectively._
====


== Entry Registers

Control transfer records are stored in a CTR buffer, such that each buffer entry stores information about a single transfer. The CTR buffer entries are logically accessed via the indirect register access mechanism defined by the
Expand Down Expand Up @@ -423,13 +440,13 @@ SCTRCLR raises an illegal-instruction exception in U-mode, and a virtual-instruc

When Smstateen is implemented, the `mstateen0`.CTR bit controls access to CTR register state from privilege modes less privileged than M-mode. When `mstateen0`.CTR=1, accesses to CTR register state behave as described in <<CSRs>> and <<Entry Registers>> above, while SCTRCLR behaves as described in <<Supervisor CTR Clear Instruction>>. When `mstateen0`.CTR=0 and the privilege mode is less privileged than M-mode, the following operations raise an illegal-instruction exception:

* Attempts to access `sctrctl`, `vsctrctl`, or `sctrstatus`
* Attempts to access `sctrctl`, `vsctrctl`, `sctrdepth`, or `sctrstatus`
* Attempts to access `sireg*` when `siselect` is in 0x200..0x2FF, or `vsireg*` when `vsiselect` is in 0x200..0x2FF
* Execution of the SCTRCLR instruction

When `mstateen0`.CTR=0, qualified control transfers executed in privilege modes less privileged than M-mode will continue to implicitly update entry registers and `sctrstatus`.

If the H extension is implemented and `mstateen0`.CTR=1, the `hstateen0`.CTR bit controls access to supervisor CTR state when V=1. This state includes `sctrctl` (really vsctrctl), `sctrstatus`, and `sireg*` (really `vsireg*`) when `siselect` (really `vsiselect`) is in 0x200..0x2FF. `hstateen0`.CTR is read-only 0 when `mstateen0`.CTR=0.
If the H extension is implemented and `mstateen0`.CTR=1, the `hstateen0`.CTR bit controls access to supervisor CTR state when V=1. This state includes `sctrctl` (really vsctrctl), `sctrdepth`, `sctrstatus`, and `sireg*` (really `vsireg*`) when `siselect` (really `vsiselect`) is in 0x200..0x2FF. `hstateen0`.CTR is read-only 0 when `mstateen0`.CTR=0.

When `mstateen0`.CTR=1 and `hstateen0`.CTR=1, VS-mode accesses to supervisor CTR state behave as described in <<CSRs>> and <<Entry Registers>> above, while SCTRCLR behaves as described in <<Supervisor CTR Clear Instruction>>. When `mstateen0`.CTR=1 and `hstateen0`.CTR=0, both VS-mode accesses to supervisor CTR state and VS-mode execution of SCTRCLR raise a virtual-instruction exception.

Expand Down

9 comments on commit 77bbe80

@sorear
Copy link

@sorear sorear commented on 77bbe80 Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is problematic for M-mode assisted virtualization because if M-mode is pretending that physical S-mode is VS-mode, there's no way to force sctrdepth to trap except by the very heavy-handed means of setting mstateen0.CTR=0.

@bcstrongx
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true. Do you think it's required to support depth constraints in such scenarios? Do we expect to see M-mode assisted virtualization in datacenters?

@sorear
Copy link

@sorear sorear commented on 77bbe80 Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we expect to see M-mode assisted virtualization in datacenters?

No. It's used to support virtualization when the hardware wasn't originally designed for it.

Do you think it's required to support depth constraints in such scenarios?

Logical HS mode doesn't know about M-mode assistance, so it has no way of knowing that logical VS mode can access sctrdepth freely. If someone decided to use reads of sctrdepth as a hypercall interface because our specification as written guarantees it to trap in VS-mode, they would get a nasty surprise.

Writable sctrdepth is mostly useful for migration, which I doubt will overlap much with M-mode assisted virtualization.

I wasn't following the discussion closely the last few months - was there ever a proposal of making a holistic migration feature restriction extension, that would apply to sctrdepth and other features of CTR and other extensions?

@bcstrongx
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe there is talk of any such migration-enabling extension, would probably be the domain of the Hypervisor SIG.

M-mode can virtualize CTR for S-mode as defined, though if it wants to limit depth options it will have to take the heavy-handed approach of clearing mstateen0.CTR. Sounds like that's unlikely to be necessary. This seems consistent with the general approach of ensuring that M-mode can play the role of hypervisor, if not as efficiently as HS-mode can (e.g., no G-stage translation). It doesn't seem worth the complexity to add ISA to extend a similar depth-limiting functionality to M-mode.

@sorear
Copy link

@sorear sorear commented on 77bbe80 Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not asking for a M-mode depth limiting functionality. Depth limiting is an optional feature and it's fine to only allow that option when the H extension exists.

If the H extension is implemented in hardware, any VS-mode access to sctrdepth raises a virtual instruction exception, which will be delegated to HS-mode. HS-mode is expected to emulate the access by providing read-only access to the CTR depth, but there's no way to enforce this.

If the H extension is not implemented in hardware, "VS"-mode access to sctrdepth provides read-only access to the CTR depth, unless M-mode interposes access to the CTR entirely. No "HS"-mode virtual instruction exception can be generated because no hardware exception was generated.

I'd like these two to match, one way or the other.

@bcstrongx
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow. I assume that, when the H extension is not implemented in HW, your "VS"-mode is actually S-mode, and "HS"-mode is M-mode?

Assuming we keep the existing trapping behavior on VS-mode accesses to sctrdepth, I think you're asking for something like: if the H extension is not implemented in HW, S-mode accesses to sctrdepth should raise an illegal instruction exception. That would provide the symmetry I think you want, but would violate CSR address mapping conventions. Less symmetrical, but perhaps close enough, would be to make sctrdepth read-only from S-mode when the H extension is not implemented. Then writes would raise an illegal instruction exception, allowing M-mode to emulate the write. Is that the idea?

I'm hesitant to add new behaviors just to make M-mode assisted virtualization faster. As you point out, M-mode can virtualize CTR properly (if slowly) by clearing mstateen0.CTR. Adding traps like I describe above would reduce virtualization overhead for M-mode assisted virtualization, but would make the common case slower by adding unneeded traps.

@sorear
Copy link

@sorear sorear commented on 77bbe80 Mar 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow. I assume that, when the H extension is not implemented in HW, your "VS"-mode is actually S-mode, and "HS"-mode is M-mode?

VS-mode is actually S-mode. HS-mode is also actually S-mode. The V bit is stored in M-mode accessible memory (perhaps HPV in the hstatus image) so M-mode knows where traps came from and can set TVM, TSR, TW appropriately.

Assuming we keep the existing trapping behavior on VS-mode accesses to sctrdepth, I think you're asking for something like: if the H extension is not implemented in HW, S-mode accesses to sctrdepth should raise an illegal instruction exception.

I was asking for a second M-mode writable bit which would behave like mstateen0.CTR but only for sctrdepth.

I'm hesitant to add new behaviors just to make M-mode assisted virtualization faster. As you point out, M-mode can virtualize CTR properly (if slowly) by clearing mstateen0.CTR.

I think at the time I was thinking that it would make all supervisor CTR access slow on systems that require M-mode assisted virtualization. On current look, it only affects the guest so it's probably fine.

@bcstrongx
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like I can close this. Feel free to re-open it if I misunderstood.

@bcstrongx
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot this was a comment, not an issue. But I'll consider it resolved.

Please sign in to comment.