Refine the Memory Reserve Handling in the Allocator (#4587)

Refining the allocator’s memory reserve for better maintainability.

The current allocator is based on the assumption that the memory reserve is at least of a WASM page size (64 KB) to guard against overflows of the shadow call stack (used in Rust and in the compiler backend).
This assumption is always met, i.e. the RTS is already safe:
* In the incremental GC: The memory reserve is at least 256 MB. When the GC is inactive, the reserve is increased by additional 640 MB.
* In the classical GCs: The reserve is constantly 256 MB.

This change relaxes this assumption and directly guards the last page in the allocator, independent of the GC’s memory reserve. This is only to increase robustness for future changes, where different memory reserves may be configured, potentially even below a page size.

rts/motoko-rts/src/memory/ic.rs

@@ -39,17 +39,23 @@ pub struct IcMemory;
 /// Page allocation. Ensures that the memory up to, but excluding, the given pointer is allocated.
 /// Ensure a memory reserve of at least one Wasm page depending on the canister state.
 /// `memory_reserve`: A memory reserve in bytes ensured during update and initialization calls.
-//  For use by queries and upgrade calls. The reserve may vary depending on the GC and the phase of the GC.
+/// For use by queries and upgrade calls. The reserve may vary depending on the GC and the phase of the GC.
 unsafe fn grow_memory(ptr: u64, memory_reserve: usize) {
     const LAST_PAGE_LIMIT: usize = 0xFFFF_0000;
     debug_assert_eq!(LAST_PAGE_LIMIT, usize::MAX - WASM_PAGE_SIZE.as_usize() + 1);
-    let limit = if keep_memory_reserve() {
-        // Spare a memory reserve during update and initialization calls for use by queries and upgrades.
-        usize::MAX - memory_reserve + 1
+    // Spare a memory reserve during update and initialization calls for use by queries and upgrades.
+    let memory_reserve = if keep_memory_reserve() {
+        memory_reserve
     } else {
-        // Spare the last Wasm memory page on queries and upgrades to support the Rust call stack boundary checks.
-        LAST_PAGE_LIMIT
+        0
     };
+    // In any case, the last Wasm memory page is reserved to guard against shadow call stack overflows.
+    // This call stack is used both by the Rust runtime system implementation and by the compiler backend,
+    // see module `Stack` in `compile.ml`. This requires function activation frames to be less than the
+    // Wasm page size.
+    debug_assert!(memory_reserve <= LAST_PAGE_LIMIT);
+    let limit = LAST_PAGE_LIMIT - memory_reserve;
+    // The pointer is one byte larger than the memory size to be allocated, see the comment above.
     if ptr > limit as u64 {
         rts_trap_with("Cannot grow memory")
     };