[wasm] Limit the reservation for shared wasm memory on 32-bit systems

This CL limits the amount of address space we reserve for shared
WebAssembly memory. Up until now we just reserved either the defined
maximum size of the memory or the V8-defined maximum memory size,
depending on whether the maximum size is defined or not. This could
cause OOMs easily on 32-bit systems due to address space exhaustion.
With this CL we limit the amount of address space we reserve for shared
WebAssembly memory.
1) We try to reserve at least the initial size;
2) If no maximum size is defined, we reserve 1GB by default;
3) If a maximum size is defined, then we reserve that maximum size
   but at most 1GB.

Note that the handling of shared memory here is different than the
handling of not-shared memory because for shared memory it is not
possible to grow with realloc.

R=clemensb@chromium.org

Bug: v8:12038
Change-Id: I00493b330ee00588d65cbffa6f042e039106736e
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3071206Reviewed-by: Clemens Backes <clemensb@chromium.org>
Commit-Queue: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/master@{#76116}

src/wasm/wasm-objects.cc

@@ -850,20 +850,22 @@ MaybeHandle<WasmMemoryObject> WasmMemoryObject::New(Isolate* isolate,
   }
 
 #ifdef V8_TARGET_ARCH_32_BIT
-  if (shared == SharedFlag::kNotShared) {
-    // On 32-bit platforms we need a heuristic here to balance overall memory
-    // and address space consumption. If a maximum memory size is defined, then
-    // we reserve that maximum size up to 1GB. If no maximum memory size is
-    // defined, we just allocate the initial size and grow with a realloc.
+  // On 32-bit platforms we need an heuristic here to balance overall memory
+  // and address space consumption.
   constexpr int kGBPages = 1024 * 1024 * 1024 / wasm::kWasmPageSize;
-    if (initial > kGBPages || !has_maximum) {
-      // We allocate at least the initial size. If no maximum is specified we
-      // also start with the initial size.
+  if (initial > kGBPages) {
+    // We always allocate at least the initial size.
     heuristic_maximum = initial;
-    } else {
-      // We reserve the maximum size, but at most 1GB.
+  } else if (has_maximum) {
+    // We try to reserve the maximum, but at most 1GB to avoid OOMs.
     heuristic_maximum = std::min(maximum, kGBPages);
-    }
+  } else if (shared == SharedFlag::kShared) {
+    // If shared memory has no maximum, we use an implicit maximum of 1GB.
+    heuristic_maximum = kGBPages;
+  } else {
+    // If non-shared memory has no maximum, we only allocate the initial size
+    // and then grow with realloc.
+    heuristic_maximum = initial;
   }
 #endif