[wasm][fuzzer] Do not execute code with potential non-determinism

The WebAssembly spec is not fully deterministic: the sign bit of NaN
can be arbitrary. This sign bit can be observed by several WebAssembly
opcodes. In the testcase the sign bit of NaN makes the difference
between terminating code and an infinite loop.

In the libfuzzer fuzzer we have to prevent infinite loops ourselves.
At the moment we do this by only execute generated code of WebAssembly
modules for which the interpretation of the code ends in a limited
number of steps. With the non-determinism described above we cannot
guarantee the absence of infinite loops with this method. Therefore
we stop now to execute generated code of WebAssembly modules for which
we observe possible non-determinism in the interpreter.

R=clemensh@chromium.org

Bug: chromium:863829
Change-Id: I461d67df87d672bed25d6c915ba7ea5134cb5890
Reviewed-on: https://chromium-review.googlesource.com/1141945Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Commit-Queue: Andreas Haas <ahaas@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54541}

test/fuzzer/wasm-fuzzer-common.cc

@@ -332,6 +332,16 @@ int WasmExecutionFuzzer::FuzzWasmModule(Vector<const uint8_t> data,
     return 0;
   }
 
+  // The WebAssembly spec allows the sign bit of NaN to be non-deterministic.
+  // This sign bit can make the difference between an infinite loop and
+  // terminating code. With possible non-determinism we cannot guarantee that
+  // the generated code will not go into an infinite loop and cause a timeout in
+  // Clusterfuzz. Therefore we do not execute the generated code if the result
+  // may be non-deterministic.
+  if (possible_nondeterminism) {
+    return 0;
+  }
+
   bool expect_exception =
       result_interpreter == static_cast<int32_t>(0xDEADBEEF);
 
@@ -349,20 +359,14 @@ int WasmExecutionFuzzer::FuzzWasmModule(Vector<const uint8_t> data,
         "main", num_args, compiler_args.get());
   }
 
-  // The WebAssembly spec allows the sign bit of NaN to be non-deterministic.
-  // This sign bit may cause result_interpreter to be different than
-  // result_compiled. Therefore we do not check the equality of the results
-  // if the execution may have produced a NaN at some point.
-  if (!possible_nondeterminism) {
-    if (expect_exception != i_isolate->has_pending_exception()) {
-      const char* exception_text[] = {"no exception", "exception"};
-      FATAL("interpreter: %s; compiled: %s", exception_text[expect_exception],
-            exception_text[i_isolate->has_pending_exception()]);
-    }
-
-    if (!expect_exception) CHECK_EQ(result_interpreter, result_compiled);
+  if (expect_exception != i_isolate->has_pending_exception()) {
+    const char* exception_text[] = {"no exception", "exception"};
+    FATAL("interpreter: %s; compiled: %s", exception_text[expect_exception],
+          exception_text[i_isolate->has_pending_exception()]);
   }
 
+  if (!expect_exception) CHECK_EQ(result_interpreter, result_compiled);
+
   // Cleanup any pending exception.
   i_isolate->clear_pending_exception();
   return 0;