[simulator] Check for C stack overflows during Invoke

Simulators use separate stacks for C++ and JS. JS stack overflow checks
are performed whenever a JS function is called. However, it can be the
case that the C++ stack grows faster than the JS stack, resulting in an
overflow there. Add a check here to make that less likely.

BUG=chromium:604376
R=bmeurer@chromium.org, yangguo@chromium.org

Review-Url: https://codereview.chromium.org/2151663003
Cr-Commit-Position: refs/heads/master@{#37749}

src/execution.cc

@@ -59,6 +59,19 @@ MUST_USE_RESULT MaybeHandle<Object> Invoke(Isolate* isolate, bool is_construct,
                                            Handle<Object> new_target) {
   DCHECK(!receiver->IsJSGlobalObject());
 
+#ifdef USE_SIMULATOR
+  // Simulators use separate stacks for C++ and JS. JS stack overflow checks
+  // are performed whenever a JS function is called. However, it can be the case
+  // that the C++ stack grows faster than the JS stack, resulting in an overflow
+  // there. Add a check here to make that less likely.
+  StackLimitCheck check(isolate);
+  if (check.HasOverflowed()) {
+    isolate->StackOverflow();
+    isolate->ReportPendingMessages();
+    return MaybeHandle<Object>();
+  }
+#endif
+
   // Entering JavaScript.
   VMState<JS> state(isolate);
   CHECK(AllowJavascriptExecution::IsAllowed(isolate));