Fix rare stack overflow in instanceof

For a very particular special case (long "chains" of bound
functions with an undefined @@hasInstance handler), evaluating
the `instanceof` operator could lead to a very deep recursion.
This patch adds a stack check to make sure we throw rather than
crash on stack overflow.

Bug: v8:11115
Change-Id: I6bf941b9e75e9fe3a52112ade27388ac4fbbda2f
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2545624Reviewed-by: Shu-yu Guo <syg@chromium.org>
Commit-Queue: Jakob Kummerow <jkummerow@chromium.org>
Cr-Commit-Position: refs/heads/master@{#71335}

src/objects/objects.cc

@@ -883,6 +883,9 @@ MaybeHandle<Object> Object::OrdinaryHasInstance(Isolate* isolate,
   // Check if {callable} is a bound function, and if so retrieve its
   // [[BoundTargetFunction]] and use that instead of {callable}.
   if (callable->IsJSBoundFunction()) {
+    // Since there is a mutual recursion here, we might run out of stack
+    // space for long chains of bound functions.
+    STACK_CHECK(isolate, MaybeHandle<Object>());
     Handle<Object> bound_callable(
         Handle<JSBoundFunction>::cast(callable)->bound_target_function(),
         isolate);

test/mjsunit/regress/regress-11115.js

+// Copyright 2020 the V8 project authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+// Flags: --allow-natives-syntax --stack-size=100
+
+var f = function() {}
+
+for (var i = 0; i < 1000; ++i) {
+  f = f.bind();
+  Object.defineProperty(f, Symbol.hasInstance, {value: undefined});
+}
+
+try {
+  ({}) instanceof f;  // Don't overflow the stack!
+} catch (e) {
+  // Throwing a RangeError is okay.
+}