[regexp] Add stack check to RegExpExec

Band-aid fix for infinite recursion in RegExp TFJ builtins.

TFJ builtins don't contain stack checks in general, so any deep
recursion involving only TFJ builtins can end up overflowing the stack
and segfaulting on the red area.

RegExp builtins in particular can only build such recursions using
RegExp.p.exec, and (as far as I can tell) only by modifying the instance
or prototype, thus hitting the slow path in all builtins.

This CL adds a stack check to RegExpExec, which is the choke point for
calling exec on slow-mode RegExps.

Bug: v8:7239, chromium:797481

Regression test

Change-Id: I78dbb5f868a775d9697606d513623f912639d7db
Reviewed-on: https://chromium-review.googlesource.com/856777Reviewed-by: Yang Guo <yangguo@chromium.org>
Commit-Queue: Jakob Gruber <jgruber@chromium.org>
Cr-Commit-Position: refs/heads/master@{#50511}

src/builtins/builtins-regexp-gen.cc

@@ -1667,6 +1667,12 @@ Node* RegExpBuiltinsAssembler::RegExpExec(Node* context, Node* regexp,
 
   BIND(&if_iscallable);
   {
+    // TFJ builtin recursion can only be introduced within RegExp builtins by
+    // modifying the RegExp prototype, in which case all exec calls go through
+    // this path.
+    // TODO(7239): A general solution for TFJ recursion & stack checks.
+    PerformStackCheck(context);
+
     Callable call_callable = CodeFactory::Call(isolate());
     Node* const result = CallJS(call_callable, context, exec, regexp, string);
 

test/mjsunit/regress/regress-797481.js

+// Copyright 2017 the V8 project authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+// Flags: --stack-size=100
+
+const a = /x/;
+
+a.exec = RegExp.prototype.test;
+assertThrows(() => RegExp.prototype.test.call(a), RangeError);