[TurboFan] Do not allow direct recursive inlining

Do not allow recursive inlining when function calls itself. i.e.f() -> f()
This is because we only get some static information for the first level
of inlining and it may not be very beneficial to just duplicate the entire
function. However, we still allow indirect recursion f() -> g() -> f() -> g1().
This helps in cases where f() is a small dispatch function. For example,
in rayTrace class.create -> obj.initialize -> class.create -> obj1.initialize.

Bug: chromium:757798
Change-Id: I0a5d9e62eabd7681849f900997b4df061b5f8ed5
Reviewed-on: https://chromium-review.googlesource.com/632622Reviewed-by: Jaroslav Sevcik <jarin@chromium.org>
Commit-Queue: Mythri Alle <mythria@chromium.org>
Cr-Commit-Position: refs/heads/master@{#47605}

src/compiler/js-inlining-heuristic.cc

@@ -106,6 +106,9 @@ Reduction JSInliningHeuristic::Reduce(Node* node) {
   // Functions marked with %SetForceInlineFlag are immediately inlined.
   bool can_inline = false, force_inline = true, small_inline = true;
   candidate.total_size = 0;
+  Node* frame_state = NodeProperties::GetFrameStateInput(node);
+  FrameStateInfo const& frame_info = OpParameter<FrameStateInfo>(frame_state);
+  Handle<SharedFunctionInfo> frame_shared_info;
   for (int i = 0; i < candidate.num_functions; ++i) {
     Handle<SharedFunctionInfo> shared =
         candidate.functions[i].is_null()
@@ -115,6 +118,21 @@ Reduction JSInliningHeuristic::Reduce(Node* node) {
       force_inline = false;
     }
     candidate.can_inline_function[i] = CanInlineFunction(shared);
+    // Do not allow direct recursion i.e. f() -> f(). We still allow indirect
+    // recurion like f() -> g() -> f(). The indirect recursion is helpful in
+    // cases where f() is a small dispatch function that calls the appropriate
+    // function. In the case of direct recursion, we only have some static
+    // information for the first level of inlining and it may not be that useful
+    // to just inline one level in recursive calls. In some cases like tail
+    // recursion we may benefit from recursive inlining, if we have additional
+    // analysis that converts them to iterative implementations. Though it is
+    // not obvious if such an anlysis is needed.
+    if (frame_info.shared_info().ToHandle(&frame_shared_info) &&
+        *frame_shared_info == *shared) {
+      TRACE("Not considering call site #%d:%s, because of recursive inlining\n",
+            node->id(), node->op()->mnemonic());
+      candidate.can_inline_function[i] = false;
+    }
     if (candidate.can_inline_function[i]) {
       can_inline = true;
       candidate.total_size += shared->bytecode_array()->length();