[heap] Only mark object immediately when invalidating slots

We use the invalidate_recorded_slots argument to signal to NotifyObjectLayoutChange whether a particular object layout change
could cause a tagged pointer to be replaced with an untagged value.
In such cases we need our snapshot protocol in order to allow marking
such objects concurrently.

The snapshot protocol consists of two main operations:
  1) Tracing and marking the object black on the main thread before
     performing the unsafe transition.
  2) The concurrent marker needs to read such objects into a buffer
     first and is only allowed to trace it when successfully marking
     that object black.

However, in some cases we were still doing 1) on the main thread when
the concurrent marker didn't use 2) the snapshot buffer anymore. This
CL cleans up this behavior and ensures that 1) and 2) are always paired
together.

Bug: v8:12578
Change-Id: Id83b3de866a80efedf4a72e440cbc767fe3eaea6
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/3644611Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Commit-Queue: Dominik Inführ <dinfuehr@chromium.org>
Cr-Commit-Position: refs/heads/main@{#80511}

src/heap/heap.cc

@@ -3899,22 +3899,24 @@ void Heap::FinalizeIncrementalMarkingIncrementally(
 void Heap::NotifyObjectLayoutChange(
     HeapObject object, const DisallowGarbageCollection&,
     InvalidateRecordedSlots invalidate_recorded_slots) {
-  if (incremental_marking()->IsMarking()) {
-    incremental_marking()->MarkBlackAndVisitObjectDueToLayoutChange(object);
-    if (incremental_marking()->IsCompacting() &&
-        invalidate_recorded_slots == InvalidateRecordedSlots::kYes &&
-        MayContainRecordedSlots(object)) {
+  if (invalidate_recorded_slots == InvalidateRecordedSlots::kYes) {
+    const bool may_contain_recorded_slots = MayContainRecordedSlots(object);
+
+    if (incremental_marking()->IsMarking()) {
+      incremental_marking()->MarkBlackAndVisitObjectDueToLayoutChange(object);
+      if (may_contain_recorded_slots && incremental_marking()->IsCompacting()) {
+        MemoryChunk::FromHeapObject(object)
+            ->RegisterObjectWithInvalidatedSlots<OLD_TO_OLD>(object);
+      }
+    }
+
+    if (may_contain_recorded_slots) {
+      MemoryChunk::FromHeapObject(object)
+          ->RegisterObjectWithInvalidatedSlots<OLD_TO_NEW>(object);
       MemoryChunk::FromHeapObject(object)
-          ->RegisterObjectWithInvalidatedSlots<OLD_TO_OLD>(object);
+          ->RegisterObjectWithInvalidatedSlots<OLD_TO_SHARED>(object);
     }
   }
-  if (invalidate_recorded_slots == InvalidateRecordedSlots::kYes &&
-      MayContainRecordedSlots(object)) {
-    MemoryChunk::FromHeapObject(object)
-        ->RegisterObjectWithInvalidatedSlots<OLD_TO_NEW>(object);
-    MemoryChunk::FromHeapObject(object)
-        ->RegisterObjectWithInvalidatedSlots<OLD_TO_SHARED>(object);
-  }
 #ifdef VERIFY_HEAP
   if (FLAG_verify_heap) {
     DCHECK(pending_layout_change_object_.is_null());