[scanner] Use TokenDesc* rather TokenDesc to keep track of the stream

This embeds LiteralBuffers in the TokenDesc directly so that we do not need to
figure out which one is free; as well as newline tracking. Instead of copying
around TokenDesc we now just update the pointer to keep track of the state.
Based on this architecture we'll be able to precompute more tokens at once.


Change-Id: Ie2e1a95f91713f7ab619fc8632f1eb644884a51f
Reviewed-on: https://chromium-review.googlesource.com/1184911
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55385}

src/parsing/scanner-inl.h

@@ -21,7 +21,7 @@ V8_INLINE Token::Value Scanner::SkipWhiteSpace() {
     // Advance as long as character is a WhiteSpace or LineTerminator.
     // Remember if the latter is the case.
     if (unibrow::IsLineTerminator(c0_)) {
-      has_line_terminator_before_next_ = true;
+      next().after_line_terminator = true;
     } else if (!unicode_cache_->IsWhiteSpace(c0_)) {
       break;
     }