Fix: Skip sanity check of illegal tokens
Additional fix: set c0_ to kEndOfInput

Bug: v8:8363, v8:7926
Change-Id: I4f1222945914462e495d9ed6b86d38e478adbe39
Reviewed-on: https://chromium-review.googlesource.com/c/1304298
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57085}

Bug: chromium:897815, chromium:897935, chromium:897766
Change-Id: Ie8c4105867f84054559a4c43c23be82365502c1e
Reviewed-on: https://chromium-review.googlesource.com/c/1296469Reviewed-by: Marja Hölttä <marja@chromium.org>
Reviewed-by: Toon Verwaest <verwaest@chromium.org>
Commit-Queue: Florian Sattler <sattlerf@google.com>
Cr-Commit-Position: refs/heads/master@{#56905}

The primary purpose of this is to untangle a circular dependency
objects.h -> handles.h -> objects.h. Most compilation units only
need message-template.h, without the rest of messages.h.
Bonus: change the enum to an enum class for improved type safety.

Bug: v8:3770
Cq-Include-Trybots: luci.v8.try:v8_linux_noi18n_rel_ng
Change-Id: I8102c55197a450811de2588a68a08e7f99ea6b9e
Reviewed-on: https://chromium-review.googlesource.com/c/1272193
Commit-Queue: Jakob Kummerow <jkummerow@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Reviewed-by: Ben Titzer <titzer@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Reviewed-by: Adam Klein <adamk@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56583}

Now duplicate parameter detection depends on tracking of unresolved references.
This also fixes finding duplicate parameters of arrow functions nested in other
arrow functions.

Change-Id: I644bfdc513244637345c1069e5c7e5fde713da63
Reviewed-on: https://chromium-review.googlesource.com/c/1270578
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56467}

It was shipped in Chrome 67.

Bug: v8:6791, v8:8238
Cq-Include-Trybots: luci.chromium.try:linux_chromium_headless_rel;luci.chromium.try:linux_chromium_rel_ng;luci.v8.try:v8_linux_noi18n_rel_ng;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I94d8f0aa18570452403a35dea270b18f155c970a
Reviewed-on: https://chromium-review.googlesource.com/1253604Reviewed-by: Georg Neis <neis@chromium.org>
Commit-Queue: Mathias Bynens <mathias@chromium.org>
Cr-Commit-Position: refs/heads/master@{#56310}

Bug: v8:7926
Change-Id: I62e05240fd2bb165ddf8965b77763ec11a40a609
Reviewed-on: https://chromium-review.googlesource.com/1202543
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Reviewed-by: Adam Klein <adamk@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55636}

This is a reland of 5bbd1921

Original change's description:
> [scanner] Inlining ScanIdentifier and Token check
>
> This improves parse time for code-load and other benchmarks.
>
> Bug: v8:7926
> Change-Id: I9c8017f2e7f73dd952c025db5abe8e4062e6ef9b
> Reviewed-on: https://chromium-review.googlesource.com/1196506
> Commit-Queue: Florian Sattler <sattlerf@google.com>
> Reviewed-by: Toon Verwaest <verwaest@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55521}

TBR=verwaest@chromium.org,marja@chromium.org

Bug: v8:7926
Change-Id: I35f265a2a2189325daaf4a361bda35bf82758b16
Reviewed-on: https://chromium-review.googlesource.com/1203731Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55612}

This reverts commit 5bbd1921.

Reason for revert: Tentative revert for crbug.com/879988

TBR=sattlerf@chromium.org,verwaest@chromium.org

Original change's description:
> [scanner] Inlining ScanIdentifier and Token check
>
> This improves parse time for code-load and other benchmarks.
>
> Bug: v8:7926
> Change-Id: I9c8017f2e7f73dd952c025db5abe8e4062e6ef9b
> Reviewed-on: https://chromium-review.googlesource.com/1196506
> Commit-Queue: Florian Sattler <sattlerf@google.com>
> Reviewed-by: Toon Verwaest <verwaest@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55521}

TBR=marja@chromium.org,verwaest@chromium.org,sattlerf@google.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: v8:7926, chromium:879988
Change-Id: Ic71d9cf63b69c0f9560851259f434b02cc082417
Reviewed-on: https://chromium-review.googlesource.com/1202086
Commit-Queue: Yang Guo <yangguo@chromium.org>
Reviewed-by: Yang Guo <yangguo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55592}

Bug: v8:7926
Change-Id: I9b8129d60fc4d65481757222c255e883b24f47ab
Reviewed-on: https://chromium-review.googlesource.com/1196549
Commit-Queue: Florian Sattler <sattlerf@google.com>
Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Reviewed-by: Georg Neis <neis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55586}

This improves parse time for code-load and other benchmarks.

Bug: v8:7926
Change-Id: I9c8017f2e7f73dd952c025db5abe8e4062e6ef9b
Reviewed-on: https://chromium-review.googlesource.com/1196506
Commit-Queue: Florian Sattler <sattlerf@google.com>
Reviewed-by: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55521}

This reverts commit 19a8f1ba.

Reason for revert: it's actually slower this way

Original change's description:
> [scanner] Use AdvanceUntil in SkipWhiteSpace
> 
> Change-Id: I4a578589290e91da2eae4cef8f936e6c888e177c
> Reviewed-on: https://chromium-review.googlesource.com/1194015
> Reviewed-by: Igor Sheludko <ishell@chromium.org>
> Commit-Queue: Toon Verwaest <verwaest@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55497}

TBR=ishell@chromium.org,verwaest@chromium.org

Change-Id: I93b51104bef984332536360c6321eaa1dab8fd74
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/1196402Reviewed-by: Toon Verwaest <verwaest@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55513}

Change-Id: I4a578589290e91da2eae4cef8f936e6c888e177c
Reviewed-on: https://chromium-review.googlesource.com/1194015Reviewed-by: Igor Sheludko <ishell@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55497}

They are properly initialized upon creation.

Cq-Include-Trybots: luci.chromium.try:linux_chromium_headless_rel;master.tryserver.blink:linux_trusty_blink_rel
Change-Id: I10ac441580bf957e97ce663c9c3ad268ddeae935
Reviewed-on: https://chromium-review.googlesource.com/1188573
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Aleksey Kozyatinskiy <kozyatinskiy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55492}

Using a circular buffer regresses performance unfortunately.

Change-Id: Id5b68cae798d21f46376141f86d5707794bf08d6
Reviewed-on: https://chromium-review.googlesource.com/1194064Reviewed-by: Igor Sheludko <ishell@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55478}

Change-Id: I6fe237d4aec3745e993a65ddf31f5fafc3ce175d
Reviewed-on: https://chromium-review.googlesource.com/1193368Reviewed-by: Igor Sheludko <ishell@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55462}

Instead of using explicit current_, next_, and next_next_ pointers we now use a
range from token_start_ to token_end_. Upon scan we "precrement" token_end_ to
update the scan_target(). This will allow us to increase the buffer size later.

Change-Id: I816d208eafc33c29a9888e7040696b56b1409fb8
Reviewed-on: https://chromium-review.googlesource.com/1189984Reviewed-by: Igor Sheludko <ishell@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55450}

Change-Id: I331ff09ee06db39e2ca051db0a91823f5b8fb402
Reviewed-on: https://chromium-review.googlesource.com/1193262Reviewed-by: Igor Sheludko <ishell@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55445}

Change-Id: I228bbd81807fdfac9e760576f89aeb133e45cf61
Reviewed-on: https://chromium-review.googlesource.com/1186326
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55399}

This embeds LiteralBuffers in the TokenDesc directly so that we do not need to
figure out which one is free; as well as newline tracking. Instead of copying
around TokenDesc we now just update the pointer to keep track of the state.
Based on this architecture we'll be able to precompute more tokens at once.


Change-Id: Ie2e1a95f91713f7ab619fc8632f1eb644884a51f
Reviewed-on: https://chromium-review.googlesource.com/1184911
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55385}

This reverts commit 7fb6109b.

Reason for revert: Speculatively reverting because of https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8-Blink%20Linux%2064%20(dbg)/13264

Original change's description:
> [scanner] Add Skip to be used after successful Peek
> 
> Change-Id: Ic3df370e2859bf77572b34a314ad8ed17b75b942
> Reviewed-on: https://chromium-review.googlesource.com/1183485
> Commit-Queue: Toon Verwaest <verwaest@chromium.org>
> Reviewed-by: Leszek Swirski <leszeks@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#55279}

TBR=leszeks@chromium.org,verwaest@chromium.org

Change-Id: Ie1825cb4c971d2ec6a00b7ce3384c97ebbf885ce
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/1184922Reviewed-by: Maya Lekova <mslekova@chromium.org>
Commit-Queue: Maya Lekova <mslekova@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55301}

Change-Id: Ic3df370e2859bf77572b34a314ad8ed17b75b942
Reviewed-on: https://chromium-review.googlesource.com/1183485
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55279}

Change-Id: I50f729eac8d8b0c25a1f83f2b1f86800f21a8a8b
Reviewed-on: https://chromium-review.googlesource.com/1183301
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55272}

Change-Id: I692ce8dbe3169cfb912647c31a9e8121dc5eff5d
Reviewed-on: https://chromium-review.googlesource.com/1183306
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55268}

This reverts the following 3 CLs:

Revert "[scanner] Templatize scan functions by encoding"
Revert "[asm] Remove invalid static cast of character stream"
Revert "[scanner] Prepare CharacterStreams for specializing scanner and parser by character type"

The original idea behind this work was to avoid copying, converting and
buffering characters to be scanned by specializing the scanner functions. The
additional benefit was for scanner functions to have a bigger window over the
input. Even though we can get a pretty nice speedup from having a larger
window, in practice this rarely helps. The cost is a larger binary.

Since we can't eagerly convert utf8 to utf16 due to memory overhead, we'd also
need to have a specialized version of the scanner just for utf8. That's pretty
complex, and likely won't be better than simply bulk converting and buffering
utf8 as utf16.

Change-Id: Ic3564683932a0097e3f9f51cd88f62c6ac879dcb
Reviewed-on: https://chromium-review.googlesource.com/1183190Reviewed-by: Andreas Haas <ahaas@chromium.org>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55258}

This way we can avoid reencoding everything to utf16 (buffered) and avoid the
overhead of needing to check the encoding for each character individually.

This may result in a minor asm.js scanning regression due to one-byte tokens
possibly being more common.

Change-Id: I90b51c256d56d4f4fa2d235d7e1e58fc01e43f31
Reviewed-on: https://chromium-review.googlesource.com/1172437
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Andreas Haas <ahaas@chromium.org>
Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55217}

Change-Id: I62ddddf64d4de79765469e883295e246a18d7f84
Reviewed-on: https://chromium-review.googlesource.com/1177749Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55172}

Change-Id: I3c04e3cedfa220d58eb1307977b99869485c7f4d
Reviewed-on: https://chromium-review.googlesource.com/1177744Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55170}

Change-Id: Ic70a83335c60ba2f1f656398fc6b19db7af99348
Reviewed-on: https://chromium-review.googlesource.com/1175915
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Leszek Swirski <leszeks@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55157}

[scanner] Explicitly add one-byte characters to a one-byte literal buffer when scanning ascii identifiers

Change-Id: Ia8337506d9f442d6887b6fa3c2ed17210ffd3c28
Reviewed-on: https://chromium-review.googlesource.com/1172789Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55096}

Change-Id: I0d23484d9a48a73575ca158b07b7ed1b83d4029e
Reviewed-on: https://chromium-review.googlesource.com/1172774Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55095}

Change-Id: If008d618d3cb27c38cf814b1858244578d0c4e84
Reviewed-on: https://chromium-review.googlesource.com/1169213Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55090}

This is a reland of c1226cea1ec11d5b766226c912c475647a731274

Original change's description:
> [scanner] Simplify TemplateSpan raw literal character handling
> 
> Instead of adding and removing literal chars, only add raw literal characters when we have to and never remove them.
> 
> Change-Id: Ib604c8c9fb69a96708eec3a03de102e0668c01d7
> Reviewed-on: https://chromium-review.googlesource.com/1167505
> Reviewed-by: Marja Hölttä <marja@chromium.org>
> Reviewed-by: Caitlin Potter <caitp@igalia.com>
> Commit-Queue: Toon Verwaest <verwaest@chromium.org>

Change-Id: Ia15501d75c3beaf336e90a80e0abb738f696ef9e
Reviewed-on: https://chromium-review.googlesource.com/1170604Reviewed-by: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55040}

This templatizes CharacterStream by char type, and makes them subclass ScannerStream.
Methods that are widely used by tests are marked virtual on ScannerStream and final on
CharacterStream<T> so the specialized scanner will know what to call. ParseInfo passes
around ScannerStream, but the scanner requires the explicit CharacterStream<T>. Since
AdvanceUntil is templatized by FunctionType, I couldn't mark that virtual; so instead
I adjusted those tests to operate directly on ucs2 (not utf8 since we'll drop that in
the future).

In the end no functionality was changed. Some calls became virtual in tests. This is
mainly just preparation.

Change-Id: I0b4def65d3eb8fa5c806027c7e9123a590ebbdb5
Reviewed-on: https://chromium-review.googlesource.com/1156690
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Michael Starzinger <mstarzinger@chromium.org>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54848}

AdvanceUntil allows the Utf16CharacterStream to advance until a charater is found
that passes the check.

Bug: v8:7926
Change-Id: Iae39fb24194aa0ee2f544a55a7847956aa324b64
Reviewed-on: https://chromium-review.googlesource.com/1151303
Commit-Queue: Florian Sattler <sattlerf@google.com>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54783}

This places the hot part of SkipWhiteSpace in the header, allowing it to be
inlined, and leaves a slow path to handle the rest. This improves comment
scanning overall by ~10%.

Bug: v8:7926
Change-Id: I2e2ebbbae0d1af619b161397712fdf667d078884
Reviewed-on: https://chromium-review.googlesource.com/1150230
Commit-Queue: Florian Sattler <sattlerf@google.com>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54744}

Avoid copying the last character to a local variable, by checking the parsing
state in a different order.

BUG=v8:7926

Change-Id: Ifb722dd3864737dc66d8e0885adbeba1376a059e
Reviewed-on: https://chromium-review.googlesource.com/1148569Reviewed-by: Marja Hölttä <marja@chromium.org>
Reviewed-by: Toon Verwaest <verwaest@chromium.org>
Commit-Queue: Florian Sattler <sattlerf@google.com>
Cr-Commit-Position: refs/heads/master@{#54680}

This reverts commit 9c4200b2.

Reason for revert: breaks arm64 msan: https://ci.chromium.org/p/v8/builders/luci.v8.ci/V8%20Linux%20-%20arm64%20-%20sim%20-%20MSAN/22108

Original change's description:
> [scanner] Faster SkipMultiLineComment by avoiding a copy.
> 
> Avoid coping the last character to a local variable, by checking the parsing
> state in a different order.
> 
> BUG=v8:7926
> 
> Change-Id: I0b62f711674beac8c81a25dd566a5ed0d681948b
> Reviewed-on: https://chromium-review.googlesource.com/1148456
> Commit-Queue: Florian Sattler <sattlerf@google.com>
> Reviewed-by: Marja Hölttä <marja@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#54657}

TBR=marja@chromium.org,sattlerf@google.com

Change-Id: I665bce8214e6d0b8f8a619820ff13f37bc722332
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: v8:7926
Reviewed-on: https://chromium-review.googlesource.com/1148620Reviewed-by: Clemens Hammacher <clemensh@chromium.org>
Commit-Queue: Clemens Hammacher <clemensh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54658}

Avoid coping the last character to a local variable, by checking the parsing
state in a different order.

BUG=v8:7926

Change-Id: I0b62f711674beac8c81a25dd566a5ed0d681948b
Reviewed-on: https://chromium-review.googlesource.com/1148456
Commit-Queue: Florian Sattler <sattlerf@google.com>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54657}

Bug: chromium:861953
Change-Id: Id3a57aca0b24c421ac959d69265c449eaa214c16
Reviewed-on: https://chromium-review.googlesource.com/1138083
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54573}

Change-Id: I21b0dfb572efab3257e25f2f3b81689c07f8f2d7
Reviewed-on: https://chromium-review.googlesource.com/1124562
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Marja Hölttä <marja@chromium.org>
Cr-Commit-Position: refs/heads/master@{#54177}