-
vogelheim authored
1, restrict use of LiteralBuffers to the tokens that actually need it. - E.g., previously the Token::FUNCTION would have a literal buffer containing "function", which was never actually used. - This eliminates copies of the string data for every call to PeekAhead or SetBookmark. 2, document & enforce the "secret" Scanner API contract w/ DCHECK - Document & check the correspondence of token value and literal buffer. - Document & check preconditions for calling PeekAhead, ScanRegExp*, ScanTemplate*. BUG=v8:4947 Review-Url: https://codereview.chromium.org/2240513003 Cr-Commit-Position: refs/heads/master@{#38677}
c677f813