• vogelheim's avatar
    Scanner::LiteralBuffer usage cleanup. · c677f813
    vogelheim authored
    1, restrict use of LiteralBuffers to the tokens that actually need it.
      - E.g., previously the Token::FUNCTION would have a literal buffer
        containing "function", which was never actually used.
      - This eliminates copies of the string data for every call to
        PeekAhead or SetBookmark.
    2, document & enforce the "secret" Scanner API contract w/ DCHECK
      - Document & check the correspondence of token value and literal buffer.
      - Document & check preconditions for calling PeekAhead, ScanRegExp*,
        ScanTemplate*.
    
    BUG=v8:4947
    
    Review-Url: https://codereview.chromium.org/2240513003
    Cr-Commit-Position: refs/heads/master@{#38677}
    c677f813
token.h 15.6 KB