This embeds LiteralBuffers in the TokenDesc directly so that we do not need to
figure out which one is free; as well as newline tracking. Instead of copying
around TokenDesc we now just update the pointer to keep track of the state.
Based on this architecture we'll be able to precompute more tokens at once.


Change-Id: Ie2e1a95f91713f7ab619fc8632f1eb644884a51f
Reviewed-on: https://chromium-review.googlesource.com/1184911
Commit-Queue: Toon Verwaest <verwaest@chromium.org>
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#55385}