json-tokens

Tokenize JSON.

Convert JSON to a token stream. This libary focuses on high performance and minimal allocations. This library is distinguished from aeson in the following ways:

• In aeson, decode parses JSON by building an AST that resembles the ABNF given in RFC 7159. Notably, this converts every JSON object to a HashMap. (This choice of intermediate data structure may not be appropritae depending on how the user wants to interpret the object). By constrast, `json-tokens` converts a document to a token sequence.

• For numbers, aeson uses scientific, but `json-tokens` uses `scientific-notation`. Although scientific and `scientific-notation` have similar APIs, `scientific-notation` includes a parser that is about 4x faster and conversion functions that are 10x faster than those found in scientific and aeson.

• For text, aeson uses the UTF-16-backed text library, but `json-tokens` uses the UTF-8-backed `text-short` library.

• Parsing is resumable in aeson, which uses attoparsec, but not in `json-tokens`, which uses bytesmith.

• In aeson, all batteries are included. In particular, the combination of typeclasses and GHC Generics (or Template Haskell) make it possible to elide lots of boilerplate. None of these are included in `json-tokens`.

The difference in design decisions means that solutions using `json-tokens` are able to decode JSON twice as fast as solutions with `aeson. In the `zeek-json` benchmark suite, a `json-tokens`-based decoding outperforms aeson's decode by a factor of two. This speed comes at a cost. Users must write more code to use `json-tokens` than they do for aeson. If high-throughput parsing of small JSON documents is paramount, this cost may be worth bearing. It is always possible to go a step further and forego tokenization entirely, parsing the desired Haskell data type directly from a byte sequence. Doing this in a low-allocation way while retaining both the ability the handle JSON object keys in any order and the ability to handle escape sequences in object keys is fiendishly difficult. Kudos to the brave soul that goes down that path. For the rest of us, `json-tokens` is a compromise worth considering.