Conduits for tokenizing streams.

Please see README.md

conduit-tokenize-attoparsec

Conduits for tokenizing streams.

conduit-tokenize-attoparsec is an extension to the conduit library based on Data.Conduit.Attoparsec that allows the tokenization of a stream whilst also tracking arbitrary state. The library includes in-built support for tracking such as line-column and stream offset, but can be extended with type-classes to support alternative state tracking.

For an example, see app/Main.hs

Building

Prerequisites: haskell-stack is installed.

Run the following in the shell:

git clone [email protected]:haskell-works/conduit-tokenize-attoparsec.git
cd conduit-tokenize-attoparsec
stack setup
stack build
stack test
stack ghci --ghc-options -XOverloadedStrings \
  --main-is conduit-tokenize-attoparsec:exe:conduit-tokenize-attoparsec-example