test-karya

Testing framework.

This is Karya's test framework, extracted to be usable standalone.

Some time before 2008 I needed to write tests for Haskell, specifically for what became https://github.com/elaforge/karya. At the time, I think HUnit existed, but it did stuff I didn't think I needed, such as manually arranging tests into a hierarchy, and didn't have stuff I thought I did, such as automatic test discovery, diffs, filename:linenumber for failed tests, etc. So I wrote my own.

Over the years some features became obsolete (e.g. ghc now supports filename:linenumber natively via call stacks), but I got used to this style of tests, so I decided to try to extract it for use in cabal projects.

Here's what I like:

• Automatic test discovery. Tests are any functions named test_* in a module named *_test.hs. In Karya, I use shake to collect the tests, but in this cabal version, I borrowed the -pgmF technique from hspec.

• Automatic test hierarchy. Tests are named by their module name, so you can run subsets by pattern matching on the name. You don't have to collect them into suites, and when running them you don't have to dig through the files to see what they named their suites.

• Low syntax overhead tests. A test is any standalone function that calls test functions (like equal):

  module M_test where
  import EL.Test.Global
  import qualified M
  
  test_something = do
      let f = M.something
      equal (f 10) 20
      equal (f 20) 30
  

  Tests don't abort on failure, so you can put a bunch together with local definitions. There's no need to name each test case, the error message will tell you module and line number. Since test_something is a normal IO () function you can run it in ghci, which is how I mainly work.

• Doesn't mess with stdout. A check, like the equal function, is any function that prints to stdout. A line starting with __-> is a failure. This was originally the simplest stupidest thing that could work, but over the last 10 years there was never a reason to change it, and it turns out to be convenient. Since test output is integrated into the stdout stream, it's in the right context with any debugging or logging.

• Colored formatted diffs. When values don't match, they are pretty printed and differing parts highlighted.

• Various other ad-hoc conveniences: equalFmt attaches a formatter so you can get higher level diffs, stringsLike for matching strings with patterns, equalf and ApproxEq typeclass for floating point, and quickcheck for quickcheck properties with formatted diffs.

how to use

See example, but the short version is:

• Install library and test-karya-generate binary with cabal install.

• Make a test module like M_test above.

• Make RunTests.hs with {-# OPTIONS_GHC -F -pgmF test-karya-generate #-}

• Add a test-suite stanza to the cabal file, like in example.

• cabal test. You can rerun the tests with dist/build/test/test, run only M_test with dist/build/test/test M_test or pass --help for more options, including running tests in parallel.