A Setup.hs helper for running doctests.

As of now (end of 2021), there isn't cabal doctest command. Yet, to properly work, doctest needs plenty of configuration. This library provides the common bits for writing a custom Setup.hs.

cabal-doctest

A Setup.hs helper for running doctests.

Simple example

Follow simple example for the common case of a single-library .cabal package with doctests.

To recap the example's code:

1. specify build-type: Custom in your .cabal file;

2. declare dependencies of Setup.hs:

   custom-setup
    setup-depends:
      base >= 4 && <5,
      cabal-doctest >= 1 && <1.1
   

   See Notes below for a caveat with cabal-install < 2.4.

3. Populate Setup.hs like so:

   module Main where
   
   import Distribution.Extra.Doctest (defaultMainWithDoctests)
   
   main :: IO ()
   main = defaultMainWithDoctests "doctests"
   

   Assuming your test-suite is called doctests, this Setup will generate a Build_doctests module during package build. If your test-suite goes by name foo, defaultMainWithDoctests "foo" creates a Build_foo module.

4. Use the generated module in a testsuite, simply like so:

   module Main where
   
   import Build_doctests (flags, pkgs, module_sources)
   import Data.Foldable (traverse_)
   import System.Environment (unsetEnv)
   import Test.DocTest (doctest)
   
   main :: IO ()
   main = do
       traverse_ putStrLn args -- optionally print arguments
       unsetEnv "GHC_ENVIRONMENT" -- see 'Notes'; you may not need this
       doctest args
     where
       args = flags ++ pkgs ++ module_sources
   

Ultimately, cabal test or stack test should run the doctests of your package.

Example with multiple cabal components

cabal-doctest also supports more exotic use cases where a .cabal file contains more components with doctests than just the main library, including:

• doctests in executables,
• doctests in internal libraries (if using Cabal-2.0 or later).

Unlike the simple example shown above, these examples involve named components. You don't need to change the Setup.hs script to support this use case. However, in this scenario Build_doctests will generate extra copies of the flags, pkgs, and module_sources values for each additional named component.

The simplest approach is to use x-doctest-components field in .cabal:

x-doctest-components: lib lib:internal exe:example

In that case, the test driver is generally:

module Main where

import Build_doctests (Component (..), components)
import Data.Foldable (for_)
import System.Environment (unsetEnv)
import Test.DocTest (doctest)

main :: IO ()
main = for_ components $ \(Component name flags pkgs sources) -> do
    print name
    putStrLn "----------------------------------------"
    let args = flags ++ pkgs ++ sources
    for_ args putStrLn
    unsetEnv "GHC_ENVIRONMENT"
    doctest args

There is also a more explicit approach: if you have an executable named foo, then Build_doctest will contain flags_exe_foo, pkgs_exe_foo, and module_sources_exe_foo. If the name has hyphens in it (e.g., my-exe), cabal-doctest will convert them to underscores (e.g., you'd get flags_my_exe, pkgs_my_exe, module_sources_my_exe). Internal library bar values will have a _lib_bar suffix.

An example testsuite driver for this use case might look like this:

module Main where

import Build_doctests
       (flags,            pkgs,            module_sources,
        flags_exe_my_exe, pkgs_exe_my_exe, module_sources_exe_my_exe)
import Data.Foldable (traverse_)
import System.Environment (unsetEnv)
import Test.DocTest

main :: IO ()
main = do
    unsetEnv "GHC_ENVRIONMENT"
    -- doctests for library
    traverse_ putStrLn libArgs
    doctest libArgs

    -- doctests for executable
    traverse_ putStrLn exeArgs
    doctest exeArgs
  where
    libArgs = flags            ++ pkgs            ++ module_sources
    exeArgs = flags_exe_my_exe ++ pkgs_exe_my_exe ++ module_sources_exe_my_exe

See the multiple-components-example.

Additional configuration

The cabal-doctest based Setup.hs supports a few extensions fields in pkg.cabal files to customise the doctest runner behaviour, without customising the default doctest.hs.

test-suite doctests:
  if impl(ghc >= 8.0)
    x-doctest-options: -fdiagnostics-color=never
  x-doctest-source-dirs: test
  x-doctest-modules: Servant.Utils.LinksSpec

• x-doctest-options Additional arguments passed into doctest command.
• x-doctest-modules Additional modules to doctest. May be useful if you have doctest in test or executables (i.e not default library component).
• x-doctest-src-dirs Additional source directories to look for the modules.

Notes

• If support for cabal-install < 2.4 is required, you'll have to add Cabal to setup-depends; see issue haskell/cabal#4288.

• Some versions of Cabal (for instance, 2.0) can choose to build a package's doctest test suite before the library. However, in order for cabal-doctest to work correctly, the library must be built first, as doctest relies on the presence of generated files that are only created when the library is built. See #19.

  A hacky workaround for this problem is to depend on the library itself in a doctests test suite. See simple-example.cabal for a demonstration. (This assumes that the test suite has the ability to read build artifacts from the library, a separate build component. In practice, this assumption holds, which is why this library works at all.)

• custom-setup section is supported starting from cabal-install-1.24. For older cabal-install's you have to install custom setup dependencies manually.

• stack respects custom-setup starting from version 1.3.3. Before that you have to use explicit-setup-deps setting in your stack.yaml; stack#2094.

• With base < 4.7 (GHC < 7.8, pre-2014), System.Environment.unsetEnv function will need to be imported from base-compat library. It is already in transitive dependencies of doctest. Simply declare the dependency upon base-compat, and then import System.Environment.Compat (unsetEnv) if you need that old GHC.

• You can use x-doctest-options field in test-suite doctests to pass additional flags to the doctest.

• For build-type: Configure packages, you can use defaultMainAutoconfWithDoctests function to make custom Setup.hs script.

• If you use the default . in hs-source-dirs, then running doctests might fail with weird errors (ambiguous module errors). Workaround is to move sources under src/ or some non-top-level directory.

• The extensions: field isn't supported. Upgrade your .cabal file to use at least cabal-version: >= 1.10 and use default-extensions or other-extensions.

• If you use QuickCheck properties (prop>) in your doctests, the test-suite doctest should depend on QuickCheck and template-haskell. This is a little HACK: These dependencies aren't needed to build the doctests test-suite executable. However, as we let Cabal resolve dependencies, we can pass the resolved (and installed!) package identifiers to to the doctest command. This way, QuickCheck and template-haskell are available to doctest, otherwise you'll get errors like:

    Variable not in scope:
      mkName
        :: [Char]
           -> template-haskell-2.11.1.0:Language.Haskell.TH.Syntax.Name
  

  or

    Variable not in scope:
      polyQuickCheck
        :: Language.Haskell.TH.Syntax.Name -> Language.Haskell.TH.Lib.ExpQ
  

• From version 2, Stack sets the GHC_ENVIRONMENT variable, and GHC (as invoked by doctest) will pick that up. This is undesirable: cabal-doctest passes all the necessary information on the command line already, and can lead to ambiguous module errors as GHC will load the environment in addition to what cabal-doctest instructs it to.

  Hence, cabal-doctest tells GHC to ignore package environments altogether on the command line. However, this is only possible since GHC 8.2. If you are using cabal-doctest with Stack 2 and GHC 8.0 or earlier and seeing ambiguous module errors or other mysterious failures, try manually unsetting GHC_ENVIRONMENT before invoking doctest.

• If you are on Nix. doctest will not pick up your version of GHC if you don't point it towards it, and therefore will result in "cannot satisfy -package-id" errors. You will need to set NIX_GHC and NIX_GHC_LIBDIR within your environment in order for doctest to pick up your GHC. Put the following in shell.nix and run nix-shell.

  # shell.nix
  { pkgs ? import <nixpkgs> {} }:
  let
    myHaskell = (pkgs.haskellPackages.ghcWithHoogle (p: with p; [
      # Put your dependencies here
      containers
      hslogger
    ]));
  in
  pkgs.mkShell {
    name = "myPackage";
  
    # These environment variables are important. Without these,
    # doctest doesn't pick up nix's version of ghc, and will fail
    # claiming it can't find your dependencies
    shellHook = ''
      export NIX_GHC=${myHaskell}/bin/ghc
      export NIX_GHC_LIBDIR=${myHaskell}/lib/ghc-8.10.7
    '';
    buildInputs = with pkgs; [
      myHaskell
    ];
  }
  

Copyright

Copyright 2017 Oleg Grenrus.

With contributions from:

• Ryan Scott
• Andreas Abel
• Max Ulidtko

Available under the BSD 3-clause license.