Test discovery for the tasty framework.

Automatic test discovery and runner for the tasty framework.

Prefix your test case names and tasty-discover will discover, collect and run them.

All popular test libraries are covered. Configure once and then just write your tests. Avoid forgetting to add test modules to your Cabal/Hpack files.

Tasty ingredients are included along with various configuration options for different use cases.

Please see the README.md below for how to get started.

tasty-discover

• Getting Started
  • Create Test Driver File
  • Configure Cabal or Hpack Test Suite
• Write Tests
  • Test Transformations (Flavored, skip, platform)
    • Flavored (test transformations)
    • Skipping Tests
    • Platform-Specific Tests
    • Combining skip and platform
    • Using skip and platform (guidelines)
  • Comment Handling
• Customise Discovery
  • No Arguments
  • With Arguments
  • Custom Main Function
• Example Project
• Change Log
• Deprecation Policy
• Contributing
• FAQ
• Maintenance
• Acknowledgements
• AI Guidelines
• Coding Style

Haskell auto-magic test discovery and runner for the tasty test framework.

Prefix your test case names and tasty-discover will discover, collect and run them. All popular Haskell test libraries are covered. Configure once then just write your tests. Remember to add your test modules to your Cabal/Hpack files. Tasty ingredients are included along with various configuration options for different use cases.

Recent improvements include:

• New --no-main option for custom test runners
• Platform-specific test filtering with platform function and logical expressions
• Skip test functionality with skip function and yellow [SKIPPED] output
• Flavored type for general-purpose test transformations with extensible design
• Enhanced support for custom test types with Tasty instances
• Better handling of backup files and directories in test discovery
• Intelligent block comment handling to prevent false test discovery
• Comprehensive documentation with more test examples

See below for full documentation and examples.

Getting Started

There are 4 simple steps:

1. Create a test driver file in the test directory
2. Mark the driver file as the main-is in the test suite
3. Mark tests with the correct prefixes
4. Customise test discovery as needed

Check out the example project to get moving quickly.

Create Test Driver File

You can name this file anything you want but it must contain the correct preprocessor definition for tasty-discover to run and to detect the configuration. It should be at the top level of the test directory.

For example (in test/Driver.hs):

{-# OPTIONS_GHC -F -pgmF tasty-discover #-}

following example, the test driver file is called Driver.hs:

test-suite test
  type: exitcode-stdio-1.0
  main-is: Driver.hs
  hs-source-dirs: test
  build-depends: base

If you use hpack, that might look like:

tests:
  test:
    main: "Driver.hs"
    source-dirs: "test"
    dependencies:
    - "base"

To ensure that tasty-discover is available even without installation, add this to the test suite in your cabal file:

  build-tool-depends:
    tasty-discover:tasty-discover

See hpack documentation for stack equivalent.

Write Tests

Create test modules and prefix the test function name with an identifier that corresponds to the testing library you wish to run the test with:

• prop_: QuickCheck properties.
• scprop_: SmallCheck properties.
• hprop_: Hedgehog properties.
• unit_: HUnit test cases.
• spec_: Hspec specifications.
• test_: Tasty TestTrees.
• tasty_: Custom tests with Tasty instances.

Here is an example test module with a bunch of different tests:

{-# LANGUAGE ScopedTypeVariables #-}

module ExampleTest where

import Data.List
import Test.Tasty
import Test.Tasty.Discover
import Test.Tasty.HUnit
import Test.Tasty.Hspec
import Test.Tasty.QuickCheck
import Test.Tasty.SmallCheck
import qualified Test.Tasty.Hedgehog as TH
import qualified Hedgehog as H
import qualified Hedgehog.Gen as G
import qualified Hedgehog.Range as R

-- HUnit test case
unit_listCompare :: IO ()
unit_listCompare = [1, 2, 3] `compare` [1,2] @?= GT

-- HUnit test case with additional info
unit_listInfo :: IO String
unit_listInfo = return "This test provides info"

-- HUnit test case with steps
unit_listSteps :: (String -> IO ()) -> IO ()
unit_listSteps step = do
  step "Setting up test data"
  step "Running the test"
  [1, 2, 3] `compare` [1,2] @?= GT

-- QuickCheck property
prop_additionCommutative :: Int -> Int -> Bool
prop_additionCommutative a b = a + b == b + a

-- SmallCheck property
scprop_sortReverse :: [Int] -> Bool
scprop_sortReverse list = sort list == sort (reverse list)

-- Hedgehog property
hprop_reverseReverse :: H.Property
hprop_reverseReverse = H.property $ do
  xs <- H.forAll $ G.list (R.linear 0 100) G.alpha
  reverse (reverse xs) H.=== xs

-- Hspec specification
spec_prelude :: Spec
spec_prelude = describe "Prelude.head" $ do
  it "returns the first element of a list" $ do
    head [23 ..] `shouldBe` (23 :: Int)

-- Simple Tasty TestTree
test_addition :: TestTree
test_addition = testProperty "Addition commutes" $ \(a :: Int) (b :: Int) -> a + b == b + a

-- List of Tasty TestTrees
test_multiplication :: [TestTree]
test_multiplication =
  [ testProperty "Multiplication commutes" $ \(a :: Int) (b :: Int) -> a * b == b * a
  , testProperty "One is identity" $ \(a :: Int) -> a * 1 == a
  ]

-- IO Tasty TestTree
test_generateTree :: IO TestTree
test_generateTree = do
  input <- pure "Some input"
  pure $ testCase input $ pure ()

-- IO List of Tasty TestTrees
test_generateTrees :: IO [TestTree]
test_generateTrees = do
  inputs <- pure ["First input", "Second input"]
  pure $ map (\s -> testCase s $ pure ()) inputs

-- Custom test with Tasty instance
--
-- Write a test for anything with a Tasty instance
-- In order to use Flavored with tasty_ functions, add tasty-discover as a library
-- dependency to your test component in the cabal file.

data CustomTest = CustomTest String Assertion

instance Tasty CustomTest where
  tasty info (CustomTest prefix act) =
    pure $ testCase (prefix ++ descriptionOf info) act

tasty_myTest :: CustomTest
tasty_myTest = CustomTest "Custom: " $ pure ()

-- Custom Tasty TestTree (can be any TestTree)
tasty_customGroup :: TestTree
tasty_customGroup = testGroup "Custom Test Group"
  [ testCase "nested test 1" $ return ()
  , testCase "nested test 2" $ (1 + 1) @?= (2 :: Int)
  ]

Test Transformations (Flavored, skip, platform)

This section covers how to transform tests using the Flavored pattern and how to apply skip and platform filters effectively.

Flavored (test transformations)

The Flavored type provides a general-purpose mechanism for transforming TestTrees generated by tasty_ functions before they are added to the test suite. This allows you to apply various options and modifications to your tests.

You can create flavored tests using the flavored constructor:

flavored :: (TestTree -> TestTree) -> a -> Flavored a

Examples

import Test.Tasty.Discover (Flavored, flavored, skip)

-- Skip a custom property test
tasty_skipProperty :: Flavored Property
tasty_skipProperty = flavored skip $ property $ do
  -- This test will be skipped and show as [SKIPPED] in yellow
  H.failure

When tests are skipped, they will show as [SKIPPED] in yellow in the test output and won't actually execute.

Skipping Tests

You can skip tests using the skip functionality provided by tasty-discover. There are multiple ways to skip tests:

For a high-level overview of when to use flavored vs direct application, see Using skip and platform (guidelines).

Using the skip function

You can use the skip function to skip any TestTree:

import Test.Tasty.Discover (skip)

-- Skip a simple test
test_skipThis :: TestTree
test_skipThis = skip $ testCase "this will be skipped" $ pure ()

-- Skip a property test
prop_skipThis :: Property -> TestTree
prop_skipThis p = skip $ testProperty "skipped property" p

Potential future uses: The Flavored mechanism is designed to be extensible and could be used for other TestTree transformations such as:

• Setting test timeouts
• Adding test metadata or descriptions
• Grouping tests under custom names
• Setting resource dependencies
• Applying multiple transformations in sequence

Important: How to skip a tasty_ test

Guideline (TL;DR): To skip a tasty_ test so it shows [SKIPPED] and doesn’t run, wrap it with flavored skip:

import Test.Tasty.Discover (Flavored, flavored, skip)

-- Skips at the TestTree level (preferred for tasty_ tests)
tasty_mySkipped :: Flavored TestTree
tasty_mySkipped = flavored skip $ testCase "will be skipped" $ pure ()

Details (why this matters): Applying skip directly to an already-constructed TestTree marks only that subtree as skipped. The test body can detect it via askOption, but the outer Tasty instance (used by tasty_ functions) may not render a top-level [SKIPPED] placeholder because the option is applied after the instance decides how to wrap the node. Using flavored skip applies the transformation early so the instance can short-circuit and substitute a skipped placeholder.

For completeness, a direct skip example that observes the option inside the test:

-- Direct skip on a TestTree: the test can read SkipTest via askOption, but the
-- outer instance may not show a top-level [SKIPPED] marker for this node.
test_directSkip :: TestTree
test_directSkip = skip $ askOption $ \(TD.SkipTest shouldSkip) ->
  testCase "observes SkipTest inside" $ assertBool "expected SkipTest" shouldSkip

Using Flavored ensures the skip is visible to the Tasty instance early enough to short-circuit with a skipped placeholder.

### Platform-Specific Tests

You can conditionally run tests based on the current operating platform using the `platform` function provided by tasty-discover. This is useful for tests that only work on specific operating systems or need to be excluded from certain platforms.

For general guidance and composition patterns with `skip`, see
[Using skip and platform (guidelines)](#using-skip-and-platform-guidelines).

The `platform` function takes a platform expression string and a `TestTree`, returning a `TestTree` that will only run if the expression evaluates to true for the current platform:

import Test.Tasty.Discover (platform)

-- Run only on Linux tasty_linuxOnly :: TestTree tasty_linuxOnly = platform "linux" $ testCase "Linux-specific functionality" $ do -- This test only runs on Linux pure ()

-- Run on all platforms except Windows
tasty_notWindows :: TestTree
tasty_notWindows = platform "!windows" $ testCase "Non-Windows functionality" $ do -- This test runs on all platforms except Windows pure ()

-- Run on Unix-like systems (Linux or Darwin) tasty_unixLike :: TestTree tasty_unixLike = platform "unix" $ testCase "Unix-like systems" $ do -- This test runs on Linux and Darwin (Unix-like systems) pure ()

#### Platform Expression Syntax

Platform expressions support the following syntax:

**Platform Names:**
- `"linux"` - Linux systems
- `"darwin"` - macOS systems 
- `"windows"` - Windows systems (mapped to "mingw32" internally)
- `"mingw32"` - Windows systems (actual System.Info.os value)
- `"unix"` - Unix-like systems (matches both "linux" and "darwin")

**Logical Operators:**
- `"!"` (NOT) - Negation, e.g., `"!windows"` means "not Windows"
- `"&"` (AND) - Conjunction, e.g., `"!windows & !darwin"` means "neither Windows nor Darwin"
- `"|"` (OR) - Disjunction, e.g., `"linux | darwin"` means "Linux or Darwin"

**Complex Examples:**

-- Run on platforms that are neither Windows nor Darwin (e.g., Linux) tasty_complexPlatform1 :: TestTree tasty_complexPlatform1 = platform "!windows & !darwin" $ testCase "Neither Windows nor Darwin" $ do pure ()

-- Run on either Linux or Darwin, but not Windows tasty_complexPlatform2 :: TestTree tasty_complexPlatform2 = platform "linux | darwin" $ testCase "Linux or Darwin only" $ do pure ()

#### Using `Flavored` with Platform Filtering

You can combine platform filtering with other test transformations using the `Flavored` type:

import Test.Tasty.Discover (Flavored, flavored, platform)

-- Apply platform filtering to custom test types tasty_platformFlavored :: Flavored TestTree tasty_platformFlavored = flavored (platform "!windows") $ testCase "Advanced platform test" $ do pure ()

-- Platform-specific property test tasty_platformProperty :: Flavored Property
tasty_platformProperty = flavored (platform "unix") $ property $ do -- This hedgehog property only runs on Unix-like systems x <- H.forAll $ G.int (R.linear 1 100) x H.=== x

#### Combining skip and platform

Platform filtering can be combined with other tasty-discover features:

-- Platform filtering with test skipping tasty_platformAndSkip :: TestTree
tasty_platformAndSkip = platform "linux" $ skip $ testCase "Linux test that's also skipped" $ do -- This would only run on Linux, but it's also skipped pure ()

-- Platform filtering with test groups tasty_platformGroup :: TestTree tasty_platformGroup = platform "unix" $ testGroup "Unix-only tests" [ testCase "Unix test 1" $ pure () , testCase "Unix test 2" $ pure () , testProperty "Unix property" $ (x :: Int) -> x >= 0 || x < 0 ]

Platform filtering works by checking the current platform against the expression at runtime. If the expression evaluates to `False`, the test is automatically skipped using the same mechanism as the `skip` function.

### Using skip and platform (guidelines)

TL;DR:
- For tests exposed via `tasty_` functions, prefer the `Flavored` pattern to apply
  transformations like `skip` and `platform` so they take effect at the TestTree level and can short-circuit execution.
- Applying `skip` directly to an already-constructed `TestTree` marks the subtree as skipped. The test can observe this via `askOption`, but the outer `Tasty` instance may not show a top-level `[SKIPPED]` placeholder.

Examples:

import Test.Tasty.Discover (Flavored, flavored, skip, platform) import Test.Tasty (TestTree, testCase)

-- Skip (preferred for tasty_ tests): tasty_mySkipped :: Flavored TestTree tasty_mySkipped = flavored skip $ testCase "will be skipped" $ pure ()

-- Platform filter with Flavored: tasty_linuxOnly :: Flavored TestTree tasty_linuxOnly = flavored (platform "linux") $ testCase "Linux only" $ pure ()

-- Compose platform and skip: tasty_linuxButSkipped :: Flavored TestTree tasty_linuxButSkipped = flavored (platform "linux") $ flavored skip $ testCase "won't run" $ pure ()

-- Direct skip on a TestTree (observed inside the test): test_directSkip :: TestTree test_directSkip = skip $ askOption $ (TD.SkipTest shouldSkip) -> testCase "observes SkipTest inside" $ assertBool "expected SkipTest" shouldSkip

Platform expressions:
- Names: "linux", "darwin", "windows" (mapped to "mingw32"), "mingw32", and "unix" (matches both linux and darwin)
- Operators: `!` (NOT), `&` (AND), `|` (OR)
- Examples:
  - `platform "!windows & !darwin"` — neither Windows nor Darwin
  - `platform "linux | darwin"` — Linux or Darwin
  - `platform "unix"` — Linux or Darwin

## Test Type Variations

### HUnit Tests (`unit_` prefix)

The `unit_` prefix supports three different function signatures:
  ### Tasty TestTrees (`test_` prefix)
- `unit_testName :: IO ()` - Basic test case
- `unit_testName :: IO String` - Test case that provides additional info
- `unit_testName :: (String -> IO ()) -> IO ()` - Test case with steps

### Tasty TestTrees (`test_` prefix)

The `test_` prefix supports four different function signatures:

- `test_testName :: TestTree` - A single test tree
- `test_testName :: [TestTree]` - A list of test trees (automatically grouped)
- `test_testName :: IO TestTree` - A test tree generated in IO
- `test_testName :: IO [TestTree]` - A list of test trees generated in IO

### Custom Tests (`tasty_` prefix)

The `tasty_` prefix works with any type that has a `Tasty` instance:

- Built-in instances for `TestTree`, `[TestTree]`, `IO TestTree`, `IO [TestTree]`
- Custom instances for your own data types
- Provides access to test metadata through `TastyInfo`

### Organizing Tests with `testGroup`

The `testGroup` function is Tasty's way of organizing tests into hierarchical groups. You can use it in several ways:

**In `test_` functions:**

test_arithmeticTests :: TestTree test_arithmeticTests = testGroup "Arithmetic Operations" [ testCase "addition" $ 2 + 2 @?= 4 , testCase "multiplication" $ 3 * 4 @?= 12 , testProperty "commutativity" $ \a b -> a + b == b + (a :: Int) ]

**In `tasty_` functions:**

tasty_myTestSuite :: TestTree tasty_myTestSuite = testGroup "My Test Suite" [ testGroup "Unit Tests" [ testCase "test 1" $ pure () , testCase "test 2" $ pure () ] , testGroup "Properties" [ testProperty "prop 1" $ \x -> x == (x :: Int) ] ]

**In list form with `test_` functions:**

test_groupedTests :: [TestTree] test_groupedTests = [ testGroup "Group 1" [testCase "test A" $ pure ()] , testGroup "Group 2" [testCase "test B" $ pure ()] ]

This creates nested test hierarchies that make test output more organized and easier to navigate.

## Comment Handling

`tasty-discover` intelligently handles Haskell comments during test discovery to prevent false positives:

### Block Comments

Tests inside multiline block comments are automatically ignored:

module MyTest where

-- This test will be discovered unit_validTest :: IO () unit_validTest = pure ()

{- This test will be ignored unit_commentedOut :: IO () unit_commentedOut = pure () -}

{- Nested comments are also handled correctly {- Even deeply nested ones unit_deeplyNested :: IO () unit_deeplyNested = pure () -} unit_alsoIgnored :: IO () unit_alsoIgnored = pure () -}

### Line Comments

Line comments (starting with `--`) are handled by the Haskell lexer and don't interfere with test discovery:

-- unit_thisIsIgnored :: IO () unit_thisIsFound :: IO () -- This test will be discovered unit_thisIsFound = pure ()

This feature prevents compilation errors that would occur if `tasty-discover` tried to reference tests that are commented out, making it easier to temporarily disable tests during development.

# Customise Discovery

You configure `tasty-discover` by passing options to the test driver file.

## No Arguments

Example: `{-# OPTIONS_GHC -F -pgmF tasty-discover -optF --debug #-}`

  - **--debug**: Output the contents of the generated module while testing.
  - **--tree-display**: Display the test output results hierarchically.
  - **--no-main**: Generate a module without a main function, exporting `tests` and `ingredients` instead.

## With Arguments

Example: `{-# OPTIONS_GHC -F -pgmF tasty-discover -optF --modules="*CustomTest.hs" #-}`

  - **--modules**: Which test modules to discover (with glob pattern).
  - **--search-dir**: Where to look for tests. This is a directory relative
    to the location of the source file. By default, this is the directory
    of the source file."
  - **--ignores**: Which test modules to ignore (with glob pattern).
  - **--generated-module**: The name of the generated test module.
  - **--ingredient**: Tasty ingredients to add to your test runner.
  - **--inplace**: Has the generated code written to the source file.

It is also possible to override [tasty test options] with `-optF`:

[tasty test options]: https://github.com/feuerbach/tasty#options

``` bash
{-# OPTIONS_GHC -F -pgmF tasty-discover -optF --hide-successes #-}

Custom Main Function

The --no-main option allows you to write your own custom main function while still using tasty-discover for test discovery. This is useful when you need to:

• Apply custom test transformations or wrappers
• Add custom logging or output formatting
• Integrate with custom test runners or CI systems
• Control exactly how tests are executed

Example Usage

Create your test discovery file (e.g., test/Tests.hs):

{-# OPTIONS_GHC -F -pgmF tasty-discover -optF --no-main -optF --generated-module -optF Tests #-}

Then create your custom main file (e.g., test/Main.hs):

module Main where

import qualified Tests
import qualified Test.Tasty as T

main :: IO ()
main = do
  putStrLn "=== Custom Test Runner ==="

  -- Get discovered tests and ingredients
  discoveredTests <- Tests.tests

  -- Apply custom transformations
  let wrappedTests = T.testGroup "My Custom Tests" [discoveredTests]

  -- Run with custom configuration
  T.defaultMainWithIngredients Tests.ingredients wrappedTests

Configure your cabal test suite to use the custom main:

test-suite my-tests
  type: exitcode-stdio-1.0
  main-is: Main.hs
  other-modules: Tests
  hs-source-dirs: test
  build-depends: base, tasty
  build-tool-depends: tasty-discover:tasty-discover

Example Project

See the testing for this package for a fully configured example.

Change Log

Please see the CHANGELOG.md for the latest changes.

We try to keep tagged releases in our release process, if you care about that.

Deprecation Policy

If a breaking change is implemented, you'll see a major version increase, an entry in the change log and a compile-time error with a deprecation warning and clear instructions on how to upgrade. Please do complain if we're doing this too much.

Contributing

All contributions welcome! The continuous integration suite is pretty comprehensive, so just get hacking and add a test case - there are plenty of examples, so this should be simple - and I'll get to review your change ASAP.

Please follow the guidelines in CODING_STYLE.md for consistent code formatting and patterns.

For AI assistants and detailed development guidelines, see AI_GUIDELINES.md.

Frequently Asked Questions

Deleting Tests Breaks The Test Run

This is a known limitation and has been reported. No fix is planned unless you have time.

Please see #145 for more information.

Maintenance

If you're interested in helping maintain this package, please let @newhoggy know!

It doesn't take much time (max ~3 hours a month) and all we need to do is:

• Triage issues that are raised.
• Review pull requests from contributors.
• Fix bugs when present.
• Make releases.
• Manage bounds issues on Stackage.

You can create an issue or drop him a line at lukewm AT riseup DOT NET.

Acknowledgements

Thanks to hspec-discover and tasty-auto for making this possible.

A huge thanks to the growing list of contributors.