Perform scientific experiments stored in a DB, and generate reports.

Please see the README on GitHub at https://github.com/schnecki/experimenter#readme

Experimenter: Scientific experiments made easy

This package can be used to run experiments and repetitions of these that are then evaluated and a report generated. The user can specify parameters, which span a set of experiment instances that are automatically run and continuously saved to a PostgreSQL database, s.t. they can be resumed on the same or other PCs. The design allows to run different experiment instances on different PCs at the same time. Thus it efficiently distributes work.

All random generators are repeated for the experiments. Thus, all (evaluation) replications use the same standard number generator, Even after a crash and restart of the experiment on a different computer.

At the end of an experiment a LaTeX report can be generated according to the user specifications. E.g. one might be interested in the mean and standard deviation of an experiment that rolls a dice. This can be specified by:

[ Mean   OverExperimentRepetitions (Of "draw") `Named` "Mean Repetitions"
, StdDev OverExperimentRepetitions (Of "draw") `Named` "StdDev Repetitions"
]

The part left of the Named tag specifies the metric and right of it is the name that will be used in the report.

Each experiment can have a preparation phase, e.g. a learning phase for machine learning algorithms, a warm-up phase, e.g. a queuing system that needs to converge to the steady state, to evaluate the properties in the evaluation phase. The later is the one used for generating report of user defined metrics. For instance, the user can define the dice draw like this:

let result = StepResult "draw" Nothing (fromIntegral $ 1 + nr `mod` 6)
in ...

The package provides a simple interface class ExperimentDef a, which is all that is needed to run experiments in. In the simplest form this amounts to a datatype that holds the experiment state and a function that implements one step of the experiment:

-- | Data type for state
data Dice =
  Dice
    StdGen -- ^ Random number generator
  deriving (Show, Generic)

-- | NFData and Serialize instances
...

-- | Experiment definition
instance ExperimentDef Dice where
  type ExpM Dice = IO
  type InputValue Dice = ()      -- ^ No Input value
  type InputState Dice = ()      -- ^ No Input state is generated when generating the input value
  type Serializable Dice = Dice  -- ^ Serialisation type is the same. One can use

  -- ^ One step, that is one dice draw.
  runStep _ (Dice g) _ _ =
    let (nr, g') = next g
        result = StepResult "draw" Nothing (fromIntegral $ 1 + nr `mod` 6)
    in return ([result], Dice g')

Here the state is saved in the type Dice, no further input information is used for each step. In an experiment of a production system this might be the demand of the current period, which should be the same over all instances of the experiment to ensure every instance has the same properties. The function runStep makes one draw of a 6 sided dice and returns the result as well as the new state.

The experiment can then be configured like this:

setup :: MkExperimentSetting a
setup _ =
  ExperimentSetting
    { _experimentBaseName             = "dice param experiment" -- ^ Gives the experiment a name.
    , _experimentInfoParameters       = []                      -- ^ For storing information which only effects the report.
    , _experimentRepetitions          = 2                       -- ^ How often to repeat the whole experiment (including the preperation phase).
    , _preparationSteps               = 0                       -- ^ How many steps to execute for the preperation phase.
    , _evaluationWarmUpSteps          = 1000                    -- ^ How many steps to execute for the warm-up phase.
    , _evaluationSteps                = 10000                   -- ^ How many steps to execute for the evaluation phase.
    , _evaluationReplications         = 1                       -- ^ How often to execute the evaluation for each experiment repetition.
    , _evaluationMaxStepsBetweenSaves = Just 100000             -- ^ Specify after how many steps the data will be saved. `Nothing` adaptively chooses a sensible value.
    }

If these settings are changed the library only deletes data when it is necessary. For instance increasing the replications or repetitions of the experiment will not delete any data. When however, the preparation steps are changed, all consecutive data, including warm-up and all evaluations, are deleted and rerun.

See Dice.hs for a full implementation, where the sides of the dices are variable over the experiment. An example report of this experiment can be found in the same folder.

Ideas and TODOs

• Time of experiment run as step result.
• Nice Reports
• Html Results?
• Statistical measures for reports
• More test for the test suite
• CSV output: average over experiments not implemented

Help

Any help is warmly welcomed. Please feel free to start coding, submit pull requests or contact me :)