A style of maintaining and upgrading Haskell projects.
From configuration in .dhall, generate projects (stack.yaml and cabal.project) and progressively upgrade projects.
π«Έ Updo π«·
A hairstyle and a way of maintaining and upgrading Haskell projects.
What is a Project?
Updo is good for project packages, constraints, source repository packages and stackage resolver[^download-stackage-config]. More than this can be added into a project template.
[^download-stackage-config]: For cabal, the cabal.config will have to be downloaded to project-stackage/resolver.config
and then edited to comment out constraints that would otherwise lead to impossible to solve conflicting constraints because cabal constraints are additive and cannot be overridden.
Goals
- Map a single set of packages, constraints and source repository packages to projects for cabal and stack.
- Minimize the effort of upgrading constraints, source repository packages and GHC versions.
We do this with configuration in .dhall
files, makefiles and dhall text
templates. We also have some simple Haskell scripts for operations not possible in Dhall like sorting a List Text
[^dhall-sort-list-nat].
[^dhall-sort-list-nat]: Dhall can only sort List Natural
.
Template Outputs
We generate projects from the configured inputs using templates. With the dhall text
command we're able to mirror to both stack and cabal projects using text templates. See templates for more.
We provide one stack template dhall2stack
. The dhall2cabal
template for cabal is so similar that the generated outputs can be compared using file diff tooling. However, we prefer the dhall2config
template because it retains package groups that can bring order to an upgrade if the cabal solving order is roughly captured by package group ordering.
We provide one level of nesting for grouping packages together as inputs. Groups are preserved or flattened depending on the template.
dhall2stack
- Stack projects cannot import so necessarily must lose the grouping and contain the list of packages.dhall2cabal
- This template lists packages within the cabal project.dhall2config
- When generating the cabal project we keep package groups, generating the same tree structure but replacing the leaves. From eachgroup-name.dhall
we generate agroup-name.config
file.
It is nice to be able to look at packages both grouped and as one sorted list. Find the flat sorted lists of packages used by some templates in the .updo
folder.
.updo
βββ pkgs-sorted.dhall β¨ List Text
βββ pkgs-upgrade-done.dhall β¨ List Text
If you don't want groups then please put all packages in the one group.
Maintaining a Project
Maintaining a project involves configuring or reconfiguring inputs and then using make to generate the projects. What do we want to configure? We don't need much:
- A stackage resolver[^download-stackage-config].
- A list of constraints (package dependencies with their versions).
- A list of source repository packages, for unpublished[^unpublished-packages] packages or for unpublished versions of packages.
- A list of packages.
[^unpublished-packages]: A package not published to hackage or stackage.
All configuration goes into ./project-dhall
(where .
is the root folder for your Haskell project) except for the cabal.config
that we'll need to download from stackage. Save this file as project-stackage/lts-m.n.config
using the exact resolver name.
project-stackage
βββ lts-m.n.config
For each ghc-x.y.z
compiler version, create this set of inputs and templates:
project-dhall
βββ ghc-x.y.z
βββ constraints.dhall β¨ List { dep : Text, ver : Text }
βββ deps-external.dhall β¨ List { loc : Text, tag : Text, sub : List Text }
βββ deps-internal.dhall β¨ List { loc : Text, tag : Text, sub : List Text }
βββ forks-external.dhall β¨ List { loc : Text, tag : Text, sub : List Text }
βββ forks-internal.dhall β¨ List { loc : Text, tag : Text, sub : List Text }
βββ text-templates
βββ dhall2cabal.dhall β¨ template for ghc-x.y.z.dhall2cabal.project
βββ dhall2config.dhall β¨ template for ghc-x.y.z.dhall2config.project
βββ dhall2stack.dhall β¨ template for ghc-x.y.z.dhall2stack.yaml
βββ stacksnippet.dhall β¨ anything for ghc-x.y.z.dhall2stack.yaml
Anything in stacksnippet.dhall
gets added to the top of the generated stack project[^base.yaml]. The rest of the files are inputs.
[^base.yaml]: Anything in stacksnippet.dhall
will be used by dhall2yaml2stack too and is put into a base.yaml
, the topmost fragment when stitching together fragments.
1. Stackage Resolver and Constraints
The stackage resolver is able to provide us with a list of packages that work together for both stack and cabal. Any other dependencies as constraints or source packages are either not found on stackage or are different versions.
By specifying a resolver with stack we get a list of exact ==
versions for packages published to stackage. Stackage also publishes a cabal.config
file that has these package versions as cabal constraints. As cabal constraints are additive, in practice importing directly from stackage often leads to conflicts between stackage's constraints and the project's own constraints and for that reason we'll usually have to download the cabal.config
from stackage[^stackage-config-url-path] and comment out lines that are conflicted.
[^stackage-config-url-path]: The URL path to download this file from is /resolver/cabal.config
. The latest nightly is always at https://www.stackage.org/nightly/cabal.config.
We don't expect any packages to impose constraints on their dependencies at the package level, in their .cabal
files, but if they do then these constraints must fit with constraints at the project level.
2. Constraints
In constraints[^constraints] put published packages that you want to use that are not on stackage or if they are on stackage where you want to use a different version.
[^constraints]: To stack, constraints and source repository packages are both extra-deps
. We use cabal-like nomenclature but, be warned, cabal constraints cover more than just package ==version
, such as flags: cabal constraints: bson -_old-network , HaXml +splitbase
[ { dep = "diagrams-postscript", ver = "1.5" }
, { dep = "diagrams-svg", ver = "1.4.3" }
]
The type of constraints is a list of records with dependency and version fields:
List { dep : Text, ver : Text }
3. Source Repository Packages
There are various reasons to depend on source packages and forks of source packages; to keep a mirror within your organization, to add fixes that haven't been upstreamed yet, or to pick up a fix someone else has made but hasn't yet been merged upstream, released and published.
project-dhall
βββ ghc-x.y.z
βββ deps-external.dhall β¨ List { loc : Text, tag : Text, sub : List Text }
βββ deps-internal.dhall β¨ List { loc : Text, tag : Text, sub : List Text }
βββ forks-external.dhall β¨ List { loc : Text, tag : Text, sub : List Text }
βββ forks-internal.dhall β¨ List { loc : Text, tag : Text, sub : List Text }
The type of these files are all the same. They contain a list of records with fields for location, tag (commit SHA) and subdirectories:
List { loc : Text, tag : Text, sub : List Text }
Any empty sub
field must be type annotated like this [] : List Text
. Each source repository package record will bring in one dependency or as many as there are sub
fields.
These source repository *.dhall
files are for original work from your organisation (internal) and from third parties (external) and for internal and external forks. If using the empty list in any one of these files then use an explicit type annotation:
[] : List { loc : Text, tag : Text, sub : List Text }
4. Package Groups
In the pkgs
folder, create one or more groups for related packages.
project-dhall
βββ ghc-x.y.z
βββ pkgs
βΒ βββ db.dhall β¨ List Text
βΒ βββ server.dhall β¨ List Text
β βββ tools.dhall β¨ List Text
βββ pkg-groups.dhall β¨ List Text
The contents of each group is a List Text
of relative paths to folders containing package .cabal
files.
-- ./project-dhall/pkgs/tools.dhall
[ "tool/linter/"
, "tool/formatter/"
]
List the package groups in pkg-groups.dhall
[^pkg-groups]. A good order to use is the order of cabal constraint solving that you may witness when upgrading a project and seeing solving failures.
[^pkg-groups]: Dhall can't do arbitrary IO like reading files from a folder.
-- ./project-dhall/pkg-groups.dhall
[ "tools"
, "db"
, "server"
]
This gives you control of the imports in the generated ./project-cabal/pkgs.config
, itself imported into ghc-x.y.z.dhall2config.project
:
- Use
CABAL_RELATIVITY=ImportRelative
to be import-compatible withcabal-3.10
.-- ./project-cabal/pkgs.config import: ./pkgs/tools.config import: ./pkgs/db.config import: ./pkgs/server.config
- The default of
CABAL_RELATIVITY=ProjectRelative
is import-compatible withcabal-3.8
.-- ./project-cabal/pkgs.config import: ./project-cabal/pkgs/tools.config import: ./project-cabal/pkgs/db.config import: ./project-cabal/pkgs/server.config
-- ./ghc-x.y.z.dhall2config.project
import: ./project-stackage/lts-m.n.config
import: ./project-cabal/pkgs.config
import: ./project-cabal/ghc-x.y.z/constraints.config
import: ./project-cabal/ghc-x.y.z/deps-external.config
import: ./project-cabal/ghc-x.y.z/deps-internal.config
import: ./project-cabal/ghc-x.y.z/forks-external.config
import: ./project-cabal/ghc-x.y.z/forks-internal.config
Upgrading a Project
GHC Upgrade
For a GHC compiler upgrade, add another folder to project-dhall
for the upgrade from ghc-u.v.w
to ghc-x.y.z
:
project-dhall
βββ ghc-u.v.w
βββ ghc-x.y.z
βββ pkgs
βββ pkg-groups.dhall β¨ List Text
βββ pkgs-upgrade-todo.dhall β¨ List Text
At the start of a GHC upgrade put all packages into project-dhall/pkgs-upgrade-todo.dhall
. As the upgrade progresses remove packages from this list as you work on them.
Constraint Upgrade
For a dependency version upgrade, add or change an entry in constraints.dhall
and if that dependency is on stackage for the resolver we're using and if the version there differs then comment out that entry in the downloaded project-stackage/resolver-name.config
file to avoid impossible constraint solving conflicts with cabal.
Source Repository Upgrade
For a source repository upgrade, bump the tag
field if picking up a newer version.
If the tag
you were on has been superseded by a published version then delete the entry and add a constraint unless the published version is already a match in the downloaded cabal.config
file from stackage.
Adding a Fork
If you are forking then you'll want to add a record to fork-internal.dhall
and then:
- If forking a stackage package: comment out the entry in the downloaded
cabal.config
- If forking a hackage package: delete the matching entry in
constraints.dhall
Using someone else's fork is the same except add the record to fork-external.dhall
.
Unforking goes the other way, remove the fork-*.dhall
entry and either fallback to the stackage version or another version for which you'll add an entry in constraints.dhall
.
Getting Started
Dhall is a prerequisite.
As well as other make targets, updo can bootstrap itself by adding one of two recipes to project-files.mk
:
UPDO_VERSION ?= 1.0.0
HACKAGE := http://hackage.haskell.org/package
UPDO_URL := ${HACKAGE}/updo-${UPDO_VERSION}/updo-${UPDO_VERSION}.tar.gz
- When opting for the scripts:
updo/Makefile:
rm -rf updo
curl -sSL ${UPDO_URL} | tar -xz
mv updo-* updo
chmod +x $$(grep -RIl '^#!' updo)
- When opting to install the included executables:
updo/Makefile:
rm -rf updo
curl -sSL ${UPDO_URL} | tar -xz
cd updo-${UPDO_VERSION}
stack install
cd -
mv updo-* updo
Instead of curl and tar, cabal get
or stack unpack
can unpack package sources.
Make Targets
In the root of your project, add two files.
.
βββ project-files.mk
βββ project-versions.mk
In project-versions.mk
set variables for GHC and STACKAGE.
GHC_VERSION := u.v.w
GHC_UPGRADE := x.y.z
STACKAGE_VERSION := lts-j.k
STACKAGE_UPGRADE := lts-m.n
Here is a basic set up for project-files.mk
:
include project-versions.mk
include updo/Makefile
Canonical Targets
We consider the following as canonical project names and no target is required to build them (they're built by the default all
target):
$ make -f project-files.mk
stack.yaml
cabal.project
stack.upgrade.yaml
cabal.upgrade.project
We use CABAL_VIA and STACK_VIA makefile variables to decide which ghc-x.y.z
prefixed projects will be copied to the default project names (cabal.project
and stack.yaml
). This way, Updo will create a single pair of projects for one version of GHC.
GHC-Prefixed Targets
We can build ghc-x.y.z
projects explicitly:
$ make -f project-files.mk ghc-x.y.z.dhall2config.project
$ make -f project-files.mk ghc-x.y.z.dhall2cabal.project
$ make -f project-files.mk ghc-x.y.z.dhall2stack.yaml
Modified Time Travel
Be careful with modified time that make relies on. If you hand-modify a target so that it is newer than a prerequisite then make will not use its recipe to build the target. This is easy to do inadvertently with a git GUI tool when discarding changes or by doing a git restore <project-name>
. As git advances the modified time, this is effectively the same as touch <project-name>
. To recover from this situation, either;
- Remove the target with
rm <project-name>
. - Run make with its
--always-make
flag but be careful with this if bootstrapping updo itself within the makefile.
Lock Files
For stack, the project and its lock are separate targets (.yaml
and .yaml.lock
).
$ make -f project-files.mk ghc-x.y.z.dhall2stack.yaml
dhall text --file project-dhall/ghc-x.y.z/project-stack.dhall > ghc-x.y.z.dhall2stack.yaml
$ make -f project-files.mk ghc-x.y.z.dhall2stack.yaml.lock
stack build --dry-run --stack-yaml ghc-x.y.z.dhall2stack.yaml
We don't provide a make target for cabal .project.freeze
files, relying instead on a combination of constraints, those in the cabal.config
downloaded from stackage and those we add ourselves to constraints.dhall
. If you do want to freeze anyway there's a cabal command to generate a freeze file.
$ make -f project-files.mk ghc-x.y.z.dhall2cabal.project
dhall text --file project-dhall/ghc-x.y.z/project-cabal.dhall > ghc-x.y.z.dhall2cabal.project
Clean GHC-Prefixed Projects
There's a make target to remove all ghc-x.y.z
prefixed projects:
$ make -f project-files.mk clean
rm -f ghc-*.stack.* ghc-*.dhall2config.* ghc-*.dhall2cabal.* ghc-*.dhall2stack.*
ghc-*.stack2cabal.* ghc-*.cabal2stack.* ghc-*.dhall2yaml2stack.* ghc-*.sha256map.nix
GHC-Prefixed Projects as Temporary
To treat ghc-x.y.z
prefixed files as temporary, add lines like these to project-files.mk
:
.INTERMEDIATE: ghc-$(GHC_VERSION).dhall2config.project
.INTERMEDIATE: ghc-$(GHC_VERSION).dhall2cabal.project
.INTERMEDIATE: ghc-$(GHC_VERSION).dhall2stack.yaml
.INTERMEDIATE: ghc-$(GHC_VERSION).sha256map.nix
SHA256 Map Generation Method
With this snippet from project-files.mk
, we can switch between two methods of generating ghc-x.y.z.sha256map.nix
.
# How to generate project-nix/ghc-$(GHC_VERSION)/sha256map.nix?
# This is copied from ghc-$(GHC_VERSION).sha256map.nix.
# - false to generate from *.dhall inputs via sha256map.hs.
# - true to generate from stack.yaml via sha256map.py.
SHA256MAP_VIA_PYTHON ?= false
# If true, generate the sha256map from the stack.yaml with python,
# overriding the recipe for this target.
ifeq ($(SHA256MAP_VIA_PYTHON), true)
ghc-$(GHC_VERSION).sha256map.nix: stack.yaml
updo/project-nix/sha256map.py <$^ >$@
endif
The default is to use the sha256map.hs
script to generate it[^replacing_versions]:
[^replacing_versions]: Using ghc-x.y.z and lts-m.n in the example output, not the actual GHC version or resolver.
$ make -f project-files.mk ghc-x.y.z.sha256map.nix
echo \
'[./project-dhall/ghc-x.y.z/deps-external.dhall
, ./project-dhall/ghc-x.y.z/deps-internal.dhall
, ./project-dhall/ghc-x.y.z/forks-external.dhall
, ./project-dhall/ghc-x.y.z/forks-internal.dhall
, ([] : List {loc : Text, tag : Text, sub : List Text})
]' \
| ./updo/project-nix/sha256map.hs > ghc-x.y.z.sha256map.nix
To use the sha256map.py
script instead that is much slower:
$ SHA256MAP_VIA_PYTHON=true \
make -f project-files.mk project-nix/ghc-x.y.z/sha256map.nix
updo/project-nix/sha256map.py <stack.yaml >ghc-x.y.z.sha256map.nix
mkdir -p project-nix/ghc-x.y.z && cp ghc-x.y.z.sha256map.nix project-nix/ghc-x.y.z/sha256map.nix
rm ghc-x.y.z.sha256map.nix
You can read more about Updo Nix and its use with haskell.nix.
Installed Exes or Scripts
To use the executables of Updo's own package, by default, instead of its scripts, add these lines to project-files.mk
:
SHA256MAP_HS_EXE ?= true
PKG_GROUPS_HS_EXE ?= true
PKGS_SORTED_HS_EXE ?= true
PKGS_UPGRADE_DONE_HS_EXE ?= true
Why Dhall for Configuration?
Dhall has excellent imports and is an intentionally limited typed programming language. We use dhall's text
command to write cabal and stack projects using dhall text templating.
The dhall
executable comes with a format
command[^format-lsp]. This helps in the maintenance of the various .dhall
files.
An example of the limits of the language we encountered in Updo is not being able to compare anything but Natural
values. Sorting something like List Text
is not possible but there is great interop with Haskell so this can be done over there as we do with scripts.
Acknowledgements
I already knew Dhall well from maintaining hpack-dhall. Thanks Simon Hengel for suggesting I take over its maintenance.
FP Complete is the corporate sponsor of this tooling. Thanks to my collegues; Daniel Herrera RendΓ³n for first seeing this could work, Neil Mayhew for makefile tips and tricks, Abel Sen for careful reviews and thoughtful suggestions and Eduardo Lemos Rocha for advocacy and enthusiasm.
Please contact me for any questions, problems, suggestions or the like. Maintenance of Updo will be coming from me and not FP Complete.
Updo grew to fill a need after every other way I could find to maintain projects was unsatisfactory for the specific problem I had, herding hundreds of packages and their dependencies into buildable projects over two compiler versions for both cabal and stack. The more I use it, the more I love it as a fast, simple and appropriate solution for maintaining Haskell projects.
Thanks FP Complete.
[^format-lsp]: Formatting is also available with the Dhall LSP Server.