Scaffolding: Dev Journal 2
This post is part of the "Dev Journal" series. Part 1 contains the series index, while the DevJournal2 tag for the CalDance project in GitLab holds the state of the repository as described here.
After the initial set up work that builds our project and packages it for deployment done, it looked might it could be time to write some code. Given we're planning to use htmx, we're going to be spending a lot of time constructing urls to inject into our site that need to match end points on the server so a good starting point seemed to be building a set of helpers for defining "bidirectional routing".
Adding the Routing.fs file to the F# project went exactly as you'd expect. We'll come back to this code when we test it properly, but to give you an idea we then updated the main server to actually start returning some HTML. The new Program.fs file new looks like this:
module Mavnn.CalDance.Server open Falco.HostBuilder open Falco.Markup open Mavnn.CalDance.Routing open Mavnn.CalDance.Routing.Operators let greetingRoute = literalSection "greetings/" ./+ stringSection "name" let indexRoute = literalSection "/" let indexEndpoint = htmxGet indexRoute (fun () _ -> Elem.html [] [ Elem.body [] [ Elem.h1 [] [ Text.raw "Hi!" ] Elem.p [] [ Text.raw "Would you like to " htmxLink greetingRoute "Bob" "greet Bob?" ] ] ]) let greetingEndpoint = htmxGet greetingRoute (fun name _ -> Elem.html [] [ Elem.body [] [ Elem.h1 [] [ Text.rawf "Hi %s!" name ] ] ]) webHost [||] { add_antiforgery endpoints [ indexEndpoint; greetingEndpoint ] }
We now have two routes, one which starts with greetings/ and then matches any string, and one which responds at /. You can see in index end point we use our new htmxLink helper to construct a link that will be matched by the greeting end point, and in the greeting end point we supply a handler that knows it is going to receive a string.
This is all type safe, and that's lovely and all... but now we have two problems.
Let's tackle the biggest problem first!
Clarity and style
Writing lists of lists is a succinct and powerful way of representing HTML, but it is also a pain in the backside to format nicely by hand. It's also very easy to bike shed1 about, leading to a lot of wasted time and churn in commits.
One of the best solutions to this is to automate code formatting following a reasonable style guide. This is especially important at the beginning of a project, or (ahem) when writing code you'd like people to follow as an example as it means all of the changes made to the project are because something meaningful has actually changed and there is a consistent style to follow along with.
Fantomas is the code formatter generally used by the F# community. We always want everyone to be using the same version and config, so let's build it into our nix configuration. The nix files we use to structure the set up of the repository are a programming language in their own right, so we can write a function to provide the correct version of Fantomas taking the version of the dotnet runtime as an input argument (we've put this in a separate file in the nix directory to keep things neat).
{ pkgs, dnc }: let version = "6.2.3"; in pkgs.stdenv.mkDerivation { pname = "fantomas"; version = version; nativeBuildInputs = with pkgs; [ unzip makeWrapper ]; src = pkgs.fetchurl { url = "https://globalcdn.nuget.org/packages/fantomas.${version}.nupkg"; hash = "sha256-Aol10o5Q7l8s6SdX0smVdi3ec2IgAx+gMksAMjXhIfU="; }; unpackPhase = '' ls -al $src unzip "$src" -d $out ''; installPhase = '' mkdir -p $out/bin cp -r * $out/bin echo '#! ${pkgs.bash}/bin/bash -e' > $out/bin/fantomas echo "FANTOMAS_PATH=$out/tools/net6.0/any/fantomas.dll" >> $out/bin/fantomas echo '${dnc.runtime_8_0}/bin/dotnet $FANTOMAS_PATH "$@"' >> $out/bin/fantomas chmod +x $out/bin/fantomas ''; }
This basically says that we want to download a particular version of Fantomas from nuget (the dotnet package library), unzip it, and then create a shell script that uses our dotnet core runtime to run it. This works because Fantomas is built using an "any CPU" build configuration, allowing us to supply the correct runtime as needed by the system we're currently using but still executing the same compiled dotnet code. For a package that included any CPU specific code the normal nix approach is to download the source and then build it ourselves.
Because we put the shell script in the bin directory of the output of this derivation (how nix refers to the definition of an enclosed package), this will be added to the path of any nix shell definition that depends on it. To make people's lives easier, we can also wrap it for common use cases which we do here to create the format-all and format-stdin commands2.
In our top level flake.nix file we can now import these tools and expose them to our developers:
let # ... snip ... fantomas = (import ./nix/fantomas.nix) { inherit pkgs dnc; }; format-all = (import ./nix/format-all.nix) { inherit pkgs fantomas; }; format-stdin = (import ./nix/format-stdin.nix) { inherit pkgs fantomas; }; # ... snip ... in rec { # Tools we want available during development devShells.default = pkgs.mkShell { buildInputs = [ dnc.sdk_8_0 pkgs.nixfmt pkgs.skopeo fantomas format-all format-stdin ]; }; # ... snip ... }
Now everybody has the same formatting tools available and an easy way to reference them. It even allows us to provide git hooks and/or attribute filters that users can choose to activate that will prevent unformatted code from being pushed or even format it as it is committed to the repository (check out the section on smudge and clean filters here if you're interested).
I'm normally quite keen on leaving the formatter settings on their default, but given the purpose of this particular repository I've also added a .editorconfig file to the repository to adjust the indentation to two spaces rather than the default four, and to reduce the aimed for line length to 60 characters to make it easier to read in the blog posts.
Testing (local)
Nearly as importantly as the code being readable is whether it actually works. Expecto is an F# unit test library that allows you to write executable test programs and defines tests as pieces of data rather than class methods with particular attributes. This can be insanely helpful in writing parameterized tests, which we'll get back to in a later post.
Right now though, we just want the tests to exist and be run in CI.
We'll start off by moving the existing server code into a directory called (... let the suspense build ...) Server. Next to it we'll create an F# console project called Server.Test and use dotnet add package to add Expecto, along with YoloDev.Expecto.TestSdk and Microsoft.NET.Test.Sdk which allow the project to also be run by calling dotnet test so everybody's editors know how to run the Expecto tests.
Finally, we add a project reference to Server from Server.Test and locally at least we're all set for running unit tests!
Let's add one to Program.fs:
module Mavnn.CalDance.Server.Test open Expecto [<Tests>] let tests = testList "My list" [ testCase "hello" (fun () -> Expect.equal "hello" "hello" "Is it me you're looking for?") ] [<EntryPoint>] let main args = // This allows running with different arguments from the command line, // as well as via `dotnet test` runTestsWithCLIArgs [] args tests
And then we can run it from the root of our project:
CalDance on main via ❄️ impure (nix-shell) ❯ dotnet run --project Server.Test # snipped warning messages about FSharp.Core versions [15:59:00 INF] EXPECTO? Running tests... <Expecto> [15:59:00 INF] EXPECTO! 1 tests run in 00:00:00.0262215 for My list.hello – 1 passed, 0 ignored, 0 failed, 0 errored. Success! <Expecto> CalDance on main via ❄️ impure (nix-shell) ❯
The current version of Expecto hasn't been updated to the latest FSharp.Core yet but it appears to work fine so we'll just keep an eye on that for now.
Testing (CI)
Now though, we have a problem. The promise of using Nix was that we wouldn't need to configure CI with lots of setup for things likes tests because our build environment is self contained, and that we could incrementally and deterministically build our sub-components. But now we either create a single nix derivation that has both our projects in, or we need to somehow package the tests separately. We don't want to create a joint derivation because we're compiling down our server code into a self contained enclosure including its own copy of the dotnet runtime.
But we can't reference that build output directly from our test project, because it is built as a self contained enclosure but in the test project we want to reference it as a library in a different executable.
This is where we play some slightly interesting tricks to get all the properties we want. Do you remember above, where we put the output of the Fantomas derivation in the bin directory to declare that the file in question was an executable? Turns out that we can also put a file in the share directory to signify that it is available to other derivations but is not directly used by any executables in this one.
It also turns out that the way the F# helpers in nix manage incremental builds is by assuming that F# nix derivations will provide a Nuget package in the share directory. This means that we can build the server code once as a self-contained executable and put it in the bin folder, but we can also build it again without the self-contained flag and package it into the share folder by adding a hook to our derivation:
# ... snip ... postInstall = '' ${dnc.sdk_8_0}/bin/dotnet \ pack \ -p:ContinuousIntegrationBuild=true \ -p:Deterministic=true \ --output "$out/share" \ --configuration "Release" ''; # ... snip ...
We'll move the derivation into its own file while we're at it to stop the main flake.nix file getting too confusing and noisy, and start passing in things like the dotnet core version and project name as variables to make it easier to keeps changes between components in sync.
Aside: there is actually a helpful boolean flag that can be used to pack F# libraries but it fairly reasonably complains if you try and package a self-contained build.
This in turn allows us to define a derivation for the test project which looks very similar to the server derivation, just that it takes to server derivation as an argument so that it can declare a project reference on it along with all the previous arguments.
Quirk alert: this works very, very, well giving us cached incremental builds but it does also require us to add a conditional package dependency on the server to our test project for the build to complete successfully under Nix. This means you end up with a project file that contains something like:
<ItemGroup> <ProjectReference Include="..\Server\CalDance.Server.fsproj" /> <PackageReference Include="CalDance.Server" Version="*" Condition=" '$(ContinuousIntegrationBuild)'=='true' " /> </ItemGroup>
To finish off our test setup, we add a new output to our flake file - a request for a JUnit formatted xml file containing our test results.
packages.test = pkgs.stdenv.mkDerivation { name = "${baseName}.TestResults"; version = version; unpackPhase = "true"; installPhase = '' ${testExecutable}/bin/CalDance.Server.Test --junit-summary $out/server.test.junit.xml ''; };
Now we can run nix build .#test in our root directory and we will get a result directory containing the test results (which will be cached unless the code of either the server or the test project changes).
Some boiler plate additions to the GitLab CI configuration finishes things off; we tell the build to build both .#dockerImage and .#test (which nix will happily build run in parallel for us) and then copy the test results to a folder in the actual build directory which we tell GitLab contains junit xml results. This is needed because the result-1 directory they are created in is a symlink to the a hash addressable store that nix uses, and it turns out GitLab's build artifact upload mechanism can't follow the symlink.
# Nothing before the build command in the script has changed since the previous post - 'nix build .#dockerImage .#test' - mkdir testResults - 'cp result-1/* testResults' - ls -lh ./result - 'skopeo inspect docker-archive://$(readlink -f ./result)' - 'skopeo copy docker-archive://$(readlink -f ./result) docker://$IMAGE_TAG' artifacts: when: always paths: - 'testResults/*.xml' reports: junit: 'testResults/*.xml'
Wrapping it all up
That seems like a nice breaking off point for now. In this next stage we have:
- Provided shared versions of formatting tools to help keep the code base consistent
- Added a test project to allow us to unit test our code
- Updated CI to run and report on those tests
- Created a standard pattern for being able to add more F# projects to our repository which will all be built deterministically and for which the build results can be independently cached
As always, if you have questions or comments on what's happened so far then leave an issue on the CalDance GitLab repository. And as a thank you note for reading this far (and to see if anyone actually is!) we now have a bonus "choose your own adventure" poll.
If you'd like to see the next post focusing on testing the code we already have, hit the thumbs up on this issue.
If you'd like to see the next post starting to actually hook up a form and a data store, hit the thumbs up on this issue instead!
Next
Part 3 continues with an end to end test of our docker container.
Footnotes:
Bike shedding is the original example used in the law of triviality as stated by C. Northcote Parkinson: "The time spent on any item of the agenda will be in inverse proportion to the sum [of money] involved." It's often used as short hand to refer to the fact that trivial matters which are easy to understand and have an opinion on will tend to create enormously more discussion and hesitation than complex problems where solving the problem even once, let alone thinking of alternative solutions, is a serious effort.
The code for the helpers looks like this:
{ pkgs, fantomas }: pkgs.writeShellScriptBin "format-all" '' ${fantomas}/bin/fantomas */src/*.fs ''
{ pkgs, fantomas }: pkgs.writeShellScriptBin "format-stdin" '' TMP_FILE=$(mktemp --suffix=".fs" || exit 1) if [ $? -ne 0 ]; then echo "$0: Cannot create temp file" exit 1 fi echo "$(</dev/stdin)" > $TMP_FILE ${fantomas}/bin/fantomas $TMP_FILE &> /dev/null cat $TMP_FILE rm $TMP_FILE ''