Author's note: This post is a quick start to help you get a single F# based service up and running on Kubernetes. If you want the full story on how to design a distributed system, we offer commercial training and consulting services to help you with that.
"Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications" - in other words, it will handle more deployment, health monitoring and service discovery needs out of the box, as long as you can turn your application into a container. So, let's have a quick look at how to do that with an F# application.
As part of my Building Solid Systems course, I'll be talking about authentication in distributed systems. I wanted a practical demonstration that people could play with, so I added token bearer authentication to a Freya API.
So… my first, slightly annoying answer is that I try not to. Mark Seeman has written about this in a great series of blog posts which I won't try and repeat here.
Still, there are occasions where you want to quickly and easily do… something.. with a dependency making use of the context that being inside a Freya workflow provides. Let's quickly walk through how I inject a logger into a Freya workflow which "knows" about things like the request ID Kestrel has assigned to the current request.
I'm going to use Serilog as an example below, but you could also use any other structured logging library (I like Logary, but there isn't a .NET Core release at time of writing).
I spent last week delivering a five day deep dive into F# for a group of (mostly) Ruby developers in Munich, and wanted to capture some of my thoughts before I lost them as well as give people an idea of the types of things internal training can give you.
I won't be mentioning personal, company or exact team names here as I've not been given explicit permission to do so; if the people who were on the course want to chime in I'll add their comments.
The background
Although mostly a Ruby on Rails shop, this company also relies on machine learning and expert systems to deliver some of its core services. The R&D department (who build the models) settled on F# for development as a good balance between:
familiarity of syntax (most have a background in Python and/or a ML language)
performance (Ruby had struggled here)
type safety
good "production" library support (logging, etc)
Having examined the available options in depth, they decided on a standard stack for creating F# microservices of:
Freya on Kestrel via .NET Core
Chiron for type safe JSON serialization/deserialization
They wanted to investigate the use of Hephaestus as a rules engine (Freya uses Hephaestus to process HTTP requests). Many of their machine learning models only work with quite constrained ranges of input values, and Hephaestus as a rules engine looked an effective way of routing decisions to the "correct" machine learning algorithm for a particular input range. This in turn would allow for the models to stay reasonably simple and testable.
The brief
Having made these decisions, the company needed to bring the production services team up to speed on what R&D were going to produce, especially because production had expressed an interest in having F# as an extra potential tool for their own projects.
My brief was to create 5 days of training, after which production needed to know enough about the F# libraries in use that they could work out what R&D's code was doing, and enough about running .NET code in production to feel confident adding error handling, logging, metrics, tests and all the rest of the "engineering" side of development which is not about the programming language but the surrounding ecosystem.
What we did
I knew that I had a lot of ground to cover in just 5 days, so there was no way that the team was going to come away with all of the new knowledge absorbed and at their finger tips. At the same time, it couldn't be an overwhelming flood of information.
I decided to split the training time between a deep dive in understanding a few key areas in depth (Freya's design, optics and testing), and providing worked examples for the rest which could be referred back to when they became needed. Although I had relevant training material on several of the areas already, it was all tailored in this course to fit a single theme: over the course of a week, we were going to build a microservice that did just one thing, and we were going to test the heck out of it.
The timetable ended up looking like this:
Monday AM: Introductions
high level microservice design
check everyone had all the software they needed installed
Monday PM: Freya overview
install the template
modify the hello world service to accept POSTs with a name
Tuesday AM: Optics
Chiron, Freya and Hephaestus all make heavy use of "Optics"
What are they?
Building our own
Tuesday PM: Handling external data
Using Chiron for translation, version handling and API design (using our new found knowledge of optics)
Wednesday AM: Start our actual microservice as a real project
write our first bit of domain logic to pass these tests, and plug it into the Freya API
Wednesday PM: Start making our service production worthy
Spin up a docker "infrastructure" with Kibana and ElasticSearch
Adding logging to our service, plugged into Freya to automatically capture context like request IDs
Health endpoint
How to capture metrics
Thursday AM: interesting bits & answers to questions asked
How do computational expressions work?
How would I structure a functional UI?
Thursday PM: flexible rules engines with Hephaestus
rebuilt the logic from Wednesday AM reusing the same property tests
looked at how we can splice Hephaestus rules graphs together
Friday AM: BenchmarkDotNet
now we know it's correct - is it fast?
benchmarked our two implementations of the same logic together
Friday PM: Using it all in real life
code review of pieces of the existing code base, looking at adding what we'd learned
How it went
Overall the course seemed to go really well. At the end of it, the delegates were confident about the basics of building HTTP resources with Freya and Chiron, and happily building benchmarks and tests for their existing code base. For other areas (the boiler plate for plugging logging into Kestrel and Freya, for example) they understood the concepts and felt the course notes were sufficiently detailed they that could make use of them in other situations as needed. That was incredibly pleasing to hear from my point of view, as the course notes for these sessions are by far the most time consuming part of the process to create.
Although they missed some of the features of Ruby when writing F#, pattern matching with discriminated unions was a big hit and they liked the enforced discipline of Freya that required separating the logic of the various stages of handling an HTTP request - and how reusable that made components for handling concerns such as authentication.
Finally, all 3 of the core participants (there were other people around for certain parts of the course) came away saying that they'd really enjoyed it and found it interesting throughout - so that's a big win right there!
Can you do this for us?
Yes; this particular course was tailored for the specific circumstances, but I've also provided training on the more conceptual side (functional programming concepts) through to the gritty detail of DevOps (with both new and existing code bases).
We can also tailor delivery to match your availability; for this course I traveled to Munich to deliver it, and so it was delivered in a single 5 day unit. For other clients we can arrange regular shorter sessions or even remote workshops (group or individual) with tools such as Zoom.
And if you just want to turn up at a venue and get trained, check out Building Solid Systems in F# happening 31st Jan-1st Feb 2018 in London.
It's not a complex card game; it's a quick and fun game designed to represent over the top martial arts combat in the style of Hong Kong cinema or a beat 'em up game.
Each player has a deck of cards which represent their martial art; different arts are differently weighted in their card distribution. These cards come in four main types:
1 Normal cards
A "normal" card comes in one of four suits:
Punch
Kick
Throw
Defend
They also carry a numerical value between 1 and 10, which represents both how "fast" they are and (except for defend cards) how much damage they do. A Defend card can never determine damage.
2 Special Attack cards
The fireballs, whirling hurricane kicks and mighty mega throws of the game. A special attack card lists two suits: one to use for the speed of the final attack, and one for the damage. This allows you to play 3 cards together to create an attack which is fast yet damaging.
3 Combo Attack cards
A flurry of blows! Combo cards also list two suits: one for speed, and one for the "follow up" flurry. This allows you to play 3 cards together, one of which determines the speed of the attack while the other adds to the total damage. For example, if you play a Punch/Kick Combo with a Punch 3 and a Kick 7 you end up with a speed 3, damage 10 attack.
4 Knockdown cards
You can combine a knockdown card with any other valid play to create an action that will "knockdown" your opponent.
And now we want to write a function that takes the rules for playing cards above, and turns a Card list into an Action option (telling you if the list is a valid play, and what action will result if it is).
This function is pretty critical to the overall game play, and may well also be used for validating input in the UI so getting it right will make a big difference to the experience of playing the game.
So we're going to property test our implementation in every which way we can think of…
First step: make yourself a placeholder version of the function to reference in your tests:
123
moduleBlackBelt.Domain.LogiclettoActioncards=None
Now, let's start adding properties. All of the rest goes in a single file, but I'm going to split it up with some commentary as we go.
moduleBlackBelt.Tests.LogicopenExpectoopenExpecto.ExpectoFsCheckopenFsCheckopenBlackBelt.Domain.LogicopenBlackBelt.Domain.TypesletallSuitsButsuit=[Punch;Kick;Throw;Defend]|>List.filter((<>)suit)|>Gen.elements// We need a custom generator here as only some// values are validtypeDomainArbs()=staticmemberNormalCard():Arbitrary<NormalCard>=gen{let!suit=Arb.generate<Suit>let!value=Gen.choose(1,10)return{Suit=suitValue=value}}|>Arb.fromGenstaticmemberSpecialAttackCard():Arbitrary<SpecialAttackCard>=gen{let!damageSuit=allSuitsButDefendlet!speedSuit=Arb.generate<Suit>return{SpeedSuit=speedSuitDamageSuit=damageSuit}}|>Arb.fromGenstaticmemberComboCard():Arbitrary<ComboCard>=gen{let!speedSuit=allSuitsButDefendlet!followupSuit=allSuitsButDefendreturn{SpeedSuit=speedSuitFollowUpSuit=followupSuit}}|>Arb.fromGen
We'll start off with a few general purpose bits for generating random types in our domain. I haven't gone the whole hog in making illegal states unrepresentable here, so we need to constrain a few things (like the fact that cards only have values from 1 to 10, and that you can't combo into a defend card for extra damage).
Now: let's start generating potential plays of cards. Our properties will be interested in whether a particular play is valid or invalid, and we will want to know what the resulting Action should be for valid plays.
Here we'll generate the combo card and two other cards, and then we'll override the suit of the two normal cards to ensure they're legal to be played with the combo card.
There's a quirk here (which in reality I noticed after trying to run these tests). If the two suits are the same, the fast card should determine the speed regardless of "order".
Special attack cards have an additional constraint: playing a high value speed card with a low value damage card would actually disadvantage the player, and so is not considered a valid play.
Now, more interesting is trying to generate plays which are not valid. We're not trusting the UI to do any validation here, so let's just come up with everything we can think of…
There's lots of ways to combine three cards which are not valid combos or specials. Here we use are allSuitsBut helper function to always play just the wrong cards compared to what's needed.
And here we create special attacks which are slower than they are damaging. If the speed and damage suit are the same, the cards could be used either way around to create a valid action, so instead we just return the Special card on it's own without companions to form a different invalid play.
There's more that could be added here, but I decided that was enough to keep me going for the moment and so added my invalid action generator here.
12345678910111213141516171819202122
typeActionArbs()=staticmemberGeneratedAction():Arbitrary<GeneratedAction>=gen{return!Gen.oneof[makeValidActionmakeInvalidAction]}|>Arb.fromGenletactionConfig={FsCheckConfig.defaultConfigwitharbitrary=[typeof<DomainArbs>typeof<ActionArbs>]}[<Tests>]lettoAction=testPropertyWithConfigactionConfig"toAction function"<|funaction->matchactionwith|ValidAction(cards,action)->Expect.equal(toActioncards)(Someaction)"Is an action"|InvalidActioncards->Expect.isNone(toActioncards)"Is not an attack"
Finally, I wired up the generators and defined the single property this function should obey: it should return the correct action for a valid play, or None if the play is erroneous.
The wrap
Hopefully this is a useful example for those of you using property based tests of how you can encode business logic into them: although this looks like a lot of code, creating even single examples of each of these cases would have been nearly as long and far less effective in testing.
It does tend to lead to a rather iterative approach to development, where as your code starts working for some of the use cases, you begin to notice errors in or missing cases you need to generate, which helps you find more edges cases in your code and round the circle you go again.
If you want, you're very welcome to take this code to use as a coding Kata - but be warned, it's not as simple a challenge as you might expect from the few paragraphs at the top of the post!
Lots of people these days seem to like giving Halloween sales, but historically and theologically, Halloween is really just the precursor to the real celebration: All Saints' Day.
So in the interest of getting the details right, we're having an All Saints' Day sale, starting today for 7 days. It's already live, get your 10% off your tickets now.
I'm very pleased to announce the release of an initial alpha of RouteMaster.
What is it? Well, I'll let the README speak for itself:
RouteMaster is a .NET library for writing stateful workflows on top of a message bus. It exists to make the implementation of long running business processes easy in event driven systems.
There is also example code in the repository so you can see what things are starting to look like.
For those of you following along, this will sound awfully familiar; that's because RouteMaster is the outcome of my decision to rebuild a Process Manager for EasyNetQ. The first cut of that was imaginatively called "EasyNetQ.ProcessManager", but I decided to rename it for three main reasons:
On re-reading Enterprise Integration Patterns, it occurred to me that RouteMaster was an enabler for many of the other patterns as well as the "Process Manager"
The message bus RouteMaster uses is provided as an interface; the main dll has no dependency on EasyNetQ at all
The previous EasyNetQ.ProcessManager is still available as a Nuget package supplied by my previous employer, and they have both the moral and legal rights to the package given I wrote the original on their time
A pre-emptive few FAQs:
Is this ready to use?
No, not yet. I'm out of time I can afford to spend on it right now, get in touch if you can/want to fund future development.
If you want to play, the code as provided does run and all of the process tests pass.
Urgh! All the examples are F#!?
Yes, but there is a C# friendly API in the works. See the first question :)
What infrastructure do I need to run this?
At the moment, I'm using EasyNetQ (over RabbitMQ) and PostgreSQL (via Marten) for transport and storage respectively.
What about things like NServiceBus and MassTransit?
In some ways they fall in a similar space to RouteMaster, but with a different philosophy. Just as EasyNetQ is a focused library that supplies only part of the functionality you'd find in these larger solutions, RouteMaster is designed to work with your chosen transport abstraction not replace it.
Ask not what your RouteMaster can do for you, but what you can do for your RouteMaster!
I'd really like feedback, ideas, use cases and suggestions - leave comments here or ping an issue onto the repository. If you're feeling really brave and can try and actually experiment with it, but at the moment I'm mostly hoping for concrete use cases and, well, funding.
Quite a few people over the years have hit my website searching for an EasyNetQ process manager, and others have asked me if it's still available. I'd like to hear from as many of you as possible to build the tightest, simplest solution which will do the job.
I've written about how nice Freya is as a library, but documentation is still a little light on the ground.
So here's a minimal implementation of a "microservice" Freya API, starting from which dotnet commands to run to install the Freya template, through to a running web service.
Make sure you have an up to date .NET Core SDK installed, and grab yourself the handy dandy Freya template:
1
dotnet new -i "Freya.Template::*"
Then create yourself a directory and go into it. The following command will set up a brand new Freya project using kestrel as the underlying webserver, and Hopac (rather than F# Async) for concurrency. Alternatively, you can leave both the options off and you'll get Freya running on Suave with standard Async.
1
dotnet new freya --framework kestrel --concurrency hopac
Your project should run at this point; dotnet run will spin up a webserver on port 5000 which will give a 404 at the root and text responses on /hello and /hello/name paths.
Api.fs is where all the magic of configuring Freya happens - KestrelInterop.fs contains boilerplate for making sure Routing information passes correctly between Kestrel and Freya, and Program.fs just starts Kestrel with our Freya API as an OWIN app.
Adding JSON
So, this is great and all, but we're building a microservice aren't we? That normally means JSON (or at least something more structured than plain text!).
Let's change things up so that as well as supplying the name to greet in the route, we can POST JSON with a name field to the /hello end point.
To respond in JSON, we need a Freya Represent record. We're sending a result with a fixed structure, so we don't need a serialization library or anything, we'll just construct the JSON by hand. Stick this near the top of Api.fs:
It's alive! The process manager code I've been reconstructing (see Intro and the in memory test bus) is slowly starting to take some shape.
As you can see, it comes with nice (no dependency) logging out of the box and it is async all the way down to the underlying transport.
This is still at the underlying plumbing phase in many ways: the code to construct a workflow like this is currently a boilerplate covered ugly mess - but it's all boilerplate which has been deliberately designed to allow powerful APIs to be built over the top.
Next up: a nice sleek API for creating "pipeline" workflows more easily. Then the real fun starts - pleasant to use abstractions over fork/join semantics…
Interested in seeing faster progress on this project? Drop [email protected] a line to talk sponsorship.