In my last article, I looked at front-end technologies and sang the praises of htmx, which I declared is a simpler way of implementing dynamic web applications. Simpler than the prevailing choice of today, which usually means some form of React or Vue, and which forces you to use JavaScript and have complicated build processes in order to get something working.

Predictably, I got some reactions from front-end developers. Some of those were positive or ‘positive but’. Many of the rest were along a few broad lines of thought, such as:

• htmx is only suitable for ‘small’ applications, and you’ll need a SPA framework when it becomes too large;
• I’ll still need a JSON API for my mobile apps, and I don’t want to do the same work twice;
• SPA frameworks are bad, but Next/Nuxt/Sveltekit are basically htmx, but magic;
• Your examples suck and a real developer would do this differently in React;
• Actually, htmx is just another framework;
• This doesn’t look any simpler;
• People promoting <not my solution> should learn what they’re talking about before they write something;
• It would be so hard/impossible to build X using htmx.

Some of these have already been addressed by the essays on the htmx site. Let’s focus on the last one. And let’s not talk about building X, the website formerly known as Twitter. Elon Musk is such a dick for associating the universal placeholder with that cesspool of hatred and censorship. Anyway, moving on. Let’s try and build a Trello clone using htmx. Why Trello? Well, because someone specifically called me out on it. ‘Try making a clone of trellix, an app built with remix’. To which I said ‘that doesn’t look too difficult’. So let’s do it.

Step 1: What are we building?

Its name implies that Trellix is a Trello clone. For those of you don’t know Trello: Trello is a work organizer, basically a very fancy ‘to do’ app, that lets you put cards, representing work, into stacks, which usually represent statuses, like ‘to do’, ‘doing’, ‘blocked’, and ‘done’. These stacks are represented as vertical ‘lanes’, and you can drag and drop cards between lanes or to a different position in the same lane. All of this is done on a ‘board’, which you can think of like a project. Boards are the ‘root’ of the user interface; except for managing boards themselves, everything is done in the context of a board.

What we’ll be creating is the following behaviors:

• Creating a new board;
• Renaming a board;
• Creating lanes;
• Creating cards;
• Reordering cards in a lane;
• Moving cards between lanes;
• Removing cards.

What about, for example, renaming lanes? Or editing cards? First of all, some of those are not implemented by Trellix, either, but second of all, the point of this is not to create an entire application and solve all of the potentially hard problems, but to demonstrate how simple it can be to implement something that looks very complicated. I’ll also skip the authentication and sign-up part, to keep the scope small.

Step 2: How are we building it?

Obviously, we’re using htmx, but what else? Because htmx is such a back-end-agnostic solution, I could have used pretty much anything. I chose something I’m intimately familiar with, to avoid the entire article being discredited because some troll goes ‘lol, that not how you do that in go’. ASP.NET Core MVC it is. I chose MVC over Razor Pages, because MVC gives you a little bit more control, and lets you, for example, easily return a different view depending on the outcome of an operation.

We’ll use Bootstrap 5 as a CSS ‘framework’, because I’m familiar with it, it’s succinct, and it provides a good starting point.

Step 3: Let’s build!

I’ll skip over all of the plumbing required to make it work in .NET, but feel free to check out the source code. Remember that the context of this application is as a htmx proof-of-concept, not as an example of how to build a .NET application. Corners have been cut, and there are undoubtably many edge cases that are not handled.¹

For the most part, discussing which what CSS features or Bootstrap classes were used is outside of the scope of this article. Where it’s really used to fix a problem, I will discuss it. I’ll also omit most of the classes and such from the code in this post, if they’re only used for styling purposes.

Creating boards

Our index page will be simple. There will be a form to create new boards, and a list of existing boards which you can navigate to.

Our first interesting thing is to let users create boards, so they actually have a canvas to work on. This will just be a form that will POST to an endpoint, which will create the board and redirect the user to the newly created board. There’s no real advantage in doing a partial update here, so we won’t. The redirect is the old PRG (POST-redirect-GET) pattern in action. Just because we can, we’ll make the form a ‘boosted’ form. This means the form will be a regular old HTML form, but when JavaScript is enabled, htmx will turn it into an asynchronous fetch request.

This is what the form looks like:

<form method="post"
      action="/boards"
      hx-boost="true">
    <div>
        <label for="newBoardName">Name</label>
        <input type="text" 
               required 
               name="Board.BoardName"
               id="newBoardName"
               value="@Model.Board.BoardName" 
               autocomplete="off" />
    </div>
    <div>
        <label for="newBoardColor">Color</label>
        <div>
            <input type="color" 
                   id="newBoardColor"
                   name="Board.Color"
                   value="@Model.Board.Color" />
        </div>
        <button type="submit">Create</button>
    </div>
</form>

If you’re not familiar, the @ syntax is used in Razor to indicate a C# expression.

Handling the request is simple enough: create the board, then redirect to it. There’s a problem, though: pressing the back button doesn’t work as you’d expect.

When the request is boosted, using htmx, here’s what happens: htmx will send a POST, which will result in a response with a 302 status (a redirect). Htmx will follow the redirect, request the indicated page and render the response just fine. A boosted request tries to behave like a ‘normal’ request in many ways, including history support, or in other words: you can use the back button to navigate to the previous state. To do this, htmx takes a snapshot of the page’s DOM just before swapping in the respone, and caches it in local storage. When you press the back button, htmx simply swaps in the cached response; there’s no request to refresh any of the data. This makes sense if you consider that whatever state the page was in could have been the result of an untold number of other interactions.

In contrast, when the form is submitted without htmx, the POST request results in a 302 status (a redirect), which prompts a second request. Because the response to the first is a redirect, the first response is not added to the browser’s history. When you click the back arrow on the newly created board page, the browser will just issue a new request to the index page, which will display the board you just added.

There is a number of ways we can handle this. We can turn the redirect to the board page into a full page load with the HX-Redirect header. This works, but it’s essentially defeating the point of having a boosted request.

We can prevent htmx from caching the index page, by using hx-history="false". This also works, but it can be a heavy-handed solution, since the entire page is requested when you navigate back to it.

We can also use the load trigger to request only the list of boards whenever the page loads. This might be the best solution if you want to avoid loading the entire page because it’s slow to load, for example. By default, htmx has no separate event for ‘a page was restored from the history cache’ (there’s an extension that adds it, though). This means that each time you navigate to the page, whether it’s ‘normally’ or backwards in the browser history, the load event will always be triggered. In other words: you click your way to the index page, which comes with a fresh list of boards, and then as soon as the page is done loading, a request is triggered to get a fresher list of boards. A way to deal with this somewhat gracefully would be to provide the request with some version information that lets the endpoint respond with a 204 (No Content) response in case there is nothing more recent. Still, that is an additional request.

Since our index page is pretty light, let’s use the simplest solution and set hx-history="false" on the list of boards.

Renaming a board

The first thing on our board page is the board title, which you should be able to change. Well, the first thing is really the background color of the board. The easiest way to implement this would be to set a style="background-color: <something>" on the body of the page, since that covers the entire page from top to bottom. Unfortunately, htmx does not touch the body element when doing partial page loads. Fortunately, we live in a day and age where it is trivial to make a lay-out that covers the entire page.

Our page set-up is like most Bootstrap pages: we have a navigation bar at the top, and a ‘container’ below it, which contains the page content. We set the body to display: flex to enable the ease of use of CSS Flexbox, and then we set min-height: 100vh (the vh unit represents 1% of the viewport’s height) to force it to cover all of the browser window. Then we set flex-grow: 1 on our container element and we’re set! Modern CSS is just amazing.

Back to renaming a board. In Trellix, the process is made very sleek; when you click the board name, it’s replaced by a borderless text field with the name, and you can immediately start typing and press Enter to commit your changes, or tab away or click outside the field to dismiss your changes. Let’s see if we can do something similar.

The first thing to solve is to show an input field when we click the title. That’s easy enough; we just expose an endpoint that returns the input field, plus whatever other markup is required, and then we wire up the board title to request it when clicked. Since ‘click’ is the default trigger for anything that’s not an input or form, this is very concise:

<h1 hx-get="/boards/@Model.BoardId/editName"
    hx-swap="outerHTML">
  @Model.Name
</h1>

The form to do the rename itself, which would be returned by the editName endpoint, is also not that special. I’m using a form here because we don’t want the request to be triggered on every key press, but instead only when the user presses Enter. We could use hx-trigger for this, but using a form is easier, cleaner, and it might eventually be necessary if you want to include CSRF tokens.

<form hx-put="/boards/@Model.BoardId/editName" 
      hx-swap="outerHTML">
  <input type="text" 
         name="boardName" 
         value="@Model.Name" 
         required 
         placeholder="Enter board name" 
         autocomplete="off"
         autofocus />
</form>

Note the autofocus attribute on the text field. It has been supported by browsers since 2011 to specify that a particular input should receive focus when the page is loaded. Htmx supports this attribute as well, so you can use it to move the focus after loading in new content, instead of having to manually write calls to focus().

The response from the endpoint handling the form should be an updated h1 tag, which can then be replaced by a form when it’s clicked, and so on.

Not renaming a board

Letting users rename a board is not that difficult, but what if they change their minds, or if they clicked the title by accident? Canceling the editing process should be as easy as clicking somewhere outside the input field, or pressing Tab to focus on something else.² At first glance, this looks like something you definitely need JavaScript for, but we actually don’t. Combining JavaScript events with the various htmx attribute will do the job.

When you press Tab, the focus moves to the next element, firing an event on the original element which JavaScript calls blur. Get it? Because it’s the opposite of focus? JavaScript has some weird event names. Anyway, the blur event is fired if, for whatever reason, the input loses focus, and that includes clicking outside it.

<div hx-get="/boards/@Model.BoardId/name"
     hx-trigger="blur from:#boardName"
     hx-swap="outerHTML">
  <form hx-put="/boards/@Model.BoardId/editName" 
        hx-swap="outerHTML"
        hx-target="closest div">
      <input type="text"
             id="boardName"
             name="boardName"
             value="@Model.Name"
             required
             placeholder="Enter board name"
             autocomplete="off"
             autofocus />
  </form>
</div>

The trigger blur from:#boardName indicates the request should be triggered whenever a blur event is fired by the element with ID boardName. Alternatively, we could have made the text field itself make the request and replace the form, but it seemed neater to have a container element for this purpose. To be honest, that is how I originally implemented it, but while editing this article, I changed my mind. It happens; you write some code and you’re happy with it, then a month later you hate it.

Note that because we now have a container element around the form, we need to make sure we replace it as well when we submit the form. We do that by specifying hx-target="closest div", or by giving the container element an ID and using that as the target.³ I find closest to be a bit more readable, but in this case that’s a matter of taste.

This completes the implementation of the ‘click-to-edit’ pattern seen on the htmx site. Click an element to replace it with an editable version and a ‘cancel’ button, which restores it to the static version, as does clicking the optional ‘save’ button or pressing Enter.

Changing the page title

Since a board is the primary focus of the board page, you would expect it to reflect the board’s name in the page title. It would be nice if the title would update when we change its name. With htmx, this is as easy as including a <title> tag in the response.

In response to changing the title, we’ll just add it in the response, next to the new board title.

@if ((bool) (ViewData["RenderTitleTag"] ?? false))
{
  <title>@Model.Name - Htmello</title>
}
<h1 hx-get="/boards/@Model.BoardId/editName"
    hx-swap="outerHTML">
  @Model.Name
</h1>

ViewData is ASP.NET’s ad hoc view data dictionary, which is a convenient (if a bit ugly) way to pass optional data from controllers to views. This is the same partial view used to render the board name as part of a full page request, so the title tag is rendered conditionally.

And yes, ‘Htmello’ is the name I came up with for my htmx-based Trello clone.

Creating lanes

Now that we have a title, we need the ‘lanes’ that hold cards. The interactions around creating lanes are a little bit more complicated.

• The ‘add lane’ button always needs to be to the right of the last lane;
• Clicking the ‘add lane’ button needs to reveal a text field and move focus to it, and moving focus away from that needs to hide the text field and show the ‘add lane’ button again;
• Pressing Enter and finishing the interaction needs to result in a lane being added to the right of the last lane, and the ‘add lane’ button re-appearing next to the new lane;
• Just to make things a bit more interesting: lane names need to be unique, so a submission can also be rejected with a validation error.

Could this have been solved purely using htmx? Probably. Would it have been the most practical? Probably not. So, what then? Obviously, I could have used plain JavaScript to wire up some of the more subtle interactions, such as resetting the add lane button when focusing away. That would still have required a fair bit of boilerplate code to wire up the code in the first place, and to do things like DOM traversal and manipulation. Instead, I’ll use a language that is better suited for these kinds of small interactions: hyperscript, from the same guy who created htmx. In fact, it is advertised as the ‘companion of htmx’.

<sidebar>: hyperscript

hyperscript is meant to be a hypermedia-friendly scripting language. This means, among others, that the language is designed to be used in-line (i.e. in an HTML attribute). JavaScript can be used in-line, of course, but it’s cumbersome and relatively verbose. Doing something simple like showing a single element would be trivial, but doing multiple things quickly starts to become hard to read. In-line JavaScript handlers also only handle a single type of event on a single element; if you need to handle an event on multiple elements, you’ll need to duplicate the same onclick attribute (or whatever attribute you need) for every element. For this reason, it’s recommended to write JavaScript separately from your HTML and write long, verbose statements describing what you want. That, or use jQuery.

document.addEventListener("DOMContentLoaded", function() {
  var button = document.querySelector(".add-lane-button");
  button.addEventListener("click", function() {
    var form = document.querySelector(".add-lane-form");
    form.style.display = "block";
    this.style.display = "none";
  });
});

Now contrast this with hyperscript, which has event handling and DOM manipulation built into the language.

<button _="on click 
             show the first .add-lane-form
             hide me">
  Add lane
</button>

_= is hyperscript for ‘apply the following behaviors to this element’. The behavior is simple: when the button is clicked, show the first element with the class add-lane-form, and then hide the button.

Obviously, this is a very simple example and a bit contrived, but it does work, and it showcases a philosophy behind both hyperscript and htmx: locality of behavior. It is much easier to understand what behaviors actually apply to a particular element when that behavior is defined in its close vicinity.

</sidebar>

So, back to our main event. Let’s first tackle clicking the ‘add lane’ button, which we already partially solved in the sidebar. In hyperscript, this almost looks like pseudo-code.

<button _="on click 
             show next .add-lane-form 
             hide me 
             focus() on the first <input[type=text] /> in the next .add-lane-form">
  Add lane
</button>
<div class="add-lane-form" style="display:none">
  <!-- form goes here -->
  <input type="text" />
</div>

So: we have a button with some hyperscript and an invisible div with a text field inside it. The hyperscript looks similar to that of the sidebar. When the button is clicked, show the ‘next’ element with the class add-lane-form, then hide the button, and then something that looks complicated.

Let’s first address the ‘next’ bit. Like htmx, hyperscript supports relative queries, i.e. finding elements relative to the current element. next will find the next element in the DOM, which includes searching through parents and ancestors. In this case, the element is the button’s sibling, but it could be somewhere else entirely.

Then the complicated bit. We can break it down into parts by putting parentheses around those parts.

focus() on
  (the first <input[type=text] /> in
    (the next .add-lane-form))

Now it looks like more like a maths equation, and it’s solved in a similar manner; find the next element with class add-lane-form, and within that find the first input that has type="text", then call focus() on it.

So, now that we have a visible text field, let’s deal with pressing Enter and actually creating the lane. This is handled by htmx using the hx-post attribute. Let’s zoom out a bit and show the button, the form, and its container.

<div class="add-lane">
  <button _="-- omitted for brevity">
    Add lane
  </button>
  <div class="add-lane-form" style="display:none">
    <form hx-post="/boards/@Model.BoardId/lanes"
          hx-target=".add-lane"
          hx-swap="beforebegin">
      <input name="LaneName"
             type="text"
             required
             placeholder="Add a lane"/>
    </form>
  </div>
</div>

The form is the tag doing the heavy lifting here. It posts to a particular endpoint, and its target is the outermost div in this snippet, the one with the class add-lane, and its swap type is set to beforebegin. To visualize why this works, let’s look at a picture that visualizes the HTML structure.

Because of the magic of CSS Flex Box, just adding a div next to the others is enough for everything to be styled correctly. Because the <div class="add-lane"> is always at the end of the ‘strip’, beforebegin will correctly add lanes after the existing ones, and move the ‘add lane’ control to the right.

The endpoint that adds the new lane only returns the HTML for the newly added lane. At this point, we’re half-way there. After you’ve added a lane, the page now looks like this:

Validation

The text field we used to add the lane is still there, with its value. That’s okay for now; let’s focus on the ‘unhappy flow’ for adding a lane: the name is not unique, so we need to show an error message. How we find that the name is not unique is not that interesting here, but how we get the error message to show up is. If we simply return the error message to the client, it will be added to the DOM where the new lane would show up.

We need to change the effective target element, but that won’t be enough, since the swap method is beforebegin. We’ll also need to change the swap method. Fortunately, htmx supports this through some HTTP response headers. To keep things simple, we’ll just return the entire form with an error message, and we’ll change the target to be this (meaning the element that initiated the request, i.e. the form) and set the swap method to outerHTML, i.e. replace the entire element. If we send the HTTP response like the following, htmx will do what we want.

HTTP 200 OK
Content-Type: text/html
HX-Retarget: this
HX-Reswap: outerHTML

<form hx-post="/boards/1/lanes"
      hx-swap="beforebegin"
      hx-target=".add-lane">
  <input name="LaneName" value="Procrastinate" />
  <!-- and so on -->
  <span class="text-danger">Lane names must be unique.</span>
</form>

The HX-Retarget and HX-Reswap headers tell htmx to ignore the target and swap method originally set on the element making the request, and use the new ones instead. This is a nice mechanism for different flows on your page, which could be used for validation, but also for authentication, for example.

Hiding the field

Now that we have validation in place, let’s circle back to the previous problem: the text field is still visible, while we want it to disappear and the ‘add lane’ button to reappear. There are several ways we can tackle this problem.

Instead of adding only the newly added lane, we could replace the entire strip of lanes, including the ‘add lane’ button. That’s not very efficient, and it does not work for another requirement, namely hiding the text field and showing the button when we move focus out of the text field. Let’s call this plan C.

We could trigger a request after we’ve added a lane to replace the ‘add lane’ form. This might be a simple solution, but it also introduces a visible ‘flicker’ between the moment the lane is visually added and the text field is replaced with a hidden one. This happens even with blindingly fast back-ends on a local machine, so imagine how distracting it would be on a spotty 3G connection.

We could use out-of-band swapping to add the new lane and replace the form in a single action, but this is not super clean, because it requires a switch in the back-end logic to add the hx-swap-oob attribute or not, depending on the situation.

Perhaps the best solution is to use hyperscript to handle these events. Responding to the user moving the focus away is easy enough; we just hide the form, show the button, and then we reset the text field’s value to an empty string and we hide any validation errors.

<input name="LaneName"
       type="text"
       required
       placeholder="Add a lane"
       _="on blur
            hide <div.add-lane-form />
            show <button.add-lane-button/>
            set my value to ''
            hide the next .text-danger"/>

This is pretty straight-forward. However, doing this when a lane has been added seems less obvious. What kind of event can we handle that would indicate this? There’s the htmx:afterOnLoad event from htmx, but that will fire for any successfully completed request, including one where the response shows a validation error. There’s the htmx:afterSettle event, which fires on the <div class="add-lane"> when a lane has been added. This is fired very late in the process, though, and that leads to flickering, the thing we were trying to avoid. In addition, it’s pretty brittle; if we change how we add lanes, we could also end up needing to change this.

The proper solution is having the server trigger a custom event whenever a lane has been added. This is done using response headers: just put the name of an event in the HX-Trigger header, and htmx will trigger that event when the request completes successfully.

HTTP 200 OK
Content-Type: text/html
HX-Trigger: laneAdded

<!-- HTML goes here -->

These ‘custom events’ are proper JavaScript events, so they can be used to trigger htmx requests, or hyperscript blocks, or anything else that responds to JavaScript events. In our case, we would just add it to the existing on clause, changing it from on blur to on blur or laneAdded from body. The from qualification is needed because htmx triggers custom events on the body element.

<input name="LaneName"
       type="text"
       required
       placeholder="Add a lane"
       _="on blur or laneAdded from body
            hide <div.add-lane-form />
            show <button.add-lane-button/>
            set my value to ''
            hide the next .text-danger"/>

And that’s it, that is all of the behavior described at the start of this section covered: toggling a button into a text field that reverts when you cancel the operation or when you complete it successfully. For a complete view, this is all of the HTML that makes this happen:

<div class="add-lane">
  <button class="btn add-lane-button"
          _="on click 
               show next .add-lane-form 
               hide me 
               focus() on the first <input[type=text] /> in the next .add-lane-form">
    Add lane
  </button>
  <div class="add-lane-form"
       style="display: none">
    <form hx-post="/boards/@Model.BoardId/lanes"
          hx-swap="beforebegin"
          hx-target=".add-lane">
      <input name="LaneName"
             type="text"
             required
             placeholder="Add a lane"
             _="on blur or laneAdded from body
                  hide <div.add-lane-form/>
                  show <button.add-lane-button/>
                  set my value to ''
                  hide the next .text-danger"/>
    </form>
  </div>
</div>

Could we have used this for the editing of the board name as well? Yes, we very well could have, and that is what is in the final version of the code (as of this writing). It replaces 6 lines of htmx attributes with 8 lines of hyperscript, merges two files together, and removes two entire endpoints. Originally, I implemented it as described earlier, and as I was writing this part of the article I thought of exploring other places where I don’t really need a request to just change what elements are visible.

Creating cards

Now that we have lanes, creating cards should be a walk in the park. The structure of the UI for creating a card is almost identical to that for creating a lane; there’s a button that turns into a field, which disappears and resets when you actually create a card or dismiss the action. The htmx part is set up in very similar way, although it has a minuscule difference, which has to do with reordering cards, the subject of the next section.

<button _="on click
             show next .add-card-form 
             hide me 
             focus() the first <input[type=text] /> in the next .add-card-form">
    Add a card
</button>
<div class="add-card-form"
     style="display:none">
    <form hx-post="/boards/@Model.BoardId/lanes/@Model.LaneId"
          hx-target="previous .card-stack"
          hx-swap="beforeend">
        <input type="text"
               name="cardName" 
               required 
               placeholder="Add new card" 
               autocomplete="off"
               _="on blur or cardAdded from body
                    hide the closest .add-card-form
                    set my value to '' 
                    show <button.add-card-button/>"/>
    </form>
</div>

Reordering cards in a lane

Reordering items in a list, because that is what cards in a lane essentially are, is not as hard as it sounds. We can just add up and down buttons on each item that move them around within a lane. For a better experience, we can let users select an item, and move them around with centrally located up and down buttons. Except of course that’s not what we’re going to do. Trello (and by extension Trellix, and most modern work organization tools) lets you move around cards by clicking and dragging. How are we going to do this in our htmx set-up? Events!

We’ve used events in our approach to resetting the UI when a lane or card has been added, but then we only triggered events using the HX-Trigger header, which then triggered blocks of hyperscript. Now we’re going to send requests, triggered by JavaScript events.

We’ll use the SortableJS library to deal with the heavy lifting of dragging and dropping. It’s a simple, but capable library, and most importantly, it triggers a JavaScript event whenever a card has been moved to a different position.

SortableJS requires a tiny bit of JavaScript to set up, and a container element in the DOM with some child elements to reorder. In our case, each lane has its own sortable instance, the elements to reorder are the lane’s cards, and the container is a form that will be submitted whenever a card has been moved. Why a form? Because the way we encode the new sort order is by sending the IDs in a particular order.

Let’s say we have three cards with IDs A, B, and C. We’ll give each card a hidden input field with its ID and a common name. When you submit a form, the fields are added in tree order, meaning the order in which they appear in the DOM. In other words, if we add a hidden field called cards to each card, with its ID as the value, and we submit the form containing everything, the form data submitted will be this:

cards=A&cards=B&cards=C

When you reorder elements using SortableJS, they are simply repositioned in the DOM, which means the order in which the IDs appear in the form data will reflect the new order. SortableJS has many different events for different parts of the drag-and-drop cycle, but the sort event is triggered for ‘any change to the list (add / update / remove)’, which make it an ideal candidate, for now.

To put this into practice, let’s first configure the SortableJS library. After adding a script tag to the document, we can do this:

htmx.onLoad(function(content) {
    const sortables = content.querySelectorAll(".card-stack");
    sortables.forEach(sortable => {
        Sortable.create(sortable);
    });
});

In my example repository, there’s a few more lines of code to enable animations and to prevent sorting whenever there’s a request in-flight, but this is definitely functional.

Now we just need to add the cards and the form.

<form class="card-stack"
      hx-put="/boards/@Model.BoardId/lanes/@Model.LaneId/sortItems"
      hx-trigger="sort">
  @foreach (var card in Model.Cards)
  {
    <div class="lane-card">
      <input type="hidden" name="cards" value="@card.CardId" />
      <div class="card-text">@card.Title</div>
    </div>
  }
</form>

Now we just need to implement an endpoint that takes the card IDs in the new order, and uses that information to sort the cards in the application’s persistent storage. In this case, we don’t need a response, because nothing would change. If you would need to deal with concurrent updates, you might need to replace the items in the DOM after sorting them.

Moving cards between lanes

In the previous section we handled reordering items within a lane. But one of the USPs of Trello is being able to drag items to a different lane to indicate its progress. Obviously, we need to be able to do this. Fortunately, SortableJS supports this scenario, which they call ‘groups’. If we specify the same group name when setting up each sortable instance, we can freely move items between them.

htmx.onLoad(function(content) {
    const sortables = content.querySelectorAll(".card-stack");
    sortables.forEach(sortable => {
        Sortable.create(sortable, { group: "cards" });
    });
});

The only problem is that, because our htmx trigger is the sort event, moving a card from one lane to the other will trigger two requests; one for the lane the card is added to, and one for the lane the card is removed from. The second request will achieve nothing, so we’d rather not have it. We can use more specific events instead:

• add for when an items is added to a lane (from another one);
• update for when an items order within the lane is changed.

Using hx-trigger="add,update" will do what we want. Our endpoint implementation will need to be a little bit smarter, because the IDs it gets could be from any lane, not just the one that’s being sorted.

And voila, we can move cards every way we want, and it only took a few lines of code.

Removing cards

While removing cards sounds trivial, I thought it’s interesting to include because it uses some of the more interesting features of htmx to show how little you need for useful interactions.

One way of implementing this is to trigger a refresh of the entire lane. That would be wasteful, though, because in the end we just want to remove a single element from the DOM. The way to go is to use hx-swap="delete".

<a hx-delete="/boards/@Model.BoardId/cards/@Model.CardId"
   hx-target="closest .lane-card"
   hx-swap="delete"
   hx-params="none">
  Delete
</a>

When the DELETE request completes successfully (and returns a 2xx-class response), the target element (in this case: the entire card) is removed from the DOM. hx-params is there because the cards are all contained within a form, and by default, htmx will then send all of the form’s fields with the request. While it’s not a problem in this case, it’s better to send a correct request, and specifying hx-params="none" will do just that.

Card counting

This feature is not a part of Trellix, but even more interesting than removing cards, because it showcases updating multiple parts of the page. We’ll add a badges to indicate how many cards are in each lane, and on the board in total, and to do this we’ll be using several mechanisms.

A board with several lanes, each have one or more cards. Each lane has a badge showing the number of cards in the lane. The board title has a badge showing the total number of cards on the board.

To show the overall total number of cards on the page, we’ll use out-of-band swaps, since there is only a single element to update (and so we can show what that looks like). There are only two scenarios that affect the overall total: creating a card and removing a card.

Next to the name of the board, we’ll add a badge with an ID.

<span class="badge bg-primary"
      id="boardCardCount">
  @Model.CardCount
</span>

In my example repository, this is implemented using a view component, which is kind of a ‘lightweight controller’; while it is a component that’s invoked by the view, it has its own ‘controller’ that can execute code and that can be tested independently from its view. This means I don’t need to query the new total number of cards in my controller action that creates a new card, nor do I need to do this in a view.

When a card has been created, I add this badge element next to the newly created card, but with a hx-swap-oob="true" attribute. The value of the hx-swap-oob attribute can be true or an hx-swap value, optionally with a CSS selector. If set to true, htmx will look up the target element by its ID and replace its contents.

When a card is removed, the response consists only of this badge, with hx-swap-oob="true".

Lane card counts

In addition to creating and removing cards, reordering or moving cards can affect the number of cards in each lane. Each lane will have its own count, so in this case, we’ll have each element update itself in response to certain events.

We could trigger the cardAdded event when we create a card, but that would cause each lane’s card count element to refresh itself, which is a bit of a waste. Instead, we can trigger an event with the lane ID added (i.e. cardAdded:12345), and only have each element respond to events within its own lane.

<span class="badge bg-primary"
      hx-get="/boards/@Model.BoardId/lanes/@Model.LaneId/cardCount"
      hx-trigger="cardAdded:@Model.LaneId from:body"
      hx-swap="innerHTML">
  @Model.Cards.Count
</span>

We can do similar things when a card is removed and when cards have been sorted. The last case is slightly more complicated from an implementation point of view, since we’ll need to keep track of which lanes we affect when we sort items.

Note how the swap method is innerHTML (which it is by default, but I added it to be explicit). This means the response to refreshing the card count is literally just a number with the text/html content type, and that’s enough.

Before we leave: Alpine.js

A coworker who I asked to review this article said he found hyperscript’s syntax to be confusing and that he would prefer something a bit more ‘like JavaScript’. I briefly investigated what it would look like to use Alpine.js instead of hyperscript.

Like hyperscript, Alpine.js is very hypermedia-friendly, and can be ‘installed’ by adding a single script file to the page. Unlike hyperscript, it styles itself as a JavaScript framework, rather than a language.

Where hyperscript is event-driven but procedural, Alpine.js is ‘reactive’, which means there is a lot of magic happening, and when it doesn’t magic, figuring out what’s wrong could be a tall order.

That said, let’s look at what editing the title would look like using Alpine.js. First, as a reminder, let’s look at the hyperscript version. I’ve trimmed it down a bit for brevity’s sake.

<h1 _="on click
         hide me
         show the next <form />
         focus() on the first <input[type=text]/> in the next <form />">
  Title
</h1>
<form style="display: none">
  <input value="Title"
         _="on blur
              reset() the closest <form />
              hide the closest <form />
              show the previous <h1 />" />
</form>

In Alpine.js, everything is a ‘component’. To create a component, we need an enclosing element and add an x-data attribute to it. This also defines the initial state of the component. The main functionality we’re looking for is toggling between two elements, so we can represent that with a boolean.

<div x-data="{ edit: false }">
  <h1>
    Title
  </h1>
  <form>
    <input value="Title" />
  </form>
</div>

To show or not show elements, we can use the x-show attribute, which lets you declare, in JavaScript, when an element should be visible.

<div x-data="{ edit: false }">
  <h1 x-show="!edit">
    Title</h1>
  <form x-show="edit">
    <input value="Title" />
  </form>
</div>

The h1 is visible when edit is false, and the form is visible when edit is true. Now to set it to true when you click the h1, and to false when you focus away from the input. We can do that by handling events with x-on attributes.

<div x-data="{ edit: false }">
  <h1 x-show="!edit"
      x-on:click="edit = true">
    Title</h1>
  <form x-show="edit">
    <input value="Title"
           x-on:blur="edit = false" />
  </form>
</div>

Here, x-on:click means ‘when clicked, execute this JavaScript’. You can guess what x-on:blur does.

Now we just need the finishing touches: focusing the text field when we start editing, and resetting its value when the user cancels the edit. Because the x-on attribute contains JavaScript, we can just add a semicolon and add more statements. But how do we focus on the text field from the h1’s event handler? Enter the x-ref attribute; it lets you give elements addressable names within the scope of a component. The way you access them is through the magic⁴ $refs property.

<div x-data="{ edit: false }">
  <h1 x-show="!edit"
      x-on:click="edit = true; $refs.input.focus()">
    Title</h1>
  <form x-show="edit"
        x-ref="form">
    <input value="Title"
           x-on:blur="edit = false; $refs.form.reset()"
           x-ref="input" />
  </form>
</div>

I appreciate that this is somewhat easier to understand, particularly the $refs part. Does it work and fit with htmx? Yes, absolutely. I personally prefer hyperscript for its procedural approach, but if you like the likeness to JavaScript and the reactiveness, this might be a better fit.

Conclusion, reflection

And there you have it: a clone of Trellix, which is itself a clone of Trello, implemented mostly using htmx, with a sprinkling of JavaScript and hyperscript where it makes sense. I’ve certainly learned a thing or two about htmx, hyperscript, Alpine.js, and even HTML and CSS.

The original premise of this article was to respond to the statement that it would be ‘so hard’ to build something like Trellix using htmx, and I hope I’ve managed to dispel that claim.

I don’t think it’s possible to truly compare the codebase to that of Trellix, because there are too many differences, but I will say that I like the fact that my implementation contains a total of 24 lines of JavaScript (excluding libraries, of course) and 25 lines of hyperscript.

Now, while my version does not look as slick as Trellix, there’s a surprising amount of polish considering there are only 116 lines of CSS. Of course it’s leaning quite a bit on the shoulders of Bootstrap, but still, I’m impressed with the state of CSS in 2024.

Regarding the build process, the project file is 12 lines long, and literally the only change from the output of dotnet new was to add Khalid Abuhakmeh’s tiny Htmx.Net package, which adds some convenience methods to work with request and response headers.

What about…

I’m sure there are a lot of details of Trellix’ implementation that my version doesn’t capture; animations and such, or visual refinements. That was not the point of this article. I don’t want to be too hand-wavy about such details, but in my opinion, they fall into one of these buckets:

1. They are easily implemented by using HTML and CSS, or a light sprinkling of hyperscript or (if you must) JavaScript.
2. They’re cumbersome or impossible to implement (although the latter is improbable).

For the use case of Trellix, details that fall into the second bucket are generally not really that important for the customer experience, and certainly not worth the trouble of setting up a whole SPA application.

Almost four decades ago, Fred Brooks, software engineer and writer of The Mythical Man-Month, wrote:

There is inherently no silver bullet.

He was writing about complexity in the building of software, and how there is no conceivable way to make the process go significantly faster. He also classified two kinds of complexity encountered when building software: essential complexity, which is inherent to the problem the software is trying to solve; and accidental complexity, which is complexity introduced by the choices we make in the development process. An example of accidental complexity is that of reusability; you don’t want to write the same piece of code twice, because you don’t want to have to make changes in two (or more) places when the piece of code needs changing.

The term ‘silver bullet’ is also used derogatorily to refer to the fact that there is no single technology or piece of software that is universally applicable in all situations, or even more than a handful. The ‘silver bullet’ technology is a dream that lingers stubbornly in the minds of developers; who wouldn’t want to stumble upon that one technology that magically solves all of your problems? However, as Fred Brooks has eloquently stated, there is no such thing. The industry seems to struggle to learn this.

Candidates for the ‘silver bullet’ often focus on accidental complexity, trying to make the developers’ lives easier, but as another evergreen phrase says, there’s no such thing as a free lunch, meaning everything comes with a cost or a trade-off. In my opinion, nowadays, the cost often being paid is that of simplicity; while a particular solution makes it easier to reuse code, it might also make it more complicated.

Front-end complexity

Front-end development is rife with examples of this. The engineering culture among many front-end developers seems to be one where simplicity is a dirty word. There are towering levels of abstraction, comically complicated and long build processes, bad performance, horrible tooling, and buggy applications, and everybody seems fine with this.

Let’s look at an example. We want to add search functionality to our site. Using nothing more than HTML 5, we can create a form with a text field.

<form method="GET" action="/search">
  <input type="search" name="query" placeholder="Search" />
  <input type="submit" value="Search" />
</form>

This is very basic, but it gets the job done. When you type something in the field and press Enter or click the Search button, your query is sent to the server and the response, hopefully containing some search results, replaces the entire page. The fact that it replaces the entire page can be jarring. When you have a slow or bad connection (which everyone who’s ever used hotel or train WiFi can confirm is still a thing) the page temporarily stops being interactive while blocking resources are being loaded. When it finishes loading, you’ll have lost all state on the page, such as scroll position, data entered in form fields, collapsed or expanded elements, and so on.

Changing only a part of the page to display the search results would result in a much better user experience. Unfortunately, this is not part of HTML. JavaScript can help us out here, surely.

Vanilla JavaScript

Let’s just add a <script> tag with some ‘vanilla’ JavaScript code to make the process asynchronous. Naturally, we’re going to request some JSON from a ‘REST endpoint’. Not GraphQL. That would be over-engineering.

<input type="search" id="searchInput" placeholder="Search...">
<button id="searchButton">Search</button>
<ul id="resultsList"></ul>

<script>
  const resultsList = document.getElementById('resultsList');
  const searchInput = document.getElementById('searchInput');
  const searchButton = document.getElementById('searchButton');

  searchButton.addEventListener('click', handleSearch);
  searchInput.addEventListener('keypress', function(event) {
    if (event.key === 'Enter') {
      handleSearch();
    }
  });

  async function handleSearch() {
    const searchTerm = searchInput.value.trim();
    const encodedSearchTerm = encodeURIComponent(searchTerm);
    const apiUrl = `/search?query=${encodedSearchTerm}`;
    resultsList.innerHTML = ''; // Clear previous results

    const response = await fetch(apiUrl);

    if (!response.ok) {
      throw new Error('Network response was not ok');
    }
    const data = await response.json();

    if (data.length === 0) {
      resultsList.innerHTML = 'No results found.';
    } else {
      data.forEach(result => {
        const listItem = document.createElement('li');
        listItem.textContent = result.title;
        resultsList.appendChild(listItem);
      });
    }
}
</script>

Yikes. There is so much more here than in the HTML version. A lot of the code is focused on translation. Translation between the raw search query and a URL. Translation between the (assumed) JSON response and DOM elements. Look at how much we need to do; grabbing and creating DOM elements, adding event handlers, handling keystrokes…

This is a very limited example with many problems:

• The representation of search results is so basic it’s not even a link;
• It works pretty well when you have awesome network conditions (say, between your browser and your local host), but when you’re using hotel WiFi, it would be nice if there was some sort of loading indicator;
• If you initiate a new request while another one is still in flight, it would be nice if the older one is canceled or at least discarded.

Note that there is an implied contract between this code running in the browser and the code on the server: the shape of the response. The code assumes the response is JSON containing an array of objects, where each object has a property called title. If that is somehow not the case, it will appear to the user as if nothing happened (because there is no error handling).

Dealing with the loading indicator and aborting in-flight requests takes a bit of extra code, but having readable code for the layout of results is a stretch. Let’s just pull out all the stops and use React!

React

import React, { useState, useEffect } from 'react';

function Search() {
  const [searchTerm, setSearchTerm] = useState('');
  const [results, setResults] = useState([]);
  const [loading, setLoading] = useState(false);
  const [abortController, setAbortController] = useState(null);

  const handleSearch = async () => {
    if (abortController) {
      // Abort the previous request if there is one
      abortController.abort();
    }

    if (searchTerm.trim() === '') {
      setResults([]);
      return;
    }

    const encodedSearchTerm = encodeURIComponent(searchTerm.trim());
    const apiUrl = `/search?q=${encodedSearchTerm}`;

    setResults([]);
    setLoading(true);

    const controller = new AbortController();
    setAbortController(controller);

    try {
      const response = await fetch(apiUrl, { signal: controller.signal });

      if (controller.signal.aborted) {
        // Request was aborted, so no further action is needed
        return;
      }

      if (!response.ok) {
        throw new Error('Network response was not ok');
      }

      const data = await response.json();
      setResults(data);
    } catch (error) {
      if (error.name !== 'AbortError') {
        // Display an error message only if it's not an abort
        console.error('Fetch error:', error);
      }
    } finally {
      setLoading(false);
    }
  };

  const handleKeyPress = (event) => {
    if (event.key === 'Enter') {
      handleSearch();
    }
  };

  useEffect(() => {
    return () => {
      // Cleanup function to abort the request if the component unmounts
      if (abortController) {
        abortController.abort();
      }
    };
  }, [abortController]);

  return (
    <input
      type="search"
      onChange={(e) => setSearchTerm(e.target.value)}
      onKeyPress={handleKeyPress}
    />
    <button onClick={handleSearch}>Search</button>
    {loading && <div>Loading...</div>}
    <ul>
      {results.map((result, index) => (
          <li key={`result-${index}`}><a href={result.href}>{result.title}</a></li>
      ))}
    </ul>
  );
}

export default Search;

Look at how massive this is becoming. To be fair, this isn’t even that much bigger than the pure JavaScript version. There’s a bunch of state declaration and handling, some error handling, and there’s a few lines of additional logic around the logic of aborting requests. I’d like to point out two things, though.

First, we now have a lot more state to manage, including some state that needs to be cleaned up when the component is removed from the DOM (or ‘unmounted’, in React-speak); that’s the useEffect call near the end.

Second, unlike the two earlier solutions, you can’t just plonk this in a file, access it using a <script> tag, and expect it to work. Essentially, you need to transpile this kind of JavaScript file into an actual JavaScript file. To be able to do that you need to set up a React application. This means running an NPM script that sets up a complicated directory structure, which is configured so you can run a command-line application to start a web server, which can then serve your React application. Even after you do a so-called production build, you’re left with an HTML file with placeholders and script tags and a whole bunch of JavaScript files called chunks, which contain tiny parts of your application and the required third-party libraries it needs. All the parts only make sense at run-time, where they hopefully all combine and work.

And for all of its complexity, this version of the search box still has limitations. It doesn’t support incremental search, which is where searching is continually performed while you’re typing, without losing focus or having to click a button. In web applications this is generally not done on every keypress, but only when the user stops typing for a fraction of a second. Another problem is that when you have searched, the URL of the page is not updated. This means it’s not possible to share your search query, bookmark it, or open it in a new browser tab. This flaw is shared by the pure JavaScript version, but not by the simple HTML version because that simply navigates to a different URL.

While it is certainly possible to overcome all of these issues in either the pure JavaScript or React versions of the code, we all understand the code would become even more complicated. Instead, let’s look at an approach that, instead, focuses on the HTML side of things, without piling on more and more JavaScript.

Htmx

Htmx is a tiny JavaScript library that lets you add Fetch functionality to HTML elements by using nothing more than HTML attributes. You can let a standard <button> element make an HTTP request when it’s clicked and handle the response by just a few attributes.

<button hx-post="/transfer/all?to=randomCharity" 
        hx-target="#thanks">
  Empty my bank account
</button>
<div id="thanks"></div>

The hx-post attribute here defines that when that button is clicked, a POST request is made to that URL. The hx-target attribute defines where to display the response as a CSS selector; in this case in an element with an id attribute equal to thanks. So how does htmx know what HTML elements to create from the response? A good question with a simple answer: the response is HTML. That way, your browser does not need to run code to transform the data it receives from a back-end system into DOM elements—it just receives HTML, which it already knows how to deal with.

Hypermedia

Unlike most SPA frameworks, htmx is not trying to run an entire application in the browser, but is instead trying to keep the browser ‘dumb’. It does this by utilizing a decades-old concept called hypermedia. What’s hypermedia again? According to the book Hypermedia Systems, by the authors of htmx:

Hypermedia is a media, for example, a text, that includes non-linear branching from one location in the media to another, via, for example, hyperlinks embedded in the media.

The crucial bit here is embedded in the media: a hypertext (a text that is hypermedia, such as HTML) contains all the information about possible interactions encoded within the text. Think of a search result page in Google a couple of years ago; you knew there were more pages available because there was a ‘next page’ link. There’s no need for you or your browser to understand what the URL of the next page is: that is embedded in the HTML your browser received from Google. If you change your URL structure, you only need to change it in one application. If you add a new feature, say navigating not to the next page but the one five pages over, it is as simple as adding another hyperlink.

REST, done poorly

For those of you who have been developing Web APIs for a while, this might cause a few bells to ring. A concept that is still often heard when talking about Web APIs is HATEOAS: Hypertext As The Engine Of Application State. This is a constraint for RESTful APIs that is derived from a few disparate concepts in Roy Fielding’s dissertation on Representational State Transfer. These concepts are stateless communication and self-descriptive messages, and the idea that clients should be able to navigate and interact with applications using only the information provided by the server, without requiring hard-coded knowledge or out-of-band information. This is precisely the thing that hypermedia solves.

REST, or Representational State Transfer, was originally conceived as an architectural style for distributed applications. Nowadays ‘REST’ usually refers to a JSON API that has an opinion on how URLs are constructed and that uses more verbs than GET and POST. The thing is: JSON, in and of itself, does not facilitate hypermedia controls (elements that allow for this non-linear branching), so most JSON APIs are not, in fact, RESTful.

Efforts have been made to make JSON APIs more RESTful by adding links to documents. One of the most familiar ones is HAL, or Hypertext Application Language. It adds a well-known property to all JSON documents and even within elements of a document. Imagine the response to our search operation looking like this:

{
  "_links": {
    "self": "https://example.org/api/search?q=complexity&p=4",
    "first": "https://example.org/api/search?q=complexity",
    "prev": "https://example.org/api/search?q=complexity&p=3",
    "next": "https://example.org/api/search?q=complexity&p=5"
    "last": "https://example.org/api/search?q=complexity&p=42"
  },
  "_embedded": {
    "results": [
      {
        "_links": {
          "self": "http://www.cs.unc.edu/techreports/86-020.pdf"
        },
        "title": "No Silver Bullet: Essence and Accidents of Software Engineering" 
      },
      // ...
    ]
  }
}

This describes how to get to the first page, the next page, and how to navigate to each of the search results. It does not, for example, describe which page you’re on, how many pages there are in total, or how to get to the seventeenth page. Yes, you could add URLs for all of the pages, but now you need hard-coded knowledge in the client to deal with these links. At least first, prev, next, and last are kind of standardized names of links, but to provide links to arbitrary pages you’d need to create a custom scheme, something like page:17. Which needs more hard-coded knowledge for it to be usable.

Another solution would be to add a link ‘template’ that describes how the URL for a particular page is constructed. Something like this:

{
  "_links": {
    "template:page": "https://example.org/api/search?q=complexity&p=%d"
  }
}

Now you need hard-coded knowledge in the client telling it to look for this link and to replace the placeholder %d with the page number it wants.

And the limitations go on: imagine how you would describe something as trivial as a search form in this fashion. You’d need all kinds of placeholders and magic values that need to somehow mean something to the client application. Truly RESTful JSON APIs are sometimes possible, but they’re very limited and brittle.

REST, done proper

HTML, on the other hand, is true hypermedia, so it’s almost trivial to create a RESTful application using only HTML. HTML has its limitations, as we discussed, and we can overcome most of those using a library like htmx. It’s easy to imagine what our search results would look like in HTML.

<ul>
  <li><a href="http://www.cs.unc.edu/techreports/86-020.pdf">No Silver Bullet: Essence and Accidents of Software Engineering</a></li>
</ul>
<nav>
  <ul>
    <li><a href="/search?q=complexity">&laquo; First</a></li>
    <li><a href="/search?q=complexity&amp;p=3">3</a></li>
    <li>4</li>
    <li><a href="/search?q=complexity&amp;p=5">5</a></li>
    <li><a href="/search?q=complexity&amp;p=42">Last &raquo;</a></li>
  </ul>
</nav>

When this is rendered by a browser, even without any styling, it is fully functional. The hyperlinks just work, because they are self-descriptive. If we want to add a search box to be able to manually enter the page number, we can do just that, without even getting fancy. We just add the following fragment into the <nav> element:

<form method="get" action="/search">
  <input type="hidden" name="q" value="complexity" />
  <input type="number" name="p" min="1" max="42" />
</form>

How simple! And because of the power of HTML, this just works. You enter a number, press Enter, and it goes. Hidden fields are used to transfer all of the state of the application, and it’s up to the browser to render the correct user interface elements and make them behave appropriately. Authentication is done the same way it is for viewing any page because it is a page. Caching works out of the box, assuming you’ve set your response headers correctly.

The remaining problem is that this still refreshes the entire page, and that causes a loss of state. Let’s remedy that using htmx.

The power of attributes

As we touched on before, htmx enhances HTML by defining new attributes that give each element the power to trigger Fetch requests, which result in elements being swapped in. These attributes let you define, among others:

• How and where to make the request; this is done by attributes like hx-get, hx-post, hx-put, hx-delete, and so on.
• What to swap in the page; primarily, this is directed by hx-target. hx-swap lets you define how to swap the content; should the inner HTML of the target be replaced, or the outer HTML (i.e. replace the entire element), or should it be added inside or after the target?
• When to make the request. The default trigger depends on the element. For example, text boxes and drop-down lists are triggered by their value changing, forms are triggered by the form being submitted, and everything else is triggered by a click. You can use hx-trigger to specify a different event to trigger on, or even multiple ones, and add modifiers, such as delays or only triggering when the value has changed.
• Which part of the response to use. By default, everything is used, but if you only want to use a specific part of the response, you can use hx-select.
• Whether to push a URL into the browser’s history, replacing the URL in the address bar. This way your application state is easily bookmarked or shared. hx-push-url will let you push either the request URL or a custom URL into the browser history.

There are a bunch more attributes (though not a ton), but these are the core ones that will solve most problems. So how can we apply this to, for example, a pagination link?

Let’s imagine we don’t change anything about the way our application renders pages. That means /search?q=complexity will return a full HTML page with the first page of results, and /search?q=complexity&p=2 will return a full HTML page with the second page of results. We’ll put the search results into a <section id="results"> element because that gives us an easy target. This is what an individual page link looks like:

<li>
  <a hx-get="/search?q=complexity&amp;p=3" 
     hx-select="section#results"
     hx-target="section#results"
     hx-swap="outerHTML"
     hx-push-url="true">3</a>
</li>

These five attributes define the following behavior:

• hx-get: When this link is clicked (or otherwise activated), make a GET request to /search?q=complexity&p=3.
• hx-select: From the response to that, select a <section> element with the ID results.
• hx-target: Swap that into the <section> element with the ID results currently on the page.
• hx-swap: Replace the outer HTML of the target. In other words, replace the entire element.
• hx-push-url: Push the URL /search?q=complexity&p=3 into the browser history and the address bar.

And that’s it! Now the pagination links replace only the part of the page they affect. We’ve had to write exactly zero JavaScript for it, and our application is still RESTful.

But wait, it gets even better. Htmx supports progressive enhancement. This means that you can start with basic, HTML-only functionality, and only if the browser allows JavaScript and the htmx library has been loaded, will its full functionality be enabled. This also makes it easier for search engines to understand your page structure. It looks like this:

<li>
  <a href="/search?q=complexity&amp;p=3" 
     hx-boost="true"
     hx-select="section#results"
     hx-target="section#results"
     hx-swap="outerHTML">3</a>
</li>

Instead of hx-get, we have a normal href attribute, and because we’ve defined hx-boost=true on an anchor tag, htmx knows to make a GET request to the URL in the href attribute and to push the URL into the browser history.

Search in htmx

If you’re still here, kudos. Now let’s circle back to the search example we’ve been looking at before and see what that would look like using htmx.

<input type="search"
       name="q"
       placeholder="Search"
       hx-get="/search"
       hx-target="section#searchResults"
       hx-select="section#searchResults"
       hx-trigger="search, click from:#searchButton"
       hx-indicator="#searchLoading"
       hx-sync="this:replace" />
<button id="searchButton">Search</button>
<div id="searchLoading" class="htmx-indicator">Loading...</div>
<section id="searchResults">
  <!-- put search results here -->
</section>

Hopefully, you understand what the hx-get, hx-target, and hx-select elements are doing here.

hx-trigger is defining when the requests are triggered. In this case, that is on the search event, or when a click event is fired by the element with ID searchButton. Next, hx-indicator sets an element that should be shown whenever this element has a request in flight. Finally, hx-sync defines that if any new requests are triggered while there is still one in flight, the one in flight is replaced by the new one.

This is functionally identical to the React version. However, using htmx, we can also get rid of those limitations I mentioned earlier: incremental search and updating the URL in the address bar.

<input type="search"
       name="q"
       placeholder="Search"
       hx-get="/search"
       hx-target="section#results"
       hx-select="section#results"
       hx-trigger="search, click from:#searchButton, keyup changed delay:300ms"
       hx-indicator="#searchLoading"
       hx-sync="this:replace"
       hx-push-url="true" />

The only new things are an additional trigger and the hx-push-url attribute. The new trigger, keyup changed delay:300ms, means the request should be triggered on ‘key up’ events, but only when the value of the field has changed, and only when no new key-up requests have been received for 300 milliseconds. This is incremental searching, but without sending a barrage of requests to the application, which is what would happen if you sent a request for every keystroke. Finally, because hx-push-url is set to true, the final URL of the request is set as the new location in the address bar, and the previous address is added to the browser history. If you’ve ever had to implement pushing and popping browser history, you’ll recognize how simple htmx’s solution is.

HOWL

If you have a front end implemented largely in JavaScript, then the following argument is likely to have reared its head:

Let’s use JavaScript on the back end as well, so we can reuse code, and front-end developers can work on the back end as well.

I loathe this argument, for several reasons.

First of all, JavaScript. JavaScript is not a great language for anything except inside the browser. It never was and it likely never will be, unless it gets a decent standard library and ditches its type coercion and many other broken language features. No amount of developer discipline and linting can truly mitigate this. Unfortunately, we’re kind of stuck with JavaScript in the browser, but let’s not use it anywhere else if we can help it.

Second of all, reusing code between the front end and the back end. It sounds like a good idea, but a web server responding to requests uses a very different application model than code running in a browser responding to user interaction. To reuse the code almost certainly means forcing one of the two into abstractions that don’t make a lot of sense. On top of that, because of the different application models and many more differences, there’s a substantial learning curve between front-end and back-end development.

The more you invest in JavaScript on the front end, the more the pressure of this argument will grow.

Fortunately, when using htmx, you won’t need to engage in this argument, since you’ll hardly write any JavaScript on the front end. You can use any server-side language and framework you like. There’s no pressure to use JavaScript if that isn’t preferred. Is most of your existing code base in Python? Django or Flask both integrate very well with htmx. Do you like Java or .NET? Maybe Ruby or Rust? All of these and more have excellent options to create HTML-based applications, and it’s trivial to add htmx to them. This is referred to by the htmx community as HOWL: Hypermedia On Whatever you Like.

Silver bullet?

I hope you’ll agree that htmx results in a simpler solution than a SPA framework, and that is worth a lot. Carson Gross (the developer behind htmx) talks about the Complexity Budget, or in other words: where in your application you allow complexity to exist. For most applications, the front-end should be just that, the front-end. What differentiates your application from others is usually the back-end functionality. Saving the complexity budget for the back-end, rather than spending it in the front-end, makes a lot of sense.

However, as this article starts, there is no such thing as a silver bullet. Htmx does not work in all situations. When a lot of your application is graphical (like a mapping or drawing app) or when a large portion of your UI needs to be redrawn in response to user input (like a spreadsheet or word processor), then you need a JavaScript-based solution. When your app needs to be available even when the user is offline, you’ll need a JavaScript-based solution. Fortunately, the vast majority of applications are just text and images and do not suffer from these limitations. For that vast majority, htmx is likely a good solution, and probably a better one than any of the currently popular SPA frameworks.

Have a look at the htmx site, where there is a lot of good documentation, as well as a collection of essays that do a much better job of explaining the merits than this brief article.

At Coolblue, we run a lot of builds and deployments. Because we practice CD, they all run automatically. This means that after we merge a pull request, our build server can spend up to half an hour building, testing, deploying, and validating. I don’t know about you, but I do not have enough attention span to watch slowly advancing progress bars for half an hour. So, instead, when one of the build or deployment stages fails, we’d like to get notified of this. Preferably on Slack, our messaging platform of choice, although this article could apply to any mechanism. Sure, most build servers have integrations with most messaging platforms nowadays, but what if you didn’t, or don’t want to rely on those?

You don’t want to have to write try/catch blocks all over the place, so it’d have to be something that you can either inject, or just run some code when the build is finished. Some build frameworks make this very easy. For NUKE, it’s as simple as overriding the OnBuildFinished method.

class Build: NukeBuild
{
  protected internal override void OnBuildFinished()
  {
    if (!IsSuccessful)
    {
      SlackTasks.SendSlackMessage(/* omitted for brevity */);
    }
  }
}

Easy. But how do we make this reusable? We don’t want people to have to copy and paste entire blocks of code. It would be nicer if it can just be ‘plugged in’.

Targets

In my previous article, I showed how NUKE’s build components make it easy to assemble a build script using only the components that you need. Adding a new target to the system is trivial, as is wiring it up to other targets.

Since we want our target to execute after the build finishes, we can just tack it onto our ‘main’ targets, right? Note that I’m simplifying the type structure for brevity.

class Build: NukeBuild
{
  Target BuildRelease =>
    _ => _.Executes(() => { });

  Target Release =>
    _ => _.Executes(() => { });

  Target NotifyFailureOnSlack =>
    _ => _.AssuredAfterFailure() // always run, regardless of other targets failing
          .TryTriggeredBy<Build>(b => b.BuildRelease, b => b.Release)
          .Executes(
            () => 
            {
              // notify if build failed
            }
          )
}

Unfortunately, things are not that simple. The first obstacle is detecting the build has failed. IsSuccessful won’t work, because it always returns false, since the build is still in progress. You’d need to look at ExecutingTargets to see if any of them has failed.

There’s a much bigger problem, though. NUKE can execute targets in any order it wants, unless you’ve specifically told it to execute a particular target before or after another one. This means if BuildRelease itself also triggers a different target, it could end up being run after NotifyFailureOnSlack. If that target then fails, it will go unnoticed. This is clearly not the right solution.

Attributes

An aspect of NUKE that is not documented at all on the official website is build extension attributes. When you setup a NUKE project, you’ll notice that the generated Build class is decorated with a bunch of different attributes.

[CheckBuildProjectConfigurations]
[ShutdownDotNetAfterServerBuild]
[DotNetVerbosityMapping]
class Build: NukeBuild
{
}

It turns out that these attributes are derived from a class called BuildExtensionAttributeBase and that NUKE has code in place to recognize attributes that inherit from it as, well, extensions to the build class. There’s also a bunch of interfaces that represent ‘events’, such as IOnBuildFinished that, when implemented on your build or a build extension, lets that component respond to build events. OnBuildFinished should sound familiar. Let us see what this looks like.

public class NotifyFailureOnSlackAttribute: BuildExtensionAttributeBase, IOnBuildFinished
{
  public void OnBuildFinished(NukeBuild build)
  {
    if (!build.IsSuccessful)
    {
      // notify
      Logger.Error($"{nameof(NotifyFailureOnSlackAttribute)}: Build failed!");
    }
  }
}

Does it work?

╬════════════
║ Failure
╬═══

> /usr/local/share/dotnet/dotnet build DoesNotExist.csproj --verbosity Minimal
Microsoft (R) Build Engine version 16.10.1+2fd48ab73 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1009: Project file does not exist.
Switch: DoesNotExist.csproj
ProcessException: Process 'dotnet' exited with code 1.
   > /usr/local/share/dotnet/dotnet build DoesNotExist.csproj --verbosity Minimal
   @ /Users/e.heemskerk/Git/Prive/NukeTest/build

   at Nuke.Common.Tooling.ProcessExtensions.AssertZeroExitCode(IProcess process)
   at Nuke.Common.Tools.DotNet.DotNetTasks.DotNetBuild(DotNetBuildSettings toolSettings)
   at Nuke.Common.Tools.DotNet.DotNetTasks.DotNetBuild(Configure`1 configurator)
   at Build.<>c.<get_Failure>b__2_1() in /Users/e.heemskerk/Git/Prive/NukeTest/build/Build.cs:line 21
   at Nuke.Common.Execution.TargetDefinition.<>c__DisplayClass61_0`1.<Executes>b__0()
   at Nuke.Common.Execution.BuildExecutor.<>c.<Execute>b__4_0(Action x)
   at Nuke.Common.Utilities.Collections.EnumerableExtensions.ForEach[T](IEnumerable`1 enumerable, Action`1 action)
   at Nuke.Common.Execution.BuildExecutor.Execute(NukeBuild build, ExecutableTarget target, IReadOnlyCollection`1 previouslyExecutedTargets, Boolean failureMode)

Repeating warnings and errors:
MSBUILD : error MSB1009: Project file does not exist.
ProcessException: Process 'dotnet' exited with code 1.
   > /usr/local/share/dotnet/dotnet build DoesNotExist.csproj --verbosity Minimal
   @ /Users/e.heemskerk/Git/Prive/NukeTest/build

═══════════════════════════════════════
Target             Status      Duration
───────────────────────────────────────
Failure            Failed        < 1sec
───────────────────────────────────────
Total                            < 1sec
═══════════════════════════════════════

Build failed on 07/03/2021 14:10:17. (╯°□°）╯︵ ┻━┻
NotifyOnFailureAttribute: The build failed!

Error messages

While this works, the message isn’t very useful. It doesn’t tell you what the failure was, or in which build it occurred. Let’s remedy that.

Unfortunately, getting error messages from NUKE isn’t that simple. Typically you’d do this by (also) writing log messages to an in-memory logger, which lets you sift through the messages it captured at your leisure. However, there is no public method or property that lets you do this; logging sinks are automatically configured by NUKE through an internal property.

You might have noticed the build output of our first attempt mentioned Repeating warnings and errors. This must mean it’s being stored somewhere, so maybe we can take advantage of that as well.

It turns out we can. The abstract class OutputSink has an internal field called SevereMessages, which is where warnings and errors are copied to as the build is running. How do you get an OutputSink? The Logger class has an internal static field called OutputSink. Given all these fields are internal, using reflection seemed the simplest approach to getting these messages out of there:

internal static class LoggerUtility
{
  public static OutputSink GetOutputSink() => (OutputSink) _outputSinkField.GetValue(null);

  static LoggerUtility()
  {
    _outputSinkField = typeof(Logger).GetField(_outputSinkFieldName, BindingFlags.Static | BindingFlags.NonPublic)
                    ?? throw new InvalidOperationException($"Couldn't find '{_outputSinkFieldName}' field on type {nameof(Logger)}.");
  }

  private static readonly FieldInfo _outputSinkField;
  private const string _outputSinkFieldName = "OutputSink";
}

internal static class OutputSinkExtensions
{
  public static string GetErrorMessages(this OutputSink sink)
  {
    var messages = (List<Tuple<LogLevel, string>>) _severeMessagesField.GetValue(sink);

    return string.Join(
      System.Environment.NewLine,
      messages!.Where(m => m.Item1 == LogLevel.Error)
               .Select(t => t.Item2)
    );
  }

  static OutputSinkExtensions()
  {
    _severeMessagesField = typeof(OutputSink).GetField(_severeMessagesFieldName, BindingFlags.Instance | BindingFlags.NonPublic)
                        ?? throw new InvalidOperationException($"Couldn't find '{_severeMessagesFieldName}' field on type {nameof(OutputSink)}.");
  }

  private static readonly FieldInfo _severeMessagesField;
  private const string _severeMessagesFieldName = "SevereMessages";
}

This should give us a relatively detailed description of what went wrong.

╬════════════
║ Failure
╬═══

> /usr/local/share/dotnet/dotnet build DoesNotExist.csproj --verbosity Minimal
Microsoft (R) Build Engine version 16.10.1+2fd48ab73 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1009: Project file does not exist.
Switch: DoesNotExist.csproj
ProcessException: Process 'dotnet' exited with code 1.
   > /usr/local/share/dotnet/dotnet build DoesNotExist.csproj --verbosity Minimal
   @ /Users/e.heemskerk/Git/Prive/NukeTest/build

[...]

═══════════════════════════════════════
Target             Status      Duration
───────────────────────────────────────
Failure            Failed        < 1sec
───────────────────────────────────────
Total                            < 1sec
═══════════════════════════════════════

Build failed on 07/03/2021 14:10:17. (╯°□°）╯︵ ┻━┻
NotifyOnFailureAttribute: MSBUILD : error MSB1009: Project file does not exist.
ProcessException: Process 'dotnet' exited with code 1.
   > /usr/local/share/dotnet/dotnet build DoesNotExist.csproj --verbosity Minimal
   @ /Users/e.heemskerk/Git/Prive/NukeTest/build

As for which build the error occurred in, the available information differs between build servers. We use TeamCity, so we can get the server URL and the build ID from NUKE’s built-in TeamCity class, which allows us to construct a direct URL to the build log of the failed build. Different CI systems might expose different information, and there might be more information available in environment variables than what is exposed by NUKE.

Conclusion

While it takes a little bit of work, you can get notifications directly from NUKE into Slack, which saves you from having to babysit your CI system each time you merge a change.

Here’s an actual notification (obviously edited):

When your application landscape expands, doing CI/CD becomes a challenge. Applications need to be built, tested, and deployed, and then tested again. Typically, each application is a bit of a snowflake in some way; it’s built in a slightly different way, or some tests need to be excluded, or publishing is more involved than with others. Maintaining CI/CD pipelines becomes a chore, and problems that arise tend to eat up a lot of time.

In this situation, it can be beneficial to simplify things a bit. In our case (that case being Coolblue), almost all applications have a three-stage build pipeline:

• Pull Request
  This is executed each time a pull request is made or updated; it builds the application and runs unit and integration tests.
• Build Release
  This is executed each time a change is merged (or pushed) to the main branch; it also builds the applications and runs unit and integration tests, but it also prepares the application to be deployed–this means packaging the application, uploading it to our deployment system, and associating some metadata with it.
• Release
  This is executed after the previous step completes successfully, and deploys the application to one of the environments. If the environment is the last one before production, it also runs automated acceptance tests.

For each stage of the pipeline, there’s only one step: execute a PowerShell script called build.ps1, which is always located in the root of the application’s repository, and pass it some strictly defined arguments. These argument always include the ‘target’, or ‘what to do’, and, depending on the stage, arguments like ‘which environment’ and so forth.

If an application needs to do something special, it can almost always be solved by just modifying this PowerShell script. The CI/CD pipelines become static and can be created at the click of a button.

Cake

Of course, only the most simplistic applications can get away with a build script as simple as dotnet publish and dotnet test. Deploying an application already involves using external tooling to be able to communicate with the deployment system. In addition, our build scripts do things beside building and testing, like:

• Adding build metadata, like when an application was built, and from which Git commit.
• Setting up secrets to be able to run integration tests against a live database, or to be able to send authenticated requests to a deployed application.
• Notifying the team when the application fails to build, tests fail, or when a deployment fails.

While this can be done in PowerShell, it’s not the easiest language to use, and not everyone is well versed in it.

Enter Cake.

For those not familiar with it, Cake is a build automation system with a C#-based DSL. It is based on the concept of tasks, which are small units of work, like restoring packages, building, or testing, which can have dependencies on, or trigger, other tasks.

For example:

Task("PullRequest")
    .IsDependentOn("Build")
    .IsDependentOn("Test");

Task("Build")
    .Does(() =>
          {
              DotNetCoreBuild("MyApplication.sln");
          });

Task("Test")
    .Does(() =>
          {
              DotNetCoreTest("test/MyApplication.Tests/MyApplication.Tests.csproj");
          });

RunTarget("PullRequest");

This will run the PullRequest task, which is dependent on Build and Test (in that order), so it will run dotnet build MyApplication.sln and dotnet test test/MyApplication.Tests/MyApplication.Tests.csproj.

So where does build.ps1 come in? Well, you can’t just run a Cake script; you need a runner. The build.ps1 script’s job is to bootstrap this. It downloads the Cake runner from NuGet and invokes the runner, which will then compile and execute the Cake script.

Bad Cake

Cake is a very nice system, but there are some problems when using it at scale, as we do.

Because Cake is not ‘pure’ C#, but rather a C#-based DSL, enhanced with some additional features, there’s no good editor support. Some editors, like Visual Studio Code, support syntax highlighting, but IntelliSense is limited to the default ‘hey I’ve seen this symbol before but it might not be appropriate in this context’ version¹. Because of this, the feedback loop is very cumbersome: edit a file, run build.ps1, get a syntax error, fix it, rinse and repeat.

It does not help that Cake’s primary building block, the task, is ‘stringly typed’. Task names are always expressed as strings. That means that if you mistype the name of a task somewhere, you won’t know until you run the build. Navigating to a particular target is also notoriously cumbersome.

Cake’s API for dealing with file system paths is well-meant, but because it is not consistently applied, it frequently means you have to convert to and from strings, and between ‘convertable paths’ and ‘actual paths’. This is a major source of compilation errors, again because there’s no proper editor support.

Finally, one of the biggest issues is that reusing code is not made very easy. You can write NuGet packages to extend Cake, but as far as I’m aware those cannot define new tasks. You can also package one or more Cake scripts into a NuGet package, but this is again cumbersome (as you have to mess about with .nuspec files), and as far as I’m aware, this also doesn’t let you define tasks.

NUKE

All of these issues were encountered relatively recently when I had to make changes to a Cake script. The way Cake scripts have evolved within our organization is that some people have invested some time into creating a single set of Cake scripts that does almost everything we want. That set of scripts was then massively copied and pasted into other projects. Over time, some scripts gained new abilities, but because there was no proper way of sharing them back to the origin or all of its clones, each build script becomes a snowflake. The result is that the mental load of modifying Cake scripts is massive, because you need to account for a specific script’s peculiarities and limitations.

It was at that point I started searching for an alternative. I quickly ran into another build automation system that I’d heard of before, but never really used: NUKE. Dennis Doomen (also known as ‘the guy who wrote FluentAssertions’) had written a very nice article outlining why he thought it was a great replacement. He outlines more reasons, but NUKE specifically addresses all of the issues I was having with Cake.

• It is ‘normal’ C# in a normal project in your solution, which means you get the benefits of IntelliSense, or Smart Completion, or whatever your editor calls it.
• Targets (the NUKE equivalent of Cake’s tasks) are regular C# symbols (either fields or properties), which means the compiler and your editor are aware of when you mistype a name and can point this out to you.
• The file system and path API is excellent, although it still looks weird when you see the ‘division’ operator being overloaded to construct paths (Root / "src" / "MyApplication" / "MyApplication.csproj").
• You can reuse code by using NuGet packages, just like in regular projects.

The Cake example from before looks like this in NUKE:

class Build: NukeBuild
{
  public static int Main() => Execute<Build>(x => x.PullRequest);

  [Solution]
  Solution Solution;

  Target PullRequest =>
    _ => _.Triggers(Build)
          .Triggers(Test);

  Target Build =>
    _ => _.Executes(
      () => DotNetTasks.DotNetBuild(o => o.SetProjectFile(Solution))
      )
    );

  Target Test =>
    _ => _.Executes(
      () => DotNetTasks.DotNetTest(o => o.SetProjectFile(Solution))
    );
}

Reusability

So how does one reuse code in a NUKE build script? Let’s say we want to reuse a target, like publishing in Release mode for the win10-x64 runtime while treating warnings as errors.

Since the build class needs to contain all the supported targets, composition is not an option. The only option left, then, is to introduce a base class.

abstract class MyBuild: NukeBuild
{
  Target Publish =>
    _ => _.Executes(() => { /* publish code goes here */ });
}

class Build: MyBuild
{
  Target BuildRelease =>
    _ => _.Triggers(Publish);
}

Because of inheritance, the Build class now exposes two targets: Publish and BuildRelease. Job done, right? Well, not so fast. What about optional features? For example, not all of our projects have automated acceptance tests. If every build exposed a target for running acceptance tests, you’d have no idea whether that target is there because of the base class or because it actually does something useful.

Of course, we could define a base class for ‘build without acceptance tests’ and ‘build with acceptance tests’, but that only works for the one feature. If you have multiple optional features that are not mutually exclusive, then you end up having to create many different subclasses for all the different combinations. If you have n number of optional features, the number of subclasses you have to create (and maintain) is 2ⁿ. For two features, that’s 4 classes; for three features, 8 classes, and for four features it adds up to 16 classes. You’ll also have a lot of code duplication, because you can’t inherit from anything (otherwise you’d end up with the same problem you were trying to solve). You can see why this isn’t a great idea.

Default interface implementations

Fortunately, NUKE’s got you covered. The solution is... (drum roll) multiple inheritance! Great, but C# doesn’t have multiple inheritance. Well, since C# 8.0, it kind of does.

C# has technically always supported multiple inheritance, using interfaces. Originally, this meant that you could only ‘inherit’ multiple contracts, but not actual behavior. That’s where default interface implementations come in. Introduced in C# 8.0, it allows any interface member to have a default implementation. An implementing class (or struct) does not need to provide an implementation for that member, although it is allowed to, in order to override the default behavior.

One of the examples of this I like, which was also used in a video where Mads Torgersen explains C# 8.0’s new features, was that of a logger. A typical logger interface has members like LogDebug, LogInformation, LogError, and so forth. It’ll usually also have something like a Log or Write method, which has the same parameters, preceded by a ‘level’ parameter. The implementation of LogDebug and similar methods is usually to call that Log method with the respective log level.

enum LogLevel
{
  Debug,
  Information,
  Warning,
  // etc.
}

interface ILog
{
  void Log(LogLevel level, string message);
  void LogDebug(string message);
  void LogInformation(string message);
  // etc.
}

class Log: ILog
{
  public void Log(LogLevel level, string message) { /* omitted for brevity */ }

  public void LogDebug(string message) => Log(LogLevel.Debug, message);
}

Every implementation of ILog needs to implement LogDebug and friends, even though that implementation will likely be exactly the same as the existing ones.

With default interface implementations, the default implementation of these methods can be provided in the interface:

interface ILog
{
  void Log(LogLevel level, string message);
  void LogDebug(string message) => Log(LogLevel.Debug, message);
  void LogInformation(string message) => Log(LogLevel.Information, message);
  // etc.
}

Now the only thing an implementation of ILog needs to implement is the Log method. This also means that you can add new methods to the interface without breaking existing implementations.

NUKE build components

So how does all of this help us with the problem of supporting optional features in our NUKE build script? NUKE calls this concept ‘build components’. It ‘allow us to organize parameters and targets in reusable build components, and freely compose our build out of them’.

In a nutshell, your Build class can ‘inherit’ from multiple interfaces, where each interface represents a ‘component’, which can have its own targets and parameters. Each interface is ultimately derived from a well-known interface (INukeBuild). NUKE understands this mechanism of defining targets, and will find targets defined on all ‘inherited’ interfaces.

Our earlier example with acceptance tests can now look like this:

interface IDefaultBuild: INukeBuild
{
  Target Publish => _ => _.Executes(() => /* publish code goes here */);
}

interface IBuildWithAcceptanceTests: INukeBuild
{
  Target RunAcceptanceTests => _ => _.Executes(() => /* code to run acceptance tests goes here */);
}

class Build: NukeBuild, IDefaultBuild, IBuildWithAcceptanceTests
{
  public static int Main() => Execute<Build>();
}

By just implementing the IBuildWithAcceptanceTests interface, the build gets a whole new set of features, whereas if you choose to not implement it, the features are not in your way.

But it gets better. In our pipeline, we run the acceptance tests as part of the Release target — first we release the application into an environment, and then we run acceptance tests. How do we combine this with the fact that acceptance tests are optional?

We can’t use Triggers on the Release target, because that would fail if the acceptance test component hasn’t been implemented. We could use TriggeredBy from the acceptance test target, but that will fail if the release component hasn’t been implemented. We don’t want to force people to implement particular components; they’re there if they are useful to you, and nothing more.

Instead, we can use TryTriggeredBy<T>. This lets you express ‘if the build is a T, this target is triggered by these targets’. This is just what we need. There are similar methods for DependsOn, DependentOn, Triggers, and even After and Before.

interface IDefaultBuild: INukeBuild
{
  Target Release => _ => _.Executes(() => /* omitted for brevity */);
}

interface IBuildWithAcceptanceTests: INukeBuild
{
  Target RunAcceptanceTests =>
    _ => _.TryTriggeredBy<IDefaultBuild>(b => b.Release)
          .Executes(() => /* omitted for brevity */);
}

Now, if your build implements IDefaultBuild and IBuildWithAcceptanceTests, the RunAcceptanceTests will automatically be triggered by the Release target. I call that pretty sweet.

Combining all of this with the excellent value-binding mechanism and all the other benefits provided by NUKE has made me switch to NUKE for good.

How do you make an application take action when something happens in another application? How do you make sure this happens reliably and in a timely fashion? In a world of service-oriented architectures and microservices, this is a hugely underestimated and overlooked problem, in my experience.

In this article, I’ll try and explain a mechanism that I’ve been working with for a while now, which I think is the simplest mechanism to reliably make applications react to one another, and which is also easily maintainable.

First I’ll describe the mechanisms typically used to make applications react to one another, and illustrate where they exhibit problems. To make my examples a bit more relatable, including the ramifications of when the system fails, let’s describe a fictional application: part of an e-commerce back-office. This will be an incredible over-simplification of how things work in the real world, but it does the job.

There are three applications:

• Ordering; this takes orders, validates them, and when they have been paid for, releases them to be fulfilled.
• Payments; this accepts and records payments from customers.
• Logistics; this makes sure products get picked, packed, and put in a box with a shipping label on it.

Obviously there is some information exchange happening between these applications. Ordering will need to know when a payment has been made, so it can mark orders as ‘paid for’. Logistics will need to know when an order has been paid for, so it can direct workers to pick and pack the products.

The payment transaction

We’ll focus on the ‘payment’ transaction. After a payment has been recorded in Payments, Ordering needs to be notified so it can evaluate whether the amount due for the order has been fully paid, after which it can ‘release’ the order to Logistics.

So how do we get this notification across?

RPC

The easy approach is to use an RPC, or Remote Procedure Call. There are many ways of doing this; you could use a TCP socket directly, or something a little more abstract, like SOAP or gRPC, or even something as simple as an HTTP request.

This is very easy to do; after a payment has been made and has been registered, Payments simply makes a request directly to Ordering, and that is that. It is done synchronously, before returning to the caller, which is likely a customer from a browser, or indirectly so, with the front-end making a call to Payments.

So what’s wrong with this? There are several problems. One of them is a high degree of coupling. Payments needs to know that Ordering is interested in this notification, and it also needs to know how it likes to receive this notification—the protocol—and how it can reach the other application. This is the kind of coupling that can slow down development; when the team maintaining Ordering wants to change some of the details, they will need to align with the team maintaining Payments, and they might need a multiple-step deployment to ensure nothing breaks. This is at odds with what is typically seen as the value of having microservices: being able to independently deploy changes.

Another problem is the need for availability. When Payments cannot reach Ordering, or an error occurs while handling the notification, what do we do then? Do we drop the notification? Do we wait and try again? We can retry, but only for some time; remember, the customer is sitting there, waiting. After a while, you have to give up. How do you handle that scenario?

If you want to notify multiple applications using RPC, pretty much the only option is to make multiple calls, which increases the coupling even further. Aside from that, how do you deal with failure? If you need to notify three separate applications, and notifying one of them fails, what do you do?

Message queues

One solution that alleviates these problems to some extent is to use a message queue. Message queues are difficult to get right, so they are usually provided in the form of an external service. This could be in the form of dedicated software, like RabbitMQ, or a service of your cloud provider, like AWS SQS and Azure Queue Storage, or even part of your database system, like SQL Server’s Service Broker or Oracle Advanced Queues.

Message queues have a well-known, fixed, API, so the only things teams will need to align on are the format of the message and the location of the queue. Queues are usually implemented in a way that makes them highly available, and because they are usually considered ‘infrastructure’, developers usually don’t have to worry a lot about keeping these services alive.

As a bonus, most queue brokers support the ‘publish/subscribe’ or ‘fan-out’ pattern, whereby a single notification is sent to multiple consumers. The publishing application only knows that it is sending a notification to a queue or a ‘topic’, nothing more. The consumers are responsible for subscribing to topics based on what they are interested in.

Note that the problems aren’t solved completely. They are simply less severe. A new problem is that both applications are now forced to use this service. This kind of coupling can make it difficult to make changes in your hosting infrastructure.

In essence, using this kind of specialized infrastructure between applications is like having multiple applications sharing the same database. There are all sorts of problems with that, like making it much more difficult to make changes, creating a single point of failure, limiting scalability, and making managing the infrastructure more complicated.

Depending on the specific queue broker you use, you might need to consider what happens if the queue broker itself experiences an outage while messages are ‘in flight’.

Atomicity and timing

Both RPC and message queues¹ suffer from a different kind of problem; that of atomicity and timing. Consider this, pretty typical, zoomed out view of what a ‘transaction’ looks like.

var transaction = BeginTransaction();
UpdateDatabase();
transaction.Commit();

// here be dragons

SendNotifications();

Note the line that says // here be dragons. The work in the database has been performed in a nice transaction, so either all of it has been performed—or, if something goes belly-up, nothing. Sending the notification, however, is not protected by anything. If an error occurs during or before sending the notification, you’re screwed; your database transaction has already been committed. Database operations can usually be reverted, but that is not always easy or desirable.

What if your application dies at that point in time? A fatal error happens, one of the cleaners unplugs the server, or the virtual machine dies. Your transaction has already been committed, but nobody knows about it. Applied to our example, this means the payment has been registered, but Ordering never hears about it, and therefore never releases the order to be fulfilled. That customer is not going to be a happy camper.

Well, we just do the notification inside the transaction, right? Unfortunately, that doesn’t magically make the notification transactional.

var transaction = BeginTransaction();
UpdateDatabase();
SendNotifications();

// here be dragons

transaction.Commit();

The point of dragons is still in the same place, but it’s now after the notification instead of before it. Why are there still dragons? Consider what happens if your application dies at that point. You’ve already sent a notification to other applications that something has happened, but the database transaction will be rolled back. In reality, nothing has happened, so you’re effectively lying to other applications.

I also mentioned that timing was a problem, and it surfaces in this order of operations. You’re telling other applications ‘this has happened’, but you haven’t committed that information to the database yet. What if the application you’ve just notified very quickly turns around and asks you for more information? Especially if the notification is synchronous, that might even be before the notification completes.

Payments might be telling Ordering ‘a payment has been made’, upon which Ordering asks ‘oh really, so how much has been paid in total?’. Because the original transaction has not yet been committed, the answer might be ‘nothing’. And yet again, we have effectively lost information and gained an unhappy customer.

Storing notifications

Because of these atomicity and timing problems, I consider sending notifications on the spot, without any back-ups, to be ‘unreliable’; there is a very real possibility a notification is lost because of a temporary disruption, like a network glitch or a VM dying.

We can make sending notifications ‘reliable’ again by making it asynchronous. Instead of immediately sending notifications, we store them somewhere, to be picked up by a background process. This is called the Outbox Pattern. If we store the notifications in a database, we can make it part of the database transaction, so either the intended changes happen and the notification is stored to be sent, or neither of them happen.

This approach has the added benefit of being resilient to temporary outages. The background process picks up the next pending notification, sends it, and then marks it as ‘done’, which can either be deleting it, or setting a flag. Rinse and repeat. If, for some reason, the notification cannot be sent, because the receiving party is not reachable or not responding, the background process backs off and tries again.

Again, this does not completely solve the atomicity problem, it changes it. Instead of not sending notifications, you might be sending them more than once. You’re still dealing with operations in two different systems—your database and another application, whether that’s an actual application or a piece of infrastructure like a queue. This can, and should, be solved by using notifications that are idempotent, which I’ll describe in a bit.

On idempotent messages

A brief interlude about what idempotent means. It derives from the Latin words idem (meaning ‘same’) and potent (meaning ‘having power’), so ‘having the same power’. It means that an operation, when performed more than once, will have the same outcome as the first time. Fine examples of this are arithmetic operations; 2 + 2 will always equal 4, no matter how often you compute it. The polar opposite would be getting a random number; that is intended to produce a different value each time.

How does this apply to messages? The go-to example to illustrate idempotent and non-idempotent messages is when a quantity of something has changed, like the quantity you have on stock of a particular product.

Consider the following two messages, describing that the stock quantity of a product has increased.

class ProductStockQuantityIncreased
{
  public string Sku { get; }
  public int QuantityAdded { get; }
}

… versus…

class ProductStockQuantityIncreased
{
  public string Sku { get; }
  public int QuantityAdded { get; }
  public int StockQuantity { get; }
}

In the first example, if you start with a quantity of zero, and you have a single message saying 5 have been added, which you receive twice, it looks like you have a total quantity of 10.

On the other hand, in the second example, you have a message saying 5 have been added and the new quantity is 5. If you receive that twice, you’ll still know the correct total quantity.

If you really want to avoid receiving a message twice, you could attach a unique ID to each message, and keeping track of which message you have already received. This comes with its own special set of problems, though, especially since RPC and most of the queueing mechanisms I talked about allow messages to be received out of order; e.g. message A, then message B, then a retry of message A. The mechanism through which you receive the message also doesn’t use the same transaction as where you keep track of which messages you’ve received¹, so you have another atomicity problem, but this time on the receiving end.

Making messages idempotent doesn’t solve any of the coupling problems attached to the RPC or message queue solution, though.

Pulling notifications (or events)

Let’s make a small change. We still store notifications, instead of sending them on the spot, but instead of pushing them to other applications, we have those applications pull the notifications from us. The notifications describe what happened, so they’re less like notifications, and more like events, which is the terminology I’ll use from now on. We create an endpoint that provides the caller a list of events it needs to process, and it is then up to that application to keep track of which ones have already been processed.

To be clear, this means Payments exposes an endpoint with a list of events to other applications. Ordering regularly polls this endpoint to retrieve any new events. In other words, Payments produces a list of events, which Ordering consumes.

How does this solve the problems we’ve seen?

Our first problem, coupling, is still there, but it’s been inverted; instead of Payments being dependent on Ordering, it’s actually Ordering which is dependent on Payments. Depending on how you implement the list of events, the coupling can also be very minimal; down to a URL and knowledge of the structure and format of events. What’s more, if you have more than one application that needs to be notified, this doesn’t increase the coupling in any single application².

The need for availability has also pretty much vanished. When Ordering can’t reach Payments to retrieve the list of events, that isn’t really a problem. Sure, orders will not be released to be fulfilled, but no notifications are lost. As soon as connectivity is restored, Ordering will pick up where it left off, and business will continue. If Ordering cannot process an event for whatever reason—maybe it cannot reach its own database—it just gives up and retries after waiting a while. The event is still there, no matter how many times you retry. This also means the process of handling events is self-recovering.

We don’t need any specific infrastructure for this solution; it involves applications communicating with other applications. The only ‘infrastructure’ we need is HTTP, and that is mature, ubiquitous, and flexible enough that it shouldn’t cause you any headaches when you make changes to your hosting infrastructure (like moving to a different cloud provider).

When Payments stores events in the same database transaction it uses to do the actual work, you’ve solved the atomicity problem. Because a notification only appears in the list of events provided by the endpoint after the database transaction has been committed, the timing problem has also been dealt with.

An additional advantage of the fact that events remain available is that you can re-process events. Let’s say Ordering maintains a private cache of how much has been paid in total for each order, by accumulating data from the events it pulls from Payments. Unfortunately, instead of adding amounts, it subtracts them. Oops. No problem; just fix the issue in code and have the application re-process all events. It can also be a tremendous advantage when implementing new functionality that uses data from the events you didn’t need previously.

Kafka?

Some of you might be thinking ‘this is very similar to a Kafka broker,’ and you’d be right, up to a certain point. Apache Kafka is a ‘distributed event streaming platform’ that uses a lot of the same ideas (an append-only list of events, clients pulling data and keeping track of where they are), but then in a distributed fashion. A lot of the smarts in Kafka is in the way they manage clusters of servers and making sure the members of a cluster agree with one another.

Kafka is, like a message queue, a stand-alone piece of infrastructure, which means it has the same ‘problems’; no atomicity³, and it forces both producer and consumer to use the same infrastructure. It is very well suited if you have to distribute a lot of data, if you have a lot of consumers, or if there are a lot of consumers. For smaller scale problems, it feels like shooting at a mosquito with a cannon.

Protocol

Now that we’ve established on a high level what the mechanism will be, let’s talk about the actual implementation of this list of events. We could, of course, just roll our own protocol. How hard could it be? It’s just a list of events. Of course, this list of events will grow larger and larger, so we’ll probably need some pagination. How does a consumer know what the URL for a given page is? What about the format of the list itself? If we don’t establish one, each team will invent their own little format, which means a consuming application will need a new implementation for each event feed they consume.

Instead, let’s pick a standardized protocol and format. We’ll have less thinking to do, and it’s very likely a client library already exists for your platform of choice. We need something that is well-suited for large collections of items and for figuring out ‘what has changed’. It should also support some mechanism indicating the different kinds of events that are stored.

Enter Atom

It turns out there’s a format that suits these requirements very nicely. It’s called Atom, and it’s most frequently used as a syndication mechanism for websites that have frequently changing content, like news sites. It supports pagination and different kinds of content (through media types). It’s XML-based and looks a little something like this:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <id>urn:publicid:PaymentHandlingEvents</id>
  <updated>2020-01-09T18:02:13.242Z</updated>
  <link rel="self" href="https://payments.myshop.eu/events/mostRecent" />
  <link rel="prev-archive" href="https://payments.myshop.eu/events/2020-01-08T0000" />
  <entry>
    <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
    <updated>2020-01-09T18:02:13.242Z</updated>
    <content type="application/vnd.myshop.payments.paid+json">
eyJQYXltZW50VHJhbnNhY3Rpb25JZCI6Mzk4MDg3MjM0Nzk4OTIsIkFtb3VudCI6MjI0LjUsIkN1cnJlbmN5IjoiRVVSIiwiUmVmZXJlbmNlIjoiMjM5ODcyOSJ9
<!-- {"PaymentTransactionId":39808723479892,"Amount":224.5,"Currency":"EUR","Reference":"2398729"} -->
    </content>
  </entry>
  <!-- more entries go here -->
</feed>

There’s a bit of ceremony in the first few lines of the XML, which are not all that interesting—yet—but let’s look at the entry tag. When using Atom as an event feed, each event will be represented by an entry element.

The entry element has an id element, containing an IRI, which in this case is a URN containing a UUID. It doesn’t have to be a UUID, per se, but it should be something that uniquely identifies the event, because this is what a consumer will use to keep track of what they’ve already seen.

The content element is where the actual payload of the event is. In order for consumers to be able to tell which kind of event this is—as there could be multiple kinds of events in a single feed—we use the type attribute and give the event a proper media type. Because the standard says any content whose media type does not start with text/ should be Base64-encoded, that’s what you see here. The giant wall of text starting with eyJQY is the Base64-encoded version of the JSON in the comment below it.

This example only contains a single event, but if there were more, they would be listed in reverse chronological order. That is, it starts with the most recent event, and ends with the oldest event.

Pagination and navigation

As I mentioned earlier, we need support for pagination, because this list of events will likely become huge. So how does pagination work in a feed like this? Given that this is the entry page of the feed, how does a consumer know how to get to the next page?

‘Next page’ is a very confusing name, by the way, because that depends on which way you’re navigating. Given our context, you always start at the entry page, with the most recent events in reverse chronological order. If you want something that’s older than the last event on the entry page, you logically want the item on the ‘next’ page.

Back to our original question: traditionally, the answer would be that you establish a ‘contract’ of sorts; if the URL of the entry page is something like /events/mostRecent, the next page could be at /events/1, followed by /events/2, and so on.

This works well, but because the consumer now needs to know about this contract, the familiar spectre of coupling rears its ugly head. Besides that, what if ‘page 1’ isn’t the next page from the entry page, but instead it’s the page with the events from the beginning of time? What if pagination isn’t based on the number of events stored, but on date and time? The document doesn’t contain any data that can be used to generate these kinds of URLs from.

Instead, the feed uses hyperdata to link to other pages, much like a normal web page would. When you google something, you don’t manually change the URL in the address bar to go to the next page; you click the ‘Next’ link at the bottom.

The document contains the following element:

<link rel="prev-archive" href="https://payments.myshop.eu/events/2020-01-08T0000" />

The rel attribute indicates what kind of relationship this link has to this document (in this case, the ‘previous page of archived data’), and the href element is simply an opaque URL that the consumer should follow to get to the next page. This way, the consumer only needs to know that the next page is indicated by a link element with a rel attribute equal to prev-archive, and the producer of the feed is free to use whatever pagination strategy they like.

But why is prev-archive an appropriate relationship? Why not prev? Or next, for that matter? From the point of view of the standard for Feed Paging and Navigation, RFC 5005, ‘previous’ is actually the term used to describe ‘older’ pages. The difference between prev and prev-archive is that prev-archive indicates that all the pages of the feed together form a complete set of entries, and that:

Archive documents are feed documents that contain less recent entries in the feed. The set of entries contained in an archive document published at a particular URI SHOULD NOT change over time. Likewise, the URI for a particular archive document SHOULD NOT change over time.

In other words, prev-archive describes the stability of the pages that are being linked to. Because ‘archive’ pages never change, this means you can have them be cached essentially forever. This becomes interesting when feeds are being used at scale.

Client processing

So how do you process this feed of events as a client application?

When you’re working with a queue, you dequeue an item from the queue, you process it, and after you acknowledge it’s been processed successfully, it’s gone. In other words, processing items changes the queue; the queue is stateful. This is not the case with the event feed; the events stay there after you process them. How do you know which event to process next? It’s like reading a book; you start with the first page, and after a while you put it down. You remember where you left off by using a bookmark.

Of course, that is where the analogy ends, because a bookmark is usually something physical that you insert in between pages of a book, but that is not something you can do with this event feed. Instead, think of it as writing down on a piece of paper which page you read last. A more accurate analogy would be that each paragraph in the book has a unique number, and you write down on a piece of paper at which paragraph you left off.

You always start with the entry page—because that’s the only page whose URL you know. Then follow these steps:

1. Try to find the entry whose <id> element matches your bookmark.
2. If you find it, then everything (including entire pages) that comes before it is new (remember, events appear in the feed in reverse chronological order).
3. If you don’t find it, navigate to the ‘next’ page, by finding a <link> element with the relationship prev-archive, read the page indicated by its href, and start from step 1.

Once you have found out what the new events are, you process them as follows:

try
{
  string? bookmark = GetBookmark();
  var events = GetNewEventsSince(bookmark);

  foreach (var @event in events)
  {
    ProcessEvent(@event);
    UpdateBookmark(@event.Id);
  }
}
catch (Exception ex)
{
  // log exception, don't rethrow
}

In this high-level code, @event.Id would contain the value of the <id> element. Obviously GetNewEventsSince contains code to navigate and parse an Atom feed and some logic to correctly deserialize events from their media types, and ProcessEvent probably uses a DI container to invoke an event handler, but otherwise, just invoke this code on an interval—and you’re done.

Why the big try/catch around the entire block of code? As I mentioned, this code will be invoked on an interval, likely a timer, and we don’t want an unhandled exception to stop that timer.

If you make sure that the code invoked by ProcessEvent uses the same transaction as that of StoreBookmark, you can achieve something pretty nice: in-order exactly-once delivery. This means that the order of events is always exactly the same as how they were stored, and you will never process an event more than once.

Of course, that is kind of a white lie, because if processing an event throws an exception, it will not update the bookmark, but at the next interval, it will definitely process the same event again. The event is, in effect, only delivered successfully once, assuming it is eventually processed successfully.

There are, as always, a lot of gotchas and subtleties around how to make this work efficiently and correctly, but I will touch on those in a separate post (probably several, actually). I will touch on one which is probably brewing in the back of your mind: latency.

Latency

In the very first paragraph I mentioned that one of the benefits was responding ‘in a timely manner.’ In its current form, this is limited to the interval at which you check for new events. What if your requirements for latency are very strict, like having responding within half a second? Obviously you can set the interval between checks to something very low, say 250 milliseconds, but this means the application hosting the event feed gets 4 requests per second, from just one application. That doesn’t scale very well.

There is a nicer approach, which scales a lot better, and which can get you latency in the order of tens of milliseconds. It involves using a backchannel to signal to clients that a new event is available. Clients then act upon that signal by checking for new events and processing them as usual. Think of it as a secondary trigger to check for new events, in addition to the timer firing on an interval.

It’s important that this backchannel only contains a notification that a new event is available, and preferrably its unique ID, but not the event itself. This is because the point of this backchannel is to be an optimization, and it can therefore be relatively unreliable. If you miss a single message, you will either pick it up, in the right order, when the next notification arrives, or when the timer fires.

This backchannel should be a push mechanism, so that clients can react as quickly as possible. It should also be client-initiated, by which I mean the client is the one initiating the request for real-time updates.

One of the options that meets this requirement is a fan-out (or ‘publish/subscribe’) queue. As I’ve said before, I don’t like to use this kind of specific (and stateful) infrastructure to communicate across application boundaries. Another option is to use a WebSocket. This lets a client connect to a well-defined URL to establish a bidirectional communication channel. SignalR is a solid option in the .NET ecosystem, for both publishing and receiving the notifications.

Example

Let’s look at an example of the payment transaction as a high-level exchange between the systems involved, in this case Ordering and Payments. I’m using numbers as unique identifiers for the events, for clarity, while you would probably use UUIDs in a real application.

Situation
Payments has a single event in its feed; a payment for order number 1. The unique ID of this event is 1. To make this sample more illustrative, it paginates based on the number of events, and its page size is 1, meaning each event gets its own page.

Ordering’s bookmark for the Payments event feed is event 1.

The transaction

1. Ordering fetches the entry page of the Payments event feed. The feed still only contains the one event, so there are no new events.
2. The customer makes a payment for order number 2, through Payments. Payments registers the payment and appends an event to its feed, with unique ID 2.
3. Ordering begins its next check of the Payments event feed. Its entry page only contains event 2. Since it doesn’t contain the bookmark event (1), Ordering requests the next page of events.
4. Ordering finds its bookmark event on the next page, meaning everything that came after it (chronologically) is new. In this case that is only event 2.
5. Ordering processes the event and marks order number 2 as ‘paid for’. It appends an event to its own event feed, to be picked up by other interested parties, like Logistics.
6. Ordering updates its bookmark for the Payments event feed to event 2.

Sample application

All of this so far has been a bit theoretical and high-level, so I will leave you with a small sample application that not only demonstrates a very simplistic event feed, but also shows off the notification backchannel. It should be a good basis for you to start playing around with event feeds yourself.

heemskerkerik/EventFeedDemo

In future posts, I will discuss error handling, reliably storing events, and scaling out, amongst others.

Previously, I wrote an article about how I got triggered after somebody on Reddit exclaimed that WCF was faster had lower response times than ASP.NET Web API and ASP.NET Core MVC.

The outcome was that, yes, with the default configuration, WCF does have lower response times. When you start tweaking with the serializers, both ASP.NET Web API and ASP.NET Core MVC come out on top.

The article kicked up a bit of a storm on Reddit, though, with people complaining that the comparison wasn’t fair, that I should have done this or that. Somebody even wrote an entire blog post about how it should be done. I wanted to address some of those comments.

Expectations

In my original article, I did not explain my intentions very well. It was not meant as a be-all-end-all determination of whether to use WCF or not. There are very valid reasons for using or not using WCF, besides the performance. I was simply trying to figure out whether WCF could have better per-request latency than the existing ASP.NET offerings.

Latency doesn’t mean better performance

Several people pointed out that performance isn’t just about latency. Specifically ASP.NET Core and Kestrel have been designed to be able to handle many requests concurrently, probably at the cost of some latency. I definitely agree with the statement, but my post was only about latency, because it is still an important metric in the performance of your service. It also happens to be one of the easier ones to measure.

You’re doing unnecessary in-place computations

Josh Bartley wrote an entire article in reply to mine. Thanks for taking the time to do that, Josh.

In the article, he mentions that I’m doing reflection in every call to construct the URI for the Web API/ASP.NET Core requests. He makes a fair point, along with some other ones, like how I was using delegates to switch between implementations. I’m not sure how much that specifically affects the numbers, but it sure is ugly. I rewrote all of the benchmarks to use a type hierarchy instead of big ugly switch statements, and I made sure to only use reflection once, during the initialization. Let’s see what has changed.

There’s a very small improvement, but it’s not very impressive. I think it’s safe to say that reflecion and using delegates was not really an issue.

Asynchronous is killing the numbers

Redditor Langebein remarked the following:

You’re doing PostAsync in the WebApi tests, while doing synchronous calls with WCF. I’d wager a lot of the difference is made up from starting tasks and spinning up threads with the fancy HttpClient.

That’s an interesting point. With my new code structure in place, I could easily add other types of clients. First, let’s see what happens if we don’t await everything.

That’s a significant difference already. But Langebein was talking about HttpClient as a whole. So let’s see what happens when we use good ol’ HttpWebRequest.

That’s a pretty big win. For all its fanciness, HttpClient is adding a lot of overhead, possibly related to its asynchronicity. For reference, I’ve added the fastest WCF result from the earlier post, and there is an undeniable advantage to ASP.NET Core and MessagePack. ASP.NET Web API does pretty well, too.

Since we’ve established that using HttpWebRequest offers the lowest latency, let’s establish a new baseline.

Now we can see MessagePack and Utf8Json (on ASP.NET Core MVC) easily overtaking WCF, and even XML on ASP.NET Core MVC is getting close.

What about binary serialization for WCF? And raw TCP?

Several people commented that I should have included more WCF options, such as binary serialization and the revered NetTcpBinding. Binary serialization was simply something I forgot about after a couple of years of not doing WCF.

NetTcpBinding is not something I think is fair to compare with. There is just an overhead to using HTTP that the ASP.NET Core and Web API tests cannot avoid. I decided to include it for argument’s sake.

Interestingly enough, the way I was invoking the WCF client meant I was creating a new client for each call. For the HTTP bindings, this was not an issue, because they’re handing off connection management to the HTTP library. NetTcpBinding has to do it by itself. The way I was doing it, I was never actually releasing clients, which meant sockets would stay open forever. I tried benchmarking it, but it never even got past the ‘pilot’ phase, where BenchmarkDotNet tries to determine the optimum number of invocations per run.

After I’d modified the code to create a client and immediately release it afterwards, things were a little better. A little better. Now I would actually reach the ‘main’ phase, but eventually, the process would either halt or throw an exception, because it was occupying sockets so quickly that they weren’t getting released in time.

Eventually I had to settle for using a single client instance for the lifetime of the benchmark. Unsurprisingly, the results are very good.

Binary serialization was a fair option, that I’d just forgotten to include. It was quickly included, and it is definitely the fastest out-of-box serialization option for WCF. Still, MessagePack on ASP.NET Core MVC has an ever so slightly lower response time.

How about a comparison between Full Framework and .NET Core?

I very much wanted to include this, but I kept running into issues with BenchmarkDotNet, which either failed half-way, or didn’t report all of the metrics.

There’s no fair comparison between WCF and the others!

Redditor IAmVerySmarter (who, incidentally, also suggested using NetTcpBinding) complained that, because there was no benchmark between WCF and ASP.NET Core MVC or Web API using the MessagePack, it wasn’t fair to compare.

Unfortunately, the only package I could find to use MessagePack with WCF is MsgPack.Wcf, which uses MsgPack.Cli instead of MessagePack. According to the benchmarks, MsgPack.Cli takes a lot more time to serialize and deserialize than MessagePack. It’s still a fair comparison, so let’s see.

As you can see, MsgPack.Cli on either ASP.NET Core MVC or ASP.NET Web API takes several times longer than MessagePack. In fact, it takes longer than any other serializer tested. However, MsgPack.Cli on WCF takes a bit less time than on ASP.NET Core or ASP.NET Web API. This might indicate that the underlying infrastructure for WCF is the faster one.

Can you add [framework/serializer] to the comparison?

I could, yes. On the other hand, the source code is right there, so you could also fiddle around with it yourself.

ZeroFormatter

One serializer that I’ve seen mentioned several times is ZeroFormatter. This serializer claims to provide ‘infinitely fast deserialization’. That’s a bold claim, even from its creator, neuecc, who’s also responsible for the excellently performing MessagePack and Utf8Json libraries.

It does pretty well, but it’s not quite as quick as MessagePack. Note that this test consists of serializing, then deserializing, followed by serializing some other stuff, and then finally deserializing that again. Deserializing might be ‘infinitely fast’, but serializing apparently is not.

Benchmark code

In response to the previous article, I got a lot of questions from people who wanted to see the code I used to benchmark. There was a link hidden somewhere in the article, so let me now make it clear:

Here is the benchmark code: GitHub

New results

The new results table is simply huge, because I’ve added a few serializers, and mostly because now every serializer/host combination based on ASP.NET is tested with HttpClient and HttpWebRequest. Making the sort of interactive graph from the previous post was a lot of work, and there’s too much uninteresting data, so I’ve decided to include a single graph of the following:

• Mean round-trip times for a single call, sending 100 ‘large’ objects to the server, in response to which the server sends 100 objects back to the client.
• For ASP.NET Web API and ASP.NET Core MVC, all of the benchmarks use HttpWebRequest.

The entire table of results is included here, but since it is so large, a CSV of the results is also available, for easier viewing.

I was casually browsing Reddit when I came across a comment that triggered me. I’m paraphrasing¹, but the gist of it was that ‘WCF is faster than Web API or ASP.NET Core’. Surely, that was a mistake.

Boy, I’ll show them. What are they claiming, actually? ‘The response times of a WCF service are much lower than those of ASP.NET Web API or ASP.NET Core MVC.’ Pfft. I’ll just write a small benchmark using trusty ol’ BenchmarkDotNet. Stand up a local web server, measure how long it takes to create a request, send it, deserialize it, generate a response, send that back, and deserialize the response. One method will use WCF, the other will use ASP.NET Web API. To quote a colleague of mine: ‘how hard could it be?’

If you’re only interested in the complete picture, take a look at the full table of results and the graphs.

First steps

I’m sending 100 very simple objects (a single property that is a GUID) to the API and it’s sending 100 items back. Writing the benchmark wasn’t hard, but processing the outcome was, kind of. WCF was faster than ASP.NET Web API.

And not just a little bit faster; WCF is putting Web API to shame by taking less than a third of the time. Alright, but Web API is a pretty obsolete technology. Surely the new and shiny ASP.NET Core MVC will do much better. Right?

Well, it’s a little faster, but WCF still takes less than a third of the time.

Different formats

What’s so different between WCF and the other two options? Well, there is a pretty obvious difference: WCF is serializing data to and from XML (SOAP, to be precise), while for Web API and ASP.NET Core MVC, I was defaulting to JSON. What if I force the APIs to use XML?

There’s a definite improvement there. WCF is still a lot faster, but at least this shows we can get better results by picking a different serializer.

MessagePack

So what’s the fastest serializer we can find? According to their own claims, MessagePack is a small and fast format, and Yoshifumi Kawai has written a high-performance MessagePack serializer for .NET that beats protobuf-net, the default Protobuf serializer for .NET. Protobuf was designed to be easy to serialize and is known for its performance. Let’s see.

It’s going in the right direction. But can we go any faster? Probably not without extensive customization. However, these benchmarks have all been about serializing and deserializing 100 very simple objects. In the real world, objects are usually more complex than just a single GUID property.

Larger objects

Let’s try a more complex object.

public class LargeItem
{
    public Guid OrderId { get; set; }
    public ulong OrderNumber { get; set; }
    public string EmailAddress { get; set; }
    public Address ShippingAddress { get; set; } = new Address();
    public Address InvoiceAddress { get; set; } = new Address();
    public DateTimeOffset RequestedDeliveryDate { get; set; }
    public decimal ShippingCosts { get; set; }
    public DateTimeOffset LastModified { get; set; }
    public Guid CreateNonce { get; set; }
    public List<OrderLine> OrderLines { get; set; }
}

public class OrderLine
{
    public string Sku { get; set; }
    public int Quantity { get; set; }
    public string Product { get; set; }
    public decimal Price { get; set; }
}

public class Address
{
    public string Name { get; set; }
    public string Street { get; set; }
    public string HouseNumber { get; set; }
    public string PostalCode { get; set; }
    public string City { get; set; }
    public string Country { get; set; }
}

This is an order with order lines, copied from one of my other projects. There is a reasonable number of properties there, so let’s see how it goes.

Bingo. When serializing and deserializing 100 ‘large’ items, MessagePack on either ASP.NET Web API or ASP.NET Core MVC beats WCF by a small margin.

#itdepends

As usual, the answer to the question ‘which is faster’ is: it depends. When performance is absolutely critical, and your service doesn’t have to be a public API that anyone can consume, WCF is probably your best bet, up to a certain point. When you need to communicate large amounts of complex data, MessagePack on top of ASP.NET Core MVC is probably a better solution. Also worth considering is the developer-friendliness, where WCF doesn’t score very high marks.

What about JSON? The default library for working with JSON in .NET is Newtonsoft.Json. It’s great when you have to have absolute control over how your JSON looks, but it’s by far the slowest option I’ve examined. Can we have our cake and eat it too?

Utf8Json

It turns out that, yes, we can. Sort of. Yoshifumi Kawai, who you’ll remember as the author of the very fast MessagePack serializer for .NET, has also written a performance-oriented JSON serializer for .NET called Utf8Json. It’s not as customizable and it doesn’t support DOM operations, but man, is it fast.

It’s not as fast as WCF, and definitely not as fast as MessagePack, but it easily takes less than half the time that Newtonsoft.Json does. It does this by avoiding memory allocations (similar to the MessagePack serializer) and by treating JSON as a binary format; it doesn’t serialize to a string which is then converted into bytes, but instead it serializes straight to UTF-8 bytes².

Conclusion

As I said before, it depends. If round-trip time performance is more important than pretty much anything else, but you still want to use a managed language, use WCF. If you care about performance, but also about legible and easy to use code, or you want to create a proper REST-ful service, use MessagePack or Utf8Json on top of ASP.NET Core MVC or ASP.NET Web API.

The code I’ve used to benchmark these libraries is available on GitHub. Feel free to play around with it, and maybe find even better options.

Data

Below you’ll find the ‘raw’ output from the benchmark I’ve run. Some annotations:

• A method prefixed with ‘Small’ means it serializes and deserializes a small class with only a single GUID property. ‘Large’ means it serializes and deserializes the large class described above.
• All WCF methods are hosted using WebServiceHost;
• All WCF methods use a ChannelFactory client, while the Web API and ASP.NET Core MVC methods use System.Net.HttpClient³;
• WcfText uses a BasicHttpBinding with the message encoding set to Text;
• WcfWebXml and WcfWebJson use a WebHttpBinding, with, respectively, XML or JSON as the message format;
• JsonNet methods use Newtonsoft.Json to serialize objects;
• I’ve included ‘0 items’ as an indication of how fast the client and web server/framework are;
• The ‘P95’ column represents the 95th percentile value of the response time;
• The ‘Gen 0’, ‘Gen 1’ and ‘Gen 2’ columns represent the number of garbage collections per 1 000 invocations;
• The ‘Allocated’ column represents the total amount of memory allocated for each invocation;

Some interesting data points that jump out:

• At 0 items, ASP.NET Core makes significantly fewer allocations, and as a result, it is the only option that results in no Generation 1 (or higher) allocations;
• Even with 0 items, the options using Newtonsoft.Json are a lot slower than any of the other options. There must be a lot of initialization happening;
• For larger item sizes, Utf8Json is frequently allocating the most memory of any option, but it still manages to be pretty fast. That just goes to show - avoiding memory allocation isn’t the only thing you need for speed;
• ASP.NET Core’s XML serializer is a lot slower than Web API’s - this is particularly visible for large items;
• WCF does very well in terms of allocated memory. If memory is constrained, WCF might be a viable option;

A note on the chart: I’ve tried my best to make it look good on desktop and on mobile, but the viewing experience is still best on a large screen.

A while ago I wrote about transitive dependencies and how, in the current day and age, MSBuild and classic NuGet (using packages.config, colloquially known as ‘Visual Studio 2015 style’) are pretty bad at this, while the new dotnet tooling (using PackageReference, known as ‘Visual Studio 2017 style’) is much better at it.

TL;DR – Changing the assembly file name for a published NuGet package causes upgraders to experience difficult and frustrating errors.

Recently, I ran into an issue. We use Datadog to record and graph metrics on our systems and environment. They’ve helpfully provided a library to emit metrics from C# applications. They’ve even published it on NuGet! Of course, we don’t want to tie our application code directly to Datadog’s API. One day, somebody might get a bright idea and say that, instead, we should be storing our metrics in Microsoft Application Insights. At that point we don’t want to have to rewrite large parts of our code. So we’ve defined an interface called Metrics and created a private NuGet package that provides an implementation of that interface. If we switch implementations, all we have to do is switch out that library, rewrite some configuration code, and we’re good to go. So far, so good.

Upgrade to dotnet

I happened to be upgrading a project from MSBuild tooling to dotnet tooling. The process for that is pretty much as follows:

1. Open your .csproj file, and replace it with this minimal XML:

<Project Sdk="Microsoft.NET.Sdk"> <!-- Or "Microsoft.NET.SDK.Web" for web apps -->
  <PropertyGroup>
    <TargetFramework>net462</TargetFramework>
  </PropertyGroup>
</Project>

2. Let your IDE reload the project and start inspecting the errors, adding NuGet and project references as you go.
3. When all the errors are gone, try running it.
4. Commit your changes.

While I was going through the errors, I saw there was a direct dependency on Datadog’s library to be able to configure it. I opened the ‘Manage NuGet Packages’ tool window and installed the DogStatsD-CSharp-Client package. The version installed was 3.1.0. A bit later I came across some code that used our Metrics implementation, so I installed our private NuGet package. It had a dependency on version 3.0.0 of the same DogStatsD-CSharp-Client package. Because we know the dotnet tooling can deal with this version conflict, I didn’t even give it a second thought.

Error time

Finally, it was time to run the code. I was presented with a very disturbing error.

Unhandled Exception: System.IO.FileNotFoundException: 
Could not load file or assembly 'DogStatsD-CSharp-Client, Version=3.0.0.94, Culture=neutral, PublicKeyToken=null' or one of its dependencies. 
The system cannot find the file specified.
   at OurMetricsAbstraction.Datadog.DatadogMetrics.IncrementCounter(String name, String[] tags)
   at WereGonnaMakeSoMuchMoneyWithThis.Program.Main(String[] args)

Wait, what?

The runtime cannot load version 3.0.0.94 of the DogStatsD-CSharp-Client library. I have just upgraded the version, so could that be a problem? It should’t be, because the build system should handle all of this for me. Maybe there’s an issue with the binding redirects. The new build system automatically generates these based on the references and stores them in MyAssemblyName.dll.config (or .exe.config, of course).

Lots of binding redirects there, but none for DogStatsD-CSharp-Client.

File not found

Wait a minute. It’s not a version mismatch. It says The system cannot find the file specified. If it had been a version mismatch, it would have looked like this:

System.IO.FileLoadException: Could not load file or assembly 'DogStatsD-CSharp-Client, Version=3.0.0.94, Culture=neutral, PublicKeyToken=null' or one of its dependencies.
The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

So, alright, definitely a missing file. Let’s look at the ‘bin’ directory. There are hundreds of files there, but, indeed, DogStatsD-CSharp-Client.dll is not one of them. What gives? I have a direct dependency on the package, so it should be there, right? Is there somehow an issue with the NuGet package? If there were, I would expect to get build errors, though.

CSI NuGet

I’m using the excellent NuGet Package Explorer to examine the contents of NuGet packages.

Well, well, well. The file in the 3.1.0 version of the NuGet package is called StatsdClient.dll. Just to drive the point home, what does the 3.0.0 version look like?

Right. Our NuGet package implementing the Metrics interface using the Datadog client was compiled against DogStatsD-CSharp-Client, version 3.0.0, so it refers to methods in DogStatsD-CSharp-Client.dll.

The hosting application, however, has a direct reference to version 3.1.0 of the package, so it refers to methods in StatsdClient.dll. Let me show you some IL code to illustrate this.

Let’s take the following C# code:

StatsdClient.DogStatsd.Configure(new StatsdClient.StatsdConfig());

In the NuGet package, the IL for that looks like this:

newobj       instance void ['DogStatsD-CSharp-Client']StatsdClient.StatsdConfig::.ctor()
call         void ['DogStatsD-CSharp-Client']StatsdClient.DogStatsd::Configure(class ['DogStatsD-CSharp-Client']StatsdClient.StatsdConfig)

For those of you not fluent in IL (which includes myself), the second line is saying ‘call a static method Configure from the type StatsdClient.DogStatsd, located in the assembly DogStatsD-CSharp-Client’.

In the hosting application, the IL for the exact same code looks like this:

newobj       instance void [StatsdClient]StatsdClient.StatsdConfig::.ctor()
call         void [StatsdClient]StatsdClient.DogStatsd::Configure(class [StatsdClient]StatsdClient.StatsdConfig)

The exact same method is being called, but declared as coming from a different assembly.

So let’s recap what’s happening.

1. The NuGet package has a dependency on DogStatsD-CSharp-Client, version 3.0.0 or, more formally, >= 3.0.0.
2. The hosting application has a dependency on the same package, but version >= 3.1.0.
3. The final resolved version of the package is 3.1.0, because it satisfies both requirements.
4. Because of this, the only file that ends up being copied to the output directory of the project is that of version 3.1.0, StatsdClient.dll.
5. At runtime, the code in the NuGet package instructs the runtime to load DogStatsD-CSharp-Client.dll, which doesn’t exist, causing an exception to be thrown.

Dangerous Subtleties

In this case, because the conflict is located in a NuGet package, the result is a file not being found, which is a pretty visible problem that also manifests itself in a clear way.

If, however, the implementation for the Metrics interface had been a project in the same solution, the issue would have been much more subtle and might never have been discovered.

In my experiments to recreate the issue for this blog post, I initially created exactly that situation. A class library project that references version 3.0.0 of the Datadog NuGet package, and then a console application project that references version 3.1.0 and the class library project.

Visual Studio 2017 nicely highlights the different versions, and coincidentally, also the different file names.

In my console application, I call out to the library project, which calls out to DogStatsD-CSharp-Client.dll. It should cause an exception, right? I thought so too, but it doesn’t. Looking in the bin directory, we can see why.

MSBuild has a lot more insight into what the library project is referencing, so it dutifully copies DogStatsD-CSharp-Client.dll into the bin directory of the main project. The library’s code is linked to that, so when that code is executed, the runtime loads DogStatsD-CSharp-Client.dll and runs its code. However, when the main project uses code from the NuGet package – to configure the client, for example – the runtime also loads StatsdClient.dll and runs its code.

Why is this bad? Well, because of the different assembly names, the runtime considers these to be different modules, which causes all kinds of ‘strange’ behavior.

For example, static members defined in DogStatsD-CSharp-Client.dll are not the same as those in StatsdClient.dll. This means that I can ‘configure’ the client in the main project, but the code in the library project doesn’t see any of that.

Let’s see this in practice. One of the settings that can be configured is the prefix that automatically gets added to metric names. Because this is a private piece of data, I wrote some code that extracts it by hacking reflection.

private static string GetPrefix()
{
    var serviceField = typeof(DogStatsd).GetField("_dogStatsdService", BindingFlags.Static | BindingFlags.NonPublic);
    var service = (DogStatsdService) serviceField.GetValue(null);

    var prefixField = typeof(DogStatsdService).GetField("_prefix", BindingFlags.Instance | BindingFlags.NonPublic);
    return (string) prefixField.GetValue(service);
}

I configured the prefix to be "test" in the main project, and when I run the method from both projects, you can see it doesn’t work as we intended:

(Main) Prefix: test
(Library) Prefix: (null)

Now, this is just a metric prefix, but you can imagine what might happen if you configure, for example, a connection string in this manner.

Dependency injection might also suffer from this problem. As you can see in the code snippet I use to extract the static prefix, there is a type called DogStatsdService. This is actually a public type that is the non-static version of the API. It also implements an interface called IDogStatsd.

I can write a class Foo (because naming is difficult) in my library project that has a dependency on IDogStatsd. In my hosting application, I’ll register DogStatsdService to be the implementation for IDogStatsd. Using SimpleInjector, trying to resolve Foo will result in the following exception:

SimpleInjector.ActivationException: 
No registration for type Foo could be found and an implicit registration could not be made.
The constructor of type Foo contains the parameter with name 'service' and type IDogStatsd that is not registered. 
Please ensure IDogStatsd is registered, or change the constructor of Foo. 
Note that there exists a registration for a different type StatsdClient.IDogStatsd while the requested type is StatsdClient.IDogStatsd.

This is both highly informative and confusing. On the one hand, it tells us that there is no registration for IDogStatsd, which is confusing. On the other hand, it does give us a clue by mentioning that there exists a registration for a different type StatsdClient.IDogStatsd while the requested type is StatsdClient.IDogStatsd.

It still requires a fair bit of investigation to resolve.

Conclusion

The assembly names of a NuGet package are much more than just file names. They’re a contract, really. As with most contracts, changing the names will cause issues. Changing them without informing users is not so nice. Changing them without incrementing the major version of the package, which indicates a change that is not backwards compatible, is very unfriendly.

In general, just don’t change the assembly name. Ever. Stick with the one you used when you first published the package.

When you decide to implement your business logic by applying DDD, one of the things you’ll run into is ‘how do I save my changes?’ The internet is full of blogs and articles about the mythical DDD repository, but all they offer is an interface. How do you actually implement it?

The interface

First, let’s take a look at what the interface for the repository should look like. But, first, what should a repository actually do? Per Martin Fowler’s Patterns of Enterprise Application Architecture Catalog:

[The repository] mediates between the domain and data mapping layers using a collection-like interface for accessing domain objects.

Alright, so at the very least, we should have a method for accessing the entire collection. And it probably makes sense to be able to single out one item if you already know its unique identifier. Let’s not get into super-generic repositories¹ and say that our domain models have a unique identifier that is truly unique: a GUID. Our domain is going to be a pretty simple one: placing and modifying orders.

public interface IOrderRepository
{
  IEnumerable<Order> GetAll();
  Order GetById(Guid orderId);
}

Right, that takes care of reading. We should of course also be able to create, update and delete orders. Wait, no, that sounds too much like CRUD, which we don’t like because we’re currently in our Ivory DDD Tower. We should be able to modify the collection of orders. Much better.

public interface IOrderRepository
{
  IEnumerable<Order> GetAll();
  Order GetById(Guid orderId);
  void Add(Order order);
  void Update(Order order);
  void Remove(Order order);
}

We’ve nicely side-stepped the CRUD anti-pattern by using different verbs (mostly) for the repository methods. Great!

YesSQL

So, let’s go about implementing this interface. Fine, you’ll say. Just spin up a MongoDB instance, and then implement GetAll by retrieving an entire collection, Get​By​Id by doing .Find​Async(Builders<Order>​.Eq(o => o.OrderId, orderId). Add, Update and Remove basically map to Insert​One​Async, Update​One​Async, and Delete​One​Async, respectively.

Whoa there, not so fast. We’re already paying through the nose for this big-ass Oracle cluster that we have just sitting there, waiting to be used. I’d like you to store data in Oracle, if you please. And if you’ll turn to page 257 of the Bible Coding Guidelines as set forth by our Enterprise Architect, you’ll notice that it is not allowed to store documents in a table. We can’t run our daily reports and business intelligence cubes on JSON! So, a table schema in proper 3NF, please.

Alright. Oracle it is.

The Order aggregate

At this point, we should probably take a look at our domain model. What is an order, really? Let’s ask our business analyst.

Well, this is just an MVP, so an order is really simple. It has some customer and shipping information (primarily the address) and some price stuff. Oh, and order lines. And you should be able to modify an order before you finally send it off to be fulfilled.

For the sake of the example, let’s not get into a discussion about what an order line is. An order line is simply a product number (a SKU), a quantity and a per-item price. So, we have something like this:

public class Order
{
  public Guid OrderId { get; }
  public Address ShippingAddress { get; }
  public Customer Customer { get; }
  public decimal TotalPriceIncludingVat { get; }
  public decimal TotalPriceExcludingVat { get; }
  public IReadOnlyCollection<OrderLine> OrderLines { get; }
  public bool IsReadyForFulfillment { get; private set; }
}

public class OrderLine
{
  public string Sku { get; }
  public int Quantity { get; private set; }
  public decimal PerItemPriceIncludingVat { get; }
}

Looks easy enough. What about the operations?

You should be able to create a new order, obviously, add and remove order lines, update an order line’s quantity, and finally send the order off for fulfillment.

So, something like this:

public class Order
{
  // rest of class omitted for brevity
  
  public static Order CreateNew(Address shippingAddress, Customer customer) { /* ... */ }
  public void AddOrderLine(string sku, int quantity, decimal perItemPriceIncludingVat) { /* ... */ }
  public void RemoveOrderLine(string sku) { /* ... */ }
  public void ChangeOrderLineQuantity(string sku, int quantity) { /* ... */ }
  public void Fulfill() { /* ... */ }
}

Now, the implementations of these operations are omitted for brevity, but you can imagine what they would be like. Change​OrderLine​Quantity, for example, would look up the order line with the provided SKU, and modify its Quantity property.

Persist all the things!

Now, how do we persist this to an Oracle database? Let’s first take a look at what the orchestrating code (the ‘use case’) should look like. I specifically want to look at the use case of adding an order line, because that’s where it starts to get interesting.

public void AddOrderLine(Guid orderId, string sku, int quantity)
{
  var order = _orderRepository.GetById(orderId);
  var perItemPrice = _productRepository.GetBySku(sku).PriceIncludingVat;
  
  order.AddOrderLine(sku, quantity, perItemPrice);
  
  _orderRepository.Update(order);
}

What about the database schema? Well, it’s pretty straight-forward; there’s an ORDER table that contains the orders themselves, and an ORDERLINE table that contains the order lines. ORDERLINE has a foreign key that references back to the ORDER table, to keep things nice and consistent.

create table "ORDER"
(
    ORDERID raw(16) primary key not null,
    SHIPPINGADDRESS varchar(1000) not null,
    CUSTOMEREMAIL varchar(250) not null,
    READY char not null,
    TOTALPRICEINCVAT number not null
);

create table "ORDERLINE"
(
    ORDERID raw(16) not null,
    SKU varchar(50) not null,
    QUANTITY int not null,
    PRICEINCVAT number not null,
    constraint ORDERLINE_PK primary key (ORDERID, SKU),
    constraint ORDERLINE_ORDER_FK foreign key (ORDERID) references "ORDER" (ORDERID)
);

Implementing creating and deleting an order is pretty much self-explanatory, so let’s try and implement IOrder​Repository.Update(), which should be more of a challenge. Updating the order itself is fairly easy. We just do something like this:

public void Update(Order order)
{
  _connection.Execute(
    "update ORDER set " + 
    "SHIPPINGADDRESS = :SHIPPINGADDRESS, " +
    "CUSTOMEREMAIL = :CUSTOMEREMAIL, " +
    "READY = :READY, " +
    "TOTALPRICEINCVAT = :TOTALPRICEINCVAT "
    "where ORDERID = :ORDERID",
    new
    {
      ShippingAddress = order.ShippingAddress.ToString(),
      CustomerEmail = order.Customer.EmailAddress,
      Ready = order.IsReadyForFulfillment ? 'Y' : 'N',
      TotalPriceIncVat = order.TotalPriceIncludingVat,
      OrderId = order.OrderId.ToByteArray(),
    }
  );
}

Done. On to the order lines. So... foreach (var line in... in what? How do we know what the state of an order line is? We don’t know which order lines are new or existing, or which of the existing ones are modified or unmodified (or ‘clean’ or ‘dirty’, if you will).

We could just iterate over all of them, see if they already exist in the database, and either insert or update all of them. Yuck. That’s not very elegant at all, and it doesn’t scale well when we have an order with hundreds or thousands of lines, or when there is heavy load on the system.

Another solution would be to sneak some extra properties into the domain model, so that it keeps track of those things itself. So we add Is​New and Is​Modified to the order and order line model. How about deletion? We could introduce an Is​Deleted property on the order, but for order lines this presents a problem; if we use a property to indicate it was deleted, we’d have to be careful not to include those deleted lines in any business logic that involved the current order lines of an order, such as checking if there’s already an order line for a particular SKU. We could introduce a separate collection of order lines that have been deleted; then we can delete an order line from Order​Lines and add it to Deleted​Order​Lines for the repository. You see how this quickly gets out of hand, and in addition to that, Deleted​Order​Lines or Is​Modified are not things that the business cares about. Let’s try and keep the domain model clean.

How about we have the repository keep track of all the entities and associated objects it returns? That’s what an ORM framework does. It means that you’ll have to deal with the increased memory usage and processing time it takes to keep and process this additional state. An ORM usually works by capturing the state of an object before it’s returned to the caller, and when saving comparing the differences. It’s really kind of a brute-force approach. Also, you’ll have to pick, configure, and wire up an ORM framework, which can be difficult to get right, even with something as ubiquitous as Entity Framework. Or you can write one yourself, which is incredibly difficult to get right.

Let’s try a different approach.

Persisting value types

One of the solutions we have is to modify the domain model a bit so that, instead of returning void, operations return a value object describing the change. We then modify the repository interface to have a single method for each kind of change. Yes, this could work. It also has the benefit of making it very explicitly clear that a new operation is not yet implemented, because the repository interface won’t have a method for it yet. The operations on the domain model would look something like this:

public class Order
{
  // rest of class omitted for brevity
  
  public static Order CreateNew(Address shippingAddress, Customer customer) { /* ... */ }
  public OrderLineAdded AddOrderLine(string sku, int quantity, decimal perItemPriceIncludingVat) { /* ... */ }
  public OrderLineRemoved RemoveOrderLine(string sku) { /* ... */ }
  public OrderLineQuantityChanged ChangeOrderLineQuantity(string sku, int quantity) { /* ... */ }
  public OrderFulfillmentRequested Fulfill() { /* ... */ }
}

Note that Create​New still returns an Order, because when creating a new order, there is no ambiguity; everything is new. Also note that the type names of the value objects returned are formulated like events: Order​Line​Added, Order​Line​Removed, et cetera. They all describe things that have already happened. You could also think of them as very explicit deltas; differences between the state of an order in two points in time.

Next, let’s modify the repository interface to make this work.

public interface IOrderRepository
{
  IEnumerable<Order> GetAll();
  Order GetById(Guid orderId);
  void Add(Order order);
  void Store(OrderLineAdded @event);
  void Store(OrderLineRemoved @event);
  void Store(OrderLineQuantityChanged @event);
  void Store(OrderFulfillmentRequested @event);
  void Remove(Order order);
}

The implementation almost writes itself. For brevity, I won’t spell it out here, but you should be able to imagine what it would look like.

There are some problems, however. The initial version of the interface is ideal for when you want to persist data in a NoSQL database, because you can simply serialize your object graph, write it to the database, and you’re done. The second version of the interface, with the value types, is ideal for when you want to persist data in an RDBMS, because for each event you run some SQL queries that get your data in the right state. This is pretty cumbersome to do when your storage mechanism is a NoSQL database. Choosing one interface pattern over the other pretty much locks your application into the associated storage mechanism.

Having to define a new method on the repository interface for each operation that your application supports exposes a flaw that is perpendicular to one of its benefits. The interface is a contract, and when you extend the interface by adding a new operation, you’ve broken that contract; all of its implementations now need to be modified, which violates the Open/Closed Principle.

Persisting events

So, how do we solve this conundrum? Driving persistence based on events seems like a good idea, but how do we avoid the Open/Closed Principle violation? Let’s use that angle as the basis for our solution. Let’s go back to a single ‘save’ method.

public interface IOrderRepository
{
  IEnumerable<Order> GetAll();
  Order GetById(Guid orderId);
  void Store(IEnumerable<IEvent> events);
}

At least the interface is now much cleaner. Note that, as well as condensing all of the Store methods into one method, the Add and Remove methods have also been removed. So how do we add a new order? That’s an event! How do we remove an order? Also an event! The Store method accepts a collection of IEvent, which can just be a marker interface to make it explicit what kind of objects we accept.

This design still has some problems, though. Because the interface accepts a collection of events, the use case (or service, or whatever you want to call it) has to collect a list of events as it’s performing operations on the order. That shouldn’t be the use case’s job. Also, what if we decide to use a NoSQL database and just want to store a serialized version of the order?

Let’s refine it once more.

public interface IOrderRepository
{
  IEnumerable<Order> GetAll();
  Order GetById(Guid orderId);
  void Store(Order order);
}

That’s better. Now it truly does not matter what kind of persistence mechanism you want to use, and the use case can stay truly ignorant about this fact. But who’s collecting the events? It’s the aggregate root itself. Each operation returns void again and, in addition to modifying the aggregate root’s state, it’s adding an event to a list of events that have not yet been persisted.

What do the internals look like? I’m going to focus on a single use case: changing an order line’s quantity. First, let’s look at the Order and OrderLine classes.

public class Order
{
  public void ChangeOrderLineQuantity(string sku, int quantity)
  {
    var orderLine = _orderLines.First(l => l.Sku == sku);
      
    orderLine.ChangeQuantity(quantity);
    
    _eventsPendingPersistence.Add(
      new OrderLineQuantityChanged(
        orderId: OrderId,
        sku: sku,
        quantity: quantity
      )
    );
  }
  
  public IReadOnlyCollection<IEvent> DequeueAllEvents()
  {
    var events = _eventsPendingPersistence.ToList();
    
    _eventsPendingPersistence.Clear();
    return events;
  }
  
  public IReadOnlyCollection<OrderLine> OrderLines => _orderLines;
  
  private readonly List<OrderLine> _orderLines = new List<OrderLine>();
  private readonly List<IEvent> _eventsPendingPersistence = new List<IEvent>();
}

public class OrderLine
{
  internal void ChangeQuantity(int quantity)
  {
      Quantity = quantity;
  }
  
  public int Quantity { get; private set; }
}

The Dequeue​All​Events method provides the repository access to the events that have not yet been persisted. It’s called ‘dequeue’ because it returns all the events in the ‘queue’, and then empties the queue. It’s implemented using a List<T> rather than a Queue<T>, because we don’t need the random access, one at a time operations that it exposes.

The use case code is trivially simple. Get the order, operate on it, store it.

public void ChangeOrderLineQuantity(Guid orderId, string sku, int quantity)
{
  var order = _orderRepository.GetById(orderId);

  order.ChangeOrderLineQuantity(sku, quantity);
  
  _orderRepository.Store(order);
}

What about the repository code? It gets an order, uses Dequeue​All​Events to get all its events and then dynamically dispatches them.

class OrderRepository: IOrderRepository
{
  public IEnumerable<Order> GetAll() { /* not implemented for brevity */ }
  public Order GetById(Guid orderId) { /* not implemented for brevity */ }
  
  public void Store(Order order)
  {
    var events = order.DequeueAllEvents();
    
    DispatchAllEvents(events);
  }
  
  private void DispatchAllEvents(IReadOnlyCollection<IEvent> events)
  {
    foreach (var @event in events)
      DispatchEvent(@event);  
  }
  
  private void DispatchEvent(IEvent @event)
  {
    Handle((dynamic) @event);
  }
  
  private void Handle(OrderLineQuantityChanged @event)
  {
    _connection.Execute(
      "update ORDERLINE set " +
      "QUANTITY = :QUANTITY " +
      "where ORDERID = :ORDERID and SKU = :SKU",
      new
      {
        OrderId = @event.OrderId.ToByteArray(),
        @event.Sku,
        @event.Quantity,
      }
    );
  }
}

Pretty straight-forward. The Dispatch​Event method does something peculiar: it casts the event to dynamic. This is the most straight-forward approach to dispatching events, and it also happens to be the fastest one.

Sidebar: dynamic performance

Let me briefly dig in to that statement. I ran a benchmark using BenchmarkDotNet, comparing various methods of dispatching events; using dynamic, using reflection, using a lambda, and using pattern matching. The results: of course pattern matching using the switch statement is the fastest option at around 6 ns per invocation on .NET Core, followed closely by... dynamic at around 12 ns per invocation. Reflection, with a pre-warmed Method​Info cache is a lot slower at around 197 ns per invocation². I expected reflection to be ‘slow’, but I was pleasantly surprised at how lightning-quick dynamic is, especially on .NET Core. Mind you, we’re still talking about nanoseconds; 200 nanoseconds is one 5 millionth of a second.

For reference, my benchmark code can be found in this Gist. You can easily see that the Dynamic​Dispatch class is by far the smallest and easiest to read, and there is no performance-related reason not to do this.

Advantages & Disadvantages

So now that we’ve established the ‘event-driven persistence’ pattern, let’s look at some of its advantages and disadvantages.

Advantages

• It allows for easy implementation of your repository for different persistence mechanisms; document-based, relational database, or Event Sourcing are all very easy to implement.
• Even if you’re not going for Event Sourcing, but you still want to store, broadcast, or publish events in your application, you basically get this for free.
• Implementing persistence for a relational database is exceedingly easy, without having to rely on an ORM framework.
• Just one method for persistence. Even creation and deletion can be represented as events, after all.
• Compared to the ‘persist value objects from the domain model’ pattern, transaction management can remain inside the persistence layer, rather than having to be managed by the orchestrating code.
• Your domain model will remain clean of persistence-related artifacts.

Disadvantages

• Depending on the complexity of your domain model, you’re going to have to model a lot of events.
• Only pattern matching gives you static type safety. On the other hand, even with pattern matching, you’ll want to test whether a particular type of event is supported by your code, so you’ll have to write tests for all of the event types, anyway.

Sidebar: On modeling events

One important characteristic to keep in mind when modeling the events your domain model will raise is this: events are things that happened in the past, and since you cannot change the past, they should be immutable. I’m saying should, but when you implement persistence using Event Sourcing, or when you expose these events to the outside world, that should becomes a must.

What this means is that you must not design an event like this:

class OrderLineAddedEvent
{
  public OrderLine NewOrderLine { get; }
}

But this looks immutable, you might say. Yes, but Order​Line, being part of the domain model, is almost definitively mutable, which makes the entire event mutable. It means that by the time you handle or serialize this event, the state of the instance you put in there might have changed from when you created the event. This means you’ve lost some information, which might not be all that bad when all you’re doing is persisting the current state, but when you’re doing Event Sourcing or exposing the event to the outside world, it makes your events lose value.

Instead, design your events like this:

class OrderLineAddedEvent
{
  public string Sku { get; }
  public int Quantity { get; }
  public decimal PerItemPriceIncludingVat { get; }
}

Also, think about how granular you want your events to be. Too coarse, like Order​Changed​Event that you raise for every change to an order, and it will be difficult to extract what has changed from the event, which is the whole point of this exercise. Too fine-grained, like Order​Shipping​Address​House​Number​Changed, and you’ll get an explosion of events and corresponding Handle methods and a lot of traffic to the database. You want to capture intent. When somebody changes the house number on the shipping address of an order, what they’re doing is either correcting the shipping address or choosing a different shipping address. You might not be able to capture the distinction from your UI and use cases, but at the very least it becomes apparent that you’ll only change the address as a whole.

Conclusion

Sorry for the very long post, but the subject matter is unfortunately not very simple. This pattern, let’s name it ‘recorded event-driven persistence’, is simple and elegant, and it allows you to persist complex domain models without your persistence code becoming total spaghetti.

If you think it can be improved, let me know. Also, if this is already a named pattern, or if you can think of a better name than ‘recorded event-driven persistence’, let me know!

The other day, a colleague and I were looking into an issue with one of our solutions’ build pipeline. The main project was a ‘classic’ MSBuild project with a transitive NuGet dependency through another classic MSBuild project. The transitive dependency wasn’t being copied into the main project’s output directory, even though the build was succeeding.

Note: I’ve fictionalized package names in this blog, because the problem has nothing to do with the authors, and I wouldn’t want their names to be associated with this problem.

What’s a transitive NuGet dependency?

Let’s first start with a slightly simpler question: what is a transitive dependency?

A transitive dependency is an indirect dependency; you depend on something which itself has another dependency that you don’t depend on.

Let’s say you have projects B, C, and D (I’ll explain why I don’t start with ‘A’ in a bit). When project B has a reference to project C and project C has a reference to project D, but B does not have a reference to D, then D is a transitive dependency of B.

Now, when you create this exact project structure, there will not be an issue; MSBuild will build project D, then project C, and finally project B. The output directory for project B will contain B.dll, C.dll, and D.dll. So far, so good.

Now let’s say that C is a NuGet package, and so is D. The picture changes slightly, because in when you install a package in a classic MSBuild project, its dependencies will also be directly referenced by B. So B has a direct reference to D. However, there is not a single line of code that actually mentions D, or any of its symbols. If you squint, it is still a transitive dependency.

What’s the problem?

So far, nothing. When you compile the project, the output directory will still contain B.dll, C.dll, and D.dll, just like when they were still projects.

What I’ve left out, so far, is that, actually, there is another layer of referencing. See, in our situation, B is actually the persistence layer of that particular solution. C is a wrapper library that we’d written around D, which is a commonly used NuGet package. The persistence layer is being referenced by the ‘host’ project, which in our case is a web API. Let’s call that A. Now you see why I started with B.

Project A has a reference to B, but does not have any reference to the dependencies of B. The problem was that, although C.dll was copied to the output directory for A, D.dll was not.

Version hell

Initially, we thought it was ‘just a bug’ in MSBuild. We’d written a hacky MSBuild task to just recursively copy all of the dependencies from all the projects referenced by the main project. I refused to believe this was a scenario that MSBuild ‘just didn’t support,’ so I started digging.

I built our solution with the verbosity set to ‘Detailed,’ and then I came across a couple of interesting messages. Some details left out for brevity.

Project "C:\Project\Project.sln" (1) is building "C:\Project\B.csproj" (4:3) on node 1 (default targets).

  There was a conflict between "D, Version=1.50.2.0, Culture=neutral, PublicKeyToken=null" and "D, Version=1.50.0.0, Culture=neutral, PublicKeyToken=null".
      "D, Version=1.50.2.0, Culture=neutral, PublicKeyToken=null" was chosen because it was primary and "D, Version=1.50.0.0, Culture=neutral, PublicKeyToken=null" was not.
      References which depend on "D, Version=1.50.2.0, Culture=neutral, PublicKeyToken=null" [C:\Project\packages\D.1.50.2\lib\net451\D.dll].
          C:\Project\packages\D.1.50.2\lib\net451\D.dll
            Project file item includes which caused reference "C:\Project\packages\D.1.50.2\lib\net451\D.dll".
              D, Version=1.50.2.0, Culture=neutral, processorArchitecture=MSIL
      References which depend on "D, Version=1.50.0.0, Culture=neutral, PublicKeyToken=null" [].
          C:\Project\packages\C.1.0.1\lib\net45\C.dll
            Project file item includes which caused reference "C:\Project\packages\C.1.0.1\lib\net45\C.dll".
              C, Version=1.0.1.0, Culture=neutral, processorArchitecture=MSIL
...              
Project "C:\Project\Project.sln" (1) is building "C:\Project\A.csproj" (3) on node 1 (default targets).
...
  Dependency "D, Version=1.50.0.0, Culture=neutral, PublicKeyToken=null".
      Could not resolve this reference. Could not locate the assembly "D, Version=1.50.0.0, Culture=neutral, PublicKeyToken=null". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors.
          For SearchPath "C:\Project\B\bin\Release".
          Considered "C:\Project\B\bin\Release\D.winmd", but it didn't exist.
          Considered "C:\Project\B\bin\Release\D.dll",
  			but its name "D, Version=1.50.2.0, Culture=neutral, PublicKeyToken=null"
  			didn't match the expected name "D, Version=1.50.0.0, Culture=neutral, PublicKeyToken=null".
      Required by "C:\Project\B\bin\Release\B.dll".

The plot thickens. What you see here is that during the build of project B, MSBuild finds a conflict between two versions of D, version 1.50.0.0 and version 1.50.2.0. It ends up picking 1.50.2.0, because that is a direct dependency. It then lists in detail that project B depends directly on version 1.50.2.0 of package D, while package C depends on version 1.50.0.0 of package D.

This makes sense; C is a utilities library, and for libraries you usually want to depend on the lowest possible version, to maximize compatibility. However, when you’re writing application code, you typically want to depend on the highest possible version, because that might have fewer bugs. This is pretty much what happened; we’d already added a reference to version 1.50.2.0 of package D to project B, and only later did we add a reference to package C.

A few things stand out from the MSBuild output:

• MSBuild is saying that package D is what caused the reference to ... itself?
• If this reference is required by your code, you may get compilation errors. Yet the build still completes successfully?
• It’s incorrectly stating that, when building project A, B.dll is what depends on version 1.50.0.0 of package D. It is actually C.dll that depends on that version.

These are not major issues (except for the build completing succesfully), but they might slow down an investigation into why something is not working.

So, to recap: because project B has a direct reference to package D, version 1.50.2.0 of that package gets copied to the output directory for project B. When building project A, MSBuild is not able to resolve the reference, because it only sees that package C depends on version 1.50.0.0 of package D. It does irk me that this kind of stuff does not fail the build.

However, at run-time this will cause exceptions; the runtime will load C.dll and see that it has a reference to version 1.50.0.0 of D.dll. It will attempt to load D.dll, but this is not available, so .NET will throw a File​Not​Found​Exception.

Binding redirects

The ‘proper’ way of fixing this issue is to create a binding redirect in the app.config or web.config for project A. This is a pretty cumbersome process, as you have to figure out the exact assembly version of the dependency, which might be different from the package version, and in a lot of cases you also have to copy and paste the public key token. In short, this is not a very pleasant experience. Unfortunately, for classic MSBuild projects, it is the only real solution.

<dependentAssembly>
  <assemblyIdentity name="D" culture="neutral" publicKeyToken="1234567890abcdef" />
  <bindingRedirect oldVersion="0.0.0.0-1.50.2.0" newVersion="1.50.2.0" />
</dependentAssembly>

Tree-shaking

But wait, we’re not done, yet. I made a remark earlier, which is significant to the problem.

However, there is not a single line of code that references D.

If you do add a line of code that references that dependency, then the binding redirect does not appear to be necessary. Apparently, MSBuild does some form of tree-shaking to figure out which dependencies are really required and which ones are only there to satisfy indirect dependencies.

When project B directly uses package D, then MSBuild sees it as a required dependency, and therefore uses the version that project B depends on. Otherwise, it uses the version that package C depends on (even though it reports it incorrectly).

.NET Core to the rescue

Fortunately, there is an even easier solution: use the .NET Core build system (or ‘the new .csproj format, or dotnet, or whatever you want to call it).

The new .csproj format only requires you to specify the direct reference. In our case, that would be package C.

<PackageReference Include="C" Version="1.0.1" />

When the project is built, .NET Core will first restore all packages, and figure out that package D is a dependency of package C. The big difference is that it doesn’t require project B to have a direct reference to package D.

If you want to upgrade to a different version of package D, you can do this by simply adding a direct reference to it to project B and specifying the version you need. Direct dependencies take precedence over indirect ones, and when packages are restored, .NET Core will resolve the versions correctly.

<PackageReference Include="C" Version="1.0.1" />
<PackageReference Include="D" Version="1.50.2.0" />

From my experiments, it did not seem to be necessary to have any binding redirects. However, when that does turn out to be necessary, you can simply enable some properties to instruct .NET Core to generate the binding redirects for you automatically!

<PropertyGroup>
  <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
  <GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>
</PropertyGroup>

Conclusion

Even with NuGet (or maybe especially with NuGet), version hell is still alive and kicking. Do yourself a favor and upgrade to the .NET Core build pipeline. It comes with so many other benefits (such as not having to update your .csproj when you add or remove a file), and this is just an added bonus.

Pretty much any application needs logging. Except maybe the simplest of applications that do not require user interaction and only do one incredibly specific thing. Or applications that only interact via the console. Pretty much any other application needs logging, though. When things are on fire, you need to be able to see what has happened. Maybe it’s the trigger to let you know that something is on fire.

All too often, though, the code that does the logging looks horrible.

_logger.Info("Created the order");

That’s just peachy. Which order? For what customer? What does it contain? This does not help me at all when trying to pin down the cause of an issue (or even just the issue itself). This also extends to error reporting, by the way. It’s not very helpful to see a message like this in your logs:

2017-05-18T16:42:15Z [Critical] An error occurred: System.InvalidOperationException: Sequence contains no elements
  at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
  at EnterpriseySystem.Common.BetterThanEntityFramework.DoTheQuery[T](MyQueryable`1 query)
  at EnterpriseySystem.DataLayer.EverythingRepository.SaveChanges()
  at EnterpriseySystem.Web.MagicUnitOfWorkFilter.OnActionExecuted(ActionExecutedContext context)

Now, of course, this is exaggerated a bit. Nevertheless, the point should come across. About the only useful factoids you can get from this message are that ‘something went wrong before, during or after the execution of a query’ and ‘it’s something related to saving changes’. It doesn’t tell you what sort of thing it was saving, what the identity of that thing was, or even what the request URL was.

So, let’s add some properties to our logging.

Properties

Properties start to give us some context and can make our lives a lot easier.

2017-05-18T16:42:15Z [Critical] An error occurred gathering changes preparing to create an order for customer 376572, containing products [ 2352, 23422, 9482 ]: System.InvalidOperationException: Sequence contains no elements
  at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
  at EnterpriseySystem.Common.BetterThanEntityFramework.DoTheQuery[T](MyQueryable`1 query)
  at EnterpriseySystem.DataLayer.EverythingRepository.SaveChanges()
  at EnterpriseySystem.Web.MagicUnitOfWorkFilter.OnActionExecuted(ActionExecutedContext context)

Now this is something useful. We know the system is trying to create a new order, and which customer it is for, and which products it contains. We also know it’s in the part of the ORM that is gathering changes. This allows us to very quickly focus our search for the issue.

However, think about what a dumpster fire the logging code for this will become.

_logger.Critical(
  ex, 
  "An error occurred gathering changes preparing to create an order " + 
  "for customer {0}, containing products [ {1} ]",
  customer.CustomerId,
  string.Join(", ", order.Lines.Select(l => l.ProductId)));

And this is if I try to make it look nice. Usually it’ll be one giant line of text and parameters. There is another problem, however. What if, during my troubleshooting, I want to find all messages relating to customer 376572? I can search for customer 376572, but that will not find the following message:

2017-05-18T15:32:15Z [Info] Removed address record. AddressId = 8748323, CustomerId = 376572

Clearly we need a better solution. Enter structured logging.

Structured logging

Structured logging makes data a first-class citizen of your logged messages, rather than something you have to pull out using regular expressions and analyzers.

Traditionally, a logged message is, at its very core, simply a couple of lines of text. Those lines of text might include a lot of commonly available data, such as a timestamp, a level (Information, Warning, Error, etc.), and other application-specific properties, but it is, for all intents and purposes, just a string.

Structured logging turns a log message into an object, with properties. Let’s take a look at how that same exception message from earlier might look when using structured logging. This is a raw message dump from Seq (a structured logging back-end).

{
    "@t": "2017-05-26T12:06:09.6427612Z",
    "@mt": "An error occurred gathering changes preparing to create an order for customer {CustomerId}, containing products {OrderProducts}",
    "@m": "An error occurred gathering changes preparing to create an order for customer 376572, containing products [ 2352, 23422, 9482 ]",
    "@i": "9060ab2b",
    "@l": "Critical",
    "@x": "System.InvalidOperationException: Sequence contains no elements\r\n  at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)\r\n  at EnterpriseySystem.Common.BetterThanEntityFramework.DoTheQuery[T](MyQueryable`1 query)\r\n  at EnterpriseySystem.DataLayer.EverythingRepository.SaveChanges()\r\n  at EnterpriseySystem.Web.MagicUnitOfWorkFilter.OnActionExecuted(ActionExecutedContext context)",
    "CustomerId": 376572,
    "OrderProducts": [
        2352,
        23422,
        9482
    ],
    "Environment": "Development",
    "HttpRemoteHost": "::1",
    "HttpUri": "http://myawesomeservice.mycompany.io/order",
    "HttpMethod": "POST",
    "ConnectionId": "0HL36VOIBEH51"
}

You can see that the customer ID is not only available in the ‘rendered message’ (the @m property), but also as a separate property called Customer​Id. Because it’s available as a separate piece of data, you can suddenly do a bunch of interesting things with it. For example, if you have a big collection of log messages like this, you could search for only those related to customer 376572. Or maybe for messages mentioning product 2352. Or you could search for this specific exception and then graph the product count into a histogram, to see if there is any correlation.

Notice there are also properties on the message related to the current execution context, such as Http​Uri and Connection​Id. The better logging frameworks allow you to set ‘ambient’ properties. These can be valid for the lifetime of the application, like the hosting machine name, or only valid for a limited scope, like a HTTP request. All log messages generated while processing that HTTP request will have those properties, without you having to explicitly add them to each log message.

Of course you need a specialized logging back-end to be able to do something useful with these properties. You’ll need a searchable document store and a front-end to display messages and render graphs. There are many currently available, but some of the big names right now are Elastic Stack, Seq and Splunk.

Consistent property names

Something you’ll quickly run into when using structured logging is the consistency of parameter names. If one piece of code logs the products as Products and another logs them as Order​Products, it’ll become a lot more difficult to reliably use those properties.

You can work around this by putting the property name on a static class that‘s used everywhere. Like so:

internal static class LogPropertyNames
{
  public const string OrderProducts = "{OrderProducts}";
  public const string CustomerId = "{CustomerId}";
}

public void CreateOrder(Order order)
{
  _logger.Information(
    $"Creating an order for customer {LogPropertyNames.CustomerId} " +
    $"with products {LogPropertyNames.OrderProducts}.",
    order.CustomerId,
    order.ProductIds);
}

This works pretty nicely. As long as you maintain the convention that all property names must be maintained in the Log​Property​Names class, it’s pretty easy to ensure consistent property names.

Still, it’s quite a bit of code. When you’re just perusing a method, this bit of code shouts at you. And this is just a very basic piece of code. You’ll often have many more properties to log, and sometimes it is a Debug or Trace message. You’ll only want to log those when there is actually a sink or target configured to accept those levels, because logging messages consumes time and memory.

Time to refactor this into a method.

public void CreateOrder(Order order)
{
  LogCreatingOrder(order);
}

private void LogCreatingOrder(Order order)
{
  if (_logger.IsInformationEnabled)
  {
    _logger.Information(
      $"Creating an order for customer {LogPropertyNames.CustomerId} " +
      $"with products {LogPropertyNames.OrderProducts}.",
      order.CustomerId,
      order.Products.Select(p => p.ProductId));
  }
}

Much better! I think we can improve even further. I don’t really like the fact that your classes will become littered with many methods that are still pretty sizable, just so we can do logging. How about we move those somewhere else?

Meaningful Logging

internal static class LoggerExtensions
{
  public static void LogCreatingOrder(this ILogger logger, Order order)
  {
    if (logger.IsInformationEnabled)
    {
      logger.Information(
        "Creating an order for customer {CustomerId} " +
        "with products {OrderProducts}.",
        order.CustomerId,
        order.Products.Select(p => p.ProductId));
    } 
  }
}

public void CreateOrder(Order order)
{
  _logger.LogCreatingOrder(order);
}

Now we actually have strongly typed and meaningful logging methods on our logging interface. Note how I removed the usage of Log​Property​Names; if all your actual logging code is in one file, it’s dead simple to ensure consistency of your property names, so you don’t really need the static class any more. This is starting to look pretty nice!

Metrics

Along with logging, most non-interactive applications, like web applications, Windows services, or web APIs also need metrics: raw numeric points of data that give you an insight into how well the application is performing and whether it is healthy. I think it’s relevant to see these as different from logging. Yes, with structured logging you can output your metrics as part of logging and use those to create graphs, but in my experience the tooling of dedicated metrics applications, like Datadog or Librato, is that much better at managing alerts and just processing large quantities of time-series data, that it warrants treating them as a separate concern.

But oftentimes, logging and metrics go hand in hand. You created a user account? Write a log message and increase a counter. You finished processing a HTTP request? Write a log message and add the elapsed time to a timer metric. Imaging the mess if you use the ‘old’ style of writing logging code and metrics code in between your other code.

Note how usually log messages and metrics are triggered by ‘events‘: a user was created, an error occurred, an HTTP request was processed. It would be nice if your code would reflect this. Instead of being explicit and talking about ‘logging this’ and ‘outputting that metric’, we can talk about events that happen. They are not application events or domain events, however.

Monitoring events

I’d like to propose the term monitoring events, because you usually use the output of the events (such as logging and metrics) to monitor the performance and health of your application. It’s also unambiguously different from the other kinds of events I mentioned.

It is, simply put, an API for your application code to whatever logging, metrics, alerting, or other monitoring solution you have. In my current solutions, it takes the form of a class, like this:

public class MonitoringEvents
{
  ILogger Logger { get; }
  IMetrics Metrics { get; }
  ILogContext LogContext { get; }
}

IMetrics and ILog​Context define a number of operations to do with metrics and ambient properties, respectively. ILogger is Serilog’s ILogger interface. The actual methods are defined as extension methods and look like this:

internal static class MonitoringEventsExtensions
{
  public static void UserCreated(this MonitoringEvents monitoringEvents, User user)
  {
    monitoringEvents.Logger.Information("Created user {UserId}, named {Username}.",
                                        user.UserId,
                                        user.Username);
    monitoringEvents.Metrics.IncrementCounter("users.created");
  }
}

In the consuming code, it looks like this:

void CreateUser()
{
  // do complicated business stuff
  var user = new User();

  MonitoringEvents.UserCreated(user);
}

In this example, Monitoring​Events looks like it’s a property. That’s because it is. It could of course be a field that gets injected via the constructor, but in conversations with my colleagues we decided that this kind of practice leads to ‘constructor’ bloat; it gets very hard to see what the actual dependencies of a class are, and which are the infrastructural dependencies.

For that reason, we opted to expose our interface using a property which gets set by property injection.

This means your actual business logic code, which is supposed to be the raison d’être for your application, can be more about actual business logic, and less about cross-cutting concerns like monitoring. What you log and which metrics you keep are decoupled from the business code. I think that’s pretty nice with regards to the Single Responsibility Principle, which is not really about responsibilities but about reasons for change. If you do decide to change to a different logging framework (but in my humble opinion, at the moment there is nothing better than Serilog), the only places you have to make changes are in your bootstrapping code, in your interface, and in your extension methods.

I really like this approach. In a lot of applications, logging and monitoring is added as an afterthought, if at all, and because there is usually little perceived business value in logging and metrics, it is treated like a red-headed step-child; its code is often sloppy, ugly, and hard to read. Please, think of the step-child.

Last week I wrote about the new features that Microsoft have planned for the next version of C#, version 7.1. Let’s also take a look at the things they have planned for a bit further out. Strap yourselves in, because this will be a rather long read.

C# 7.2 and 7.3

The next-up versions of the C# language, 7.2 and 7.3, are less clearly defined than 7.1. C# 7.2 will be focused on enabling you to write low-level code without having to resort to the unsafe keyword. There are some features already defined, like read-only references, blittables, and ref-like types. For C# 7.3, the picture is even less clear. The roadmap slide in the BUILD 2017 presentation only mentions ‘Next steps for pattern matching?’.

The presentation also touches on some of the features they’re thinking about for C# 8.0, so let’s have a look at those. I think they’re pretty exciting, but because this is much further out in terms of planning and releasing, many things are still uncertain. Features might change or might not make it at all.

Asynchronous sequences (C# 8.0)

C# 5.0 was all about async and await, as we know. However, one of the scenarios that was left unsupported is enumeration (you know, foreach). In order to use foreach (or the entirety of LINQ) with the result of an asynchronous method, you have to either retrieve all of the results at once, asynchronously, or be content with the fact that enumeration is not asynchronous.

There is a proposal to support this in the language. It would look something like this:

IAsyncEnumerable<SearchResult> results = 
    searchEngine.GetAllResults(query);

foreach await (var result in results) { // ... }

Now this looks simple enough, but for proper support they would also have to support all of this in LINQ query operators, which is pretty large body of code. They could probably use a lot of the work from System.​Interactive, from the Rx project. There’s not a lot of official documentation there, but Bart de Smet’s blog has some interesting information.

For a lot of scenarios, like querying a database, your data provider would also have to support this scenario. It will probably be quite some time before third party data providers will start supporting this, if at all. I’m looking at you, Oracle. The official Oracle driver does not, to this day, support asynchronous operations at all. Never mind all the Async methods returning Task; they don’t even support the old Begin/End pattern of asynchronous operations.

Regardless of the third party buy-in this feature needs in order to be really useful in day-to-day code, it is very nice to see a way you can asynchronously stream in a large set of items of an unknown size, like, for example, a Twitter feed. Right now you have to retrieve the feed page by page. This should be an implementation detail of the Twitter client you’re using and it should not be represented in your code. With asynchronous sequences, you can abstract away this detail, which is very nice.

Asynchronous Dispose (C# 8.0)

Let’s say you have a desktop application that is directly connecting to a database. Yes, I know it’s 2017, but just go along for the example. You begin a transaction and start doing a lot of work there, all asynchronously so you don’t block the UI. Of course your transaction is initialized in a using statement, so that when an exception occurs, it is neatly disposed of, which in the case of a database transaction means it’s rolled back.

If your transaction has affected a large number of records, rolling back might take a while. And since Dispose is a synchronous call, this means your user interface will be frozen while this is happening. Yes, you can do something like this:

IDbTransaction transaction = null;

try
{
    transaction = connection.BeginTransaction();

    // do important work

    transaction.Commit();
}
finally
{
    await Task.Run(() => transaction.Dispose());
}

... but that’s basically hacking around a shortcoming in the Disposable pattern. A feature is proposed for, at this point, C# 8.0, which would make this a lot easier.

using await (var transaction = connection.BeginTransaction())
{
    // do important work

    transaction.Commit();
}

Again, the usefulness of this feature probably depends a lot on third party buy-in for the IAsync​Disposable interface that will be added.

Extension everything (C# 8.0)

This is one I’m pretty excited about. You’re already able to write extension methods that extend a class’s functionality without having to modify it. That’s it though. You’re not able to add static methods or properties of any kind.

The proposal adds new syntax for defining extensions, which makes it possible for you to add, as the feature suggests, anything to a type. The obvious ones are things like instance properties and static methods and properties, but the slide shown at the presentation also shows a static field. Although the slide does not mention it, the proposal mentions that they would (eventually) be able to support instance fields, using the Conditional​Weak​Table class.

Supporting instance fields would mean you would be able to attach a whole new set of features to an existing object, without having to modify it and without the object being able to interfere with that functionality. It sounds nice, but I have some reservations about it. On the one hand, it would be cleaner and easier on the eyes than using composition. On the other hand, like with extension methods, it’s sometimes not always clear what is happening when you’re just looking at the code. You pretty much need an IDE to be able to figure out that something is an extension. Probably it’s best to use this feature sparingly, only when it really makes sense.

The syntax looks a bit Java-ish (or TypeScript-ish), but remember, it’s not final, so it might still improve.

extension Enrollee extends Person
{
    // static field
    static Dictionary<Person, Professor> enrollees = 
        new Dictionary<Person, Professor>();

    // instance method
    public void Enroll(Professor supervisor) =>
        enrollees[this] = supervisor;

    // instance property
    public Professor Supervisor =>
        enrollees.TryGetValue(this, out var supervisor) 
            ? supervisor
            : null;

    // static property
    public static ICollection<Person> Students => enrollees.Keys;

    // instance constructor
    public Person(string name, Professor supervisor)
        : this(name)
    {
        this.Enroll(supervisor);
    }
}

This is a direct transcription of the slide from the BUILD 2017 talk.

Records (C# 8.0)

I can be very short about this: this is awesome. A record type is nothing more than a collection of fields. You only specify the types and names of the fields, and the compiler will do the tedious work of implementing those the right way. The syntax is very simple.

class Person(string First, string Last);

When you see what the compiler expands it to, you’ll see all of the boilerplate you no longer have to implement.

class Person: IEquatable<Person>
{
    public string First { get; }
    public string Last { get; }

    public Person(string First, string Last)
    {
        this.First = First;
        this.Last = Last;
    }

    public void Deconstruct(out string First, out string Last)
    {
        First = this.First;
        Last = this.Last;
    }

    public bool Equals(Person other) =>
        other != null && 
        Equals(First, other.First) && 
        Equals(Last, other.Last);

    public override bool Equals(object other) =>
        (other as Person)?.Equals(this) == true;

    public override int GetHashCode() =>
        (First?.GetHashCode() * 17 + 
         Last?.GetHashCode())
        .GetValueOrDefault();

    public Person With(string First = this.First, 
                       string Last = this.Last) => 
        new Person(First, Last);
}

That is about 33 lines of code you don’t have to write in order to get a nicely functioning DTO class. As I often say, the best line of code is the one you don’t have to write.

Default interface implementations (C# 8.0)

When this feature was introduced, I was pretty skeptical. Why would you ever add implementations to an interface? Why wouldn’t you use an abstract class for that?

The case made in the presentation was a pretty good one, however. Let’s look at a logging library from two perspectives, namely from that of the client application writing log messages and from that of the third party extension library that is implementing a new feature, like logging to FTP. Yes, I know that is a stupid idea. Just go along.

The full interface looks like this:

public interface ILogger
{
    void Log(LogLevel level, string message);
    void Log(LogLevel level, string format, params obj[] arguments);
    void Debug(string message);
    void Debug(string format, params obj[] arguments);
    void Information(string message);
    void Information(string format, params obj[] arguments);
    void Warning(string message);
    void Warning(string format, params obj[] arguments);
    void Error(string message);
    void Error(string format, params obj[] arguments);
}

Now, from the client application’s perspective, the most interesting methods are those with a specific log level, such as Debug, Information, Warning, and Error, and their overloads. Possibly the Log methods are also interesting, but less so.

From the FTP logging feature’s perspective, the only interesting method is the Log(LogLevel, string) method; all of the other methods are basically convenience overloads of that method.

Currently, because it is an interface, an implementation of ILogger must implement all of the methods in the interface. If we add a new method, for example void Error(Exception ex, string format, params object[] arguments), we have broken the contract. All implementations must now be changed to also implement this method.

With default interface implementations, we can define the implementations of the ‘overloads’ in the interface, so that only the Log(LogLevel, string) method is mandatory to implement. It looks like this (edited for brevity):

public interface ILogger
{
    void Log(LogLevel level, string message);

    void Log(LogLevel level, string format, params object[] arguments)
    {
        Log(level, string.Format(format, arguments));
    }

    void Debug(string message)
    {
        Log(LogLevel.Debug, message);
    }

    void Debug(string format, params object[] arguments)
    {
        Log(LogLevel.Debug, string.Format(format, arguments));
    }
}

How this technically works (from what I understand) is that the methods that are actually implemented become virtual methods on the interface. An implementation can override them if it so chooses, but it doesn’t have to. The primary reason, then, for default interface implementations is being able to extend an interface without breaking backwards compatibility. Pretty good stuff.

Nullable reference types

In 1965, the concept of null was first introduced in the Algol W language by Sir Tony Hoare. He famously described this himself as his ‘billion dollar mistake’ and publicly apologized for it. Whether someone else wouldn’t have introduced null if he hadn’t is debatable, but it is undeniable that null reference errors are the cause of many errors.

It would be great if we could make sure that a parameter or property can never be assigned the value null. While there are some partial solutions in C#, such as using Code Contracts or Fody NullGuard, the community has been asking for years for a first-class solution to this problem.

Some have suggested being able to mark a type as not nullable, using !. A variable string a would be nullable, but string! a would not be. If you did attempt to assign null to non-nullable variable, or assign a value from a nullable variable without checking it for null, you would get a compiler error. Now, for variables, the compiler could solve this pretty easily, but for parameters or properties, they would have to find a way to add extra metadata. And there are still cases they couldn’t solve that way, like newing up an array (where all the elements are initially null). They would have to drastically change the .NET type system, which would break a lot of existing code.

So immediately, we are reduced to only generating warnings when doing something that might result in a null reference error. The C# team have, for now, chosen to do the opposite of marking a variable as non-nullable; all reference types will become non-nullable by default, and you can mark the type of a variable as ‘null-safe’ by decorating them with ?, similar to nullable value types. Using a non-nullable variable that might be null (because you didn’t check if it was yet) will result in a warning, as will assigning the value of a nullable variable to one that is non-nullable.

What does this look like?

// bar is nullable because its type is string?
void Foo(string? bar)
{
    string baz = bar; 
    // this will generate a warning because baz is non-nullable 
    // but bar is nullable, and we haven’t checked bar 
    // to not be null
}

To fix this, we simply have to check for null.

void Foo(string? bar)
{
    if (bar == null)
        throw new ArgumentNullException(nameof(bar));

    string baz = bar; 
}

This behavior will be opt-in, because you can be sure this will generate tons of warnings for existing code bases. I think this is a great stride forward in making C# an even more safe language and, for a large part, getting rid of one of the most prolific errors in software.

Summary

Microsoft is really stepping up the language enhancements, and it is great to see that they are so transparent and open about this. If you have something to say about these changes, you can hop on over to the Roslyn site at GitHub and say it! If you make a strong enough case, you might even effect change.

What do you guys think about these new features? Let me know in the comments. For now, that is the final word on this round of new C# features.

Last year I talked about the new features of C# 7.0. Visual Studio 2017 has been released a few months ago, so you can now enjoy using these features in your day-to-day work. But Microsoft has, of course, not been idle in the meantime. They’ve detailed what they were, and are, working on in a presentation at BUILD 2017 called The Future of C#.

They explain how they will move forward from C# 7.0. Microsoft will actually start doing ‘point releases’ of the C# language and the compiler. This means they can release incremental improvements as updates to Visual Studio and the build tools. Finally, we won’t have to wait until the Microsoft gods favor us and we are blessed with a new major release!

It also means it might get a bit harder to figure out if a particular language feature is supported. Before, you could simply ask ‘are you using Visual Studio (insert year)?’, which would immediately translate into a C# version. Now, you have to find out which specific update of Visual Studio or the build tools is installed. On the other hand, why would you not install an update?

Still, on the whole, I think it’s great that we get these ‘feature updates’ during the lifetime of a Visual Studio release.

All of the features that I’m going to discuss are still in draft; they might not make it at all or only in a later version.

Discards

Well, except this one. This is a feature that I did not discuss in my previous post, but it’s actually already in C# 7.0. A discard is an indication that you are not interested in using a parameter or declared variable.

Imagine that you have a function that returns a tuple (int count, int sum). You are only interested in the count field. Without discards, you would have to do something like this:

(int count, int dummy) = Tally();

Now, of course, this works, but it’s very impractical. If you have more than one of these in a single the first one will declare the variable dummy, while the second one will only use it.

(int count, int dummy) = Tally();
(int average, dummy) = GetStatistics();

Your IDE will probably complain that the value assigned to dummy is not being used in any execution path. Not very pretty. And it gets even worse when you have several ‘dummies’ of different types. dummy1, dummy2, dummy3…

Discards solve this by basically making _ a keyword. When you use _, you’re telling the compiler ‘I don’t care about this, don’t even bother creating a variable for it.’ This was already possible for delegate arguments, but when you wanted to ignore multiple arguments, you (obviously) couldn’t reuse names, so you got something like (_, __, ___). Ugly. Since discards are a special case, you can reuse them. It’s much prettier and more readable.

// in deconstruction
(int count, _) = Tally();

// in delegate parameters
var usersWithPosts = 
    context.Users
           .Join(context.Posts,
                 user => user.UserId,
                 post => post.UserId,
                 (user, _) => user);

// in out arguments
int.TryParse(input, out _);

// in pattern matching
switch (input)
{
    case int number:
        Add(number);
        break;
    case object[] _:
        // ignore
        break;
}

Thank you, Microsoft, for letting us not care about stuff.

async Task Main() (C# 7.1)

The async and await keywords, introduced in C# 5.0, are an awesome solution to the headache that is asynchronous programming. They make it very easy to write code that calls out to external systems, or does I/O, without blocking a thread. This makes web applications scalable, and keeps UI applications responsive.

However, once a part of your code is asynchronous, everything that’s calling into that code also needs to become asynchronous. When your hosting application is a console application or Windows service, this is a bit of a problem. You can write async void Main, but this means that as soon as you hit your first await, the entire application terminates. This is because async void is a fire-and-forget mechanism. Since you cannot return a Task from Main, you will have to use something like AsyncContext from Stephen Cleary’s excellent AsyncEx library.

Soon, that will no longer be necessary. The compiler and the framework will recognize and properly execute an async Task Main method.

Better null type inference (rumored - C# 7.1?)

Have you ever written a ternary expression that returns a nullable value type and one of the branches returns null? It sucks.

int? x = flag ? 1 : (int?) null;

Why do I need to specify the type explicitly? If you don’t, you get an error stating that there is no implicit conversion between 'int' and '<null>'. In a future version of C#, you won’t have to specify the type explicitly any more.

int? x = flag ? 1 : null;

Much nicer.

default expression (C# 7.1¹)

This is simply an improvement on something that has been supported since C# 2.0 and the advent of generics, back in 2005. It came in the form of default(T), where T is of course a type. For reference types, it represents null, and for value types such as int, it represents that type’s default, uninitialized, value.

The new feature is that, when the type of T can be inferred, you don’t need to specify it any more.

Instead of writing this:

int x = default(int);

... you can write this:

int x = default;

Of course this also extends to other usages of default, such as return statements, default values for arguments, and, expanding on the previous point, ternary expressions. This will be valid in a future version of C#:

var x = flag ? 1 : default;

That is pretty clean.

Tuple projection initializers (C# 7.1)

I can’t decide if this feature’s name sounds awesome or if it is simply incomprehensible. Anyway, remember tuples from C# 7.0? Let’s say you want to create a tuple (string firstName, string lastName). And let’s say that you already have local variables called firstName and lastName. Right now, you have to write something like this:

var t = (firstName: firstName, lastName: lastName);

This feature infers the names of the tuple elements from their initialization expression, similar to what you can do with anonymous types.

var t = (firstName, lastName);

Console.WriteLine($"Hello, {t.firstName}!");

Simple, and very effective; I like it. One wonders why this wasn’t in there to begin with.

Pattern matching with generics (C# 7.1)

This is another one of those ‘make a good feature even more useful’ things. The whole point of pattern matching is so instead of this:

var objString = obj as string;
if (objString != null)
{
    Console.WriteLine($"String: {objString}");
}
else if (obj is int)
{
    var objInt = (int) obj;
    Console.WriteLine($"Int: {objInt}");
}

... you can write this:

switch (obj)
{
    case string s:
        Console.WriteLine($"String: {s}");
        break;
    case int i:
        Console.WriteLine($"Int: {i}");
        break;
}

There was a limitation in C# 7.0, however. Consider the following code:

class MyList<T>
{
    public void Add(T item)
    {
        var stringItem = item as string;
        if (stringItem != null)
        {
            Console.WriteLine($"String: {stringItem}");
        }
    }    
}

If we use as to determine what kind of item it is, that will compile. However, if we use pattern matching with is (or case), it will not.

public void Add(T item)
{
    if (item is string s)
    {
        Console.WriteLine($"String: {s}");
    }
}

Pattern matching in C# 7.0 has a requirement stating that there must be an explicit or implicit conversion from the left-hand-side type (T) to the right-hand-side type (string). When either is an open type, the compiler determines that this might not be the case. For instance, the closed type for T might be int, and there is obviously no implicit or explicit conversion from int to string. The C# team have determined this is an omission in the spec. In C# 7.1, the spec will be expanded so that either the left-hand-side or the right-hand-side can be an open type.

But wait, there’s more!

As said, the feature list for C# 7.1 is not yet definitive, but the things Microsoft have lined up right now look very promising and will be sure to add to the expressiveness of C# and, by doing so, make it just a little bit easier to write readable and maintainable code. And isn’t that what we all want?

What do you think of the improvements that are being made? Let me know in the comments.

Next week, we’ll look at some of the more speculative, but also more radical, improvements that are currently scheduled to be included in C# 8.0.

In my last post, about CQRS and how to keep your read model synchronized, I mentioned how you could ‘broadcast’ your events to other parts of the system in order to keep read-optimized representations of your application’s state in sync with the actual state. You use an ‘event bus’ that will distribute an event to all interested parties.

This event bus can be stupidly simple; it could be a list of every read model in the system and publishing could consist of broadcasting every event to every read model in the system. You could make it a little more intelligent by only distributing an event to a read model that is interested in it. What if updating your read-optimized representation means running some heavy queries against a database and then inserting or updating a ton of records? You don’t want to make your users wait for those updates, right? That’s why it’s often a good idea to make your event bus asynchronous. This doesn’t mean the method broadcasting your events returns a Task, it means ‘broadcasting’ the events after you save them is actually putting them on a queue to be broadcast. Your save immediately returns, and the events will get processed in the background.

Eventual Consistency

If your ‘event bus’ is asynchronous, you will have to come to grips with the realities of Eventual Consistency.

This is a statement I made in my previous post. What does Eventual Consistency mean? Consider the following: you are writing a blogging application (because at one point or another, we all write a blogging application). One of your requirements is that after the user has finished writing a blog post, he is redirected to the overview of blog posts. However, for whatever reason, it takes a couple of seconds to update the list of blog posts — or maybe just a single second, or several hundreds of milliseconds — which is probably way longer than the time it takes for the user to be redirected and for the application to request a list of blog items from the read model, which is still updating the database.

The result, in the end, is that the user’s blog post is not in the overview. What the hell? The user spends several moments in confusion. Did the blog post get created or not? They press F5. Lo and behold, there’s the blog post.

In the moments after creating the blog post, the system was ‘inconsistent’; that is, not all views of the system (or read models, if you will) were returning the same data. A read model that is eventually consistent will regularly ‘fall out of sync’ with the underlying data, as changes are being processed in the background. Given enough time, it will catch up with all the changes, and it will be consistent again. Eventually, it will be consistent.

Observing readers will notice I wrote ‘a read model that is eventually consistent’. If you use an asynchronous event bus, then yes, technically your entire application is eventually consistent. However, if updating a read model takes less time than it takes for the user to round-trip a request, then the read model will never appear to be out of sync. For all intents and purposes, it is not eventually consistent but immediately consistent. Hence the nuance: sometimes eventual consistency applies only to several read models, but the rest of the application is not affected.

What to do about it

There is a number of things you can do when a view or read model is Eventually Consistent.

Ignore it

This means making it the user’s problem. In general this will annoy the hell out of users. This might be a perfectly valid solution, depending on the application. If the user’s action was creating a blog post and it doesn’t show up in their list of blog posts, then I’m sure you agree that it’s not a very good solution.

Notify the user

Your blog post has been created and is being processed. It may take several minutes for it to become available.

This is the ‘fastest’ solution. It requires the least amount of modifications and your application remains responsive. It also requires the user to trust your application. If the read model encounters a problem when processing events, there is, by default, nothing to notify the user. They check after a few minutes, but there is still nothing there.

If this happens often enough, users will stop trusting the application. Which is obviously a bad thing.

If you do go this route, it would probably be a good idea to add a system which will notify users (and possibly administrators) if a read model encounters an error, possibly by sending them an e-mail.

Wait for the view or read model to become consistent again

After the blog post has been created, before redirecting the user to the overview, your application can wait, polling the read model every once in a while (probably every hundred milliseconds or so) until the newly created blog post is listed. Only then will you actually redirect the user to their overview. To the user, it appears as if creating the blog post takes longer than it actually does. Another possibility is to wait when you’re requesting the overview of blog posts.

If the event is still in the queue, waiting to be processed because there is a high workload, that might cause your application to wait for an unreasonably long time before the read model is updated.

Fake it

This is not always a possibility, but one way to make an application seem responsive is to ‘fake’ stuff. Of course I’m not talking about actually faking the creation of a blog post. After the blog post has successfully been created, we redirect the user to their blog post overview, supplying it with a parameter that indicates ‘the user just created a blog post with ID such-and-such.’

The overview page will then request a list of blog posts from the read model and check if that blog post is among the results. If it is; great, just do the regular thing and show the results. If it isn’t, get information about the blog post from the system and display it among the results as if it were a regular result. The user will be none the wiser.

Actually, he might be. If the user navigates away and reaches the overview page some other way, chances are he won’t see his newly minted post any more, because the overview page doesn’t receive the ‘newly created post ID’ parameter any more.

You might ask: how does the overview page get information about the new blog post? Doesn’t it need to wait for a read model to become consistent? Why, yes it does. A way to solve this is to create a separate queue for a special read model.

Every event broadcast is put on the ‘regular queue’ and this special queue. The read model could store its information in-memory and contain only the most basic information about pretty much all blog posts ever created. Because it’s in-memory, the queue would never be very long, so the time to become consistent would be virtually zero.

All the posts, though. Hmm, that might still be a lot of information. How about only the most recent x blog posts? Then we don’t even need the passing around of parameters and the view would remain consistent even if you navigated away! You could even tweak how many posts it keeps in memory based on how busy the application is and how long it takes for read models to become consistent! There is a slightly better way, though.

Discriminate against your read models

If you have the ability to create and broadcast to multiple queues, why not maintain the normal read models, but separate them into ‘high speed’ (or ‘high priority’) and ‘low speed’ (or, conversely, ‘low priority’)? You could make sure that all read models that need to update in near real-time feed off the high-speed queue, while all the other read models feed off the low-speed queue.

If you have a slow database, you might introduce a cache. The high-speed read models update into and read from the cache. The low-speed versions of those read models write into and read from the database. The cache-backed read models can use the database-backed read models as a fall-back, so that when your application has just started up, you don’t need to replay all the events in the universe to get a working application.

Fully parallel processing

Having a low-speed and a high-speed queue solves the issue of having slow read models interfering with near real-time read models. However, it’s still possible for badly behaving read models to interfere with one another.

A couple of great minds have written a manifesto, the Reactive Manifesto, which deals with how systems should be designed to make them “more robust, more resilient, more flexible and better positioned to meet modern demands.” One of the characteristics of a reactive system is as follows:

Failures are contained within each component, isolating components from each other and thereby ensuring that parts of the system can fail and recover without compromising the system as a whole.

What that means for our read models is that each read model should be as isolated and self-contained as they can be. A way to achieve this is to have each read model have their own queue of events. A failure or delay in one read model has very little chance of interfering with other read models. One read model might not be consistent because it is failing, but at least the others are likely to be.

It also decouples the read models on a temporal level. Each read model can process events as fast as possible, so the fastest, in-memory, read models are almost immediately up-to-date, while the slow read models, that update reporting databases, are allowed to lag behind and catch up when there is less load.

By isolating each read model, it is also possible to scale out read models independently. An in-memory read model that merely copies data from events to a state representation probably has little to gain from being scaled out. A read model that does a lot of heavy processing probably has a lot to gain.

Beware of the queue

A note about using queues, though. A friend and former colleague of mine would probably have my head on a pike if I didn’t mention the danger of using queues. Even though memory and disk space are ‘virtually limitless’, in practice they have very real limits. When messages are being put onto the queue faster than they are pulled off and processed, you will have a problem. Eventually, the queue will be full (because there is no more memory or disk space) and you can’t put new messages onto the queue. What should your application do in that case?

You could discard messages. This is known as load-shedding. For events, it means that, at the very least, some information is missing in some parts of your system. At the very worst, a critical piece of information for a critical business process is missing and your system shits itself and breaks down.

You could also block the write side until there is room on the queue. This is known as back-pressure. At least the system will remain consistent (eventually), but users end up waiting a for a while whenever they do something, possibly getting a time-out.

A much better write-up of this phenomenon is at ferd.ca.

Actor Model

Independent queues (or mailboxes, if you will), isolation, fault tolerance... Those familiar with the pattern (and that have made it this far) have probably been shouting it for a while now: this is the Actor Model!

I won’t go into too much detail about the Actor Model, because it is a huge and complex topic. The Actor Model is a pattern that lets you create highly concurrent and fault-tolerant systems. An Actor Model system is comprised of ‘actors’ who are driven by messages and can, in response to a message, only do a limited number of things:

• Create more actors
• Send messages to other actors
• Modify their own state
• Modify how they will respond to future events

An actor only knows how to reach other actors by their address; they don’t have a direct reference to them. The reason for this is location transparency; an actor doesn’t have to know whether another actor lives inside the same process, another process or maybe another machine altogether.

The Actor Model is an insanely powerful way to model your system and achieve resilience and concurrency. It is also calls for a very different way of thinking about and designing your system; even more so than Event Sourcing.

If you want to know more about the Actor Model, I suggest you take a look at Petabridge’s Akka.NET Bootcamp. Akka.NET is an Actor Model framework for .NET and it’s a port of the Akka framework for Java.

Responding to Events

Eventual Consistency towards the user is one thing, but you can also encounter Eventual Consistency internally. Let’s say you have an event handler that needs to respond to an event X occurring by storing and publishing a different kind of event. To create this new event, you need data from a read model. That read model depends on the data from event X to become fully consistent. The event handler receives the event, but the read model you need to use is not yet up-to-date. What to do?

When you have a fully parallel event bus, where each event handler — which an read model is in its core — has its own queue, the solution is pretty simple. When you receive a message and the read model you depend on doesn’t yet have all the data you need, put the message back on the back of the queue. This is called stashing in the Actor Model. Depending on how busy your system is and how quickly the read model you depend on becomes consistent, you might re-encounter the event several times before you are able to successfully process it. On the other hand, you might not need to make any changes to the read model you depend on and only very little changes to the event handler.

In the case of a (partially) sequential event bus, the problem is much more tedious to solve. You might be able to change the order in which event handlers are invoked, but this is very brittle. You could surround the code that controls the order with large exclamatory comments like /**** DON'T CHANGE THIS CODE OR IT WILL BREAK TEMPORAL COUPLING IN EVENT HANDLERS... YOU PROBABLY WOULDN'T UNDERSTAND ANYWAY ****/. This does not guarantee that someone won’t try and do it, though.

If you’ve implemented low-speed and high-speed queues, you might ensure that the event handler and the read model are on different queues and wait inside the event handler until the read model is consistent, but you will also block everything else on that queue.

You could introduce a new queue specifically for this event handler, but if it blocks until the read model becomes consistent, you might have a pretty low throughput. If the frequency of the event is pretty low, that might be acceptable. When you are able to ‘stash’ events, the throughput will improve. It would require a special case code path, which you have to maintain and test separately.

If your event bus is fully sequential, you might get away with sending an intermediate ‘respond to event X’ message. That way, the read model will definitely have processed event X and will be consistent. However, as we saw, there are a lot of drawbacks to using a fully sequential event bus.

Conclusion

Eventual Consistency is a difficult problem to deal with. Solving it usually requires thinking carefully about how your read models interact with one another and whether you need a read model to update in near real-time or if it’s fine for it to spend some time updating the underlying data store.

If parallel processing makes sense in your case, it’s probably a good idea for each read model to be fully independent by introducing a separate queue for each read model.

In my previous post I introduced the concept of Event Sourcing, which is a radically different way of storing application state: instead of storing the current state, we only store the events that lead up to that state. More than one person asked the following question in response: how do you query your application state? If you have a number of appointments in your system, how do you get a list of all the appointments on a given day?

I consciously skirted around that question in the previous post, because it was already long and loaded with information. In this post I will attempt to address that question properly.

Don’t query your events

One of the rules of Event Sourcing is: you don’t query your events. Why not?

First of all, they are hard, if not impossible, to query. If you use an RDBMS to store your events, you’re most likely serializing events into something like JSON, so unless your RBDMS supports a JSON datatype, it’s incredibly hard to get some data out of them inside of a query, let alone using it to build an index of some sort.

If you use a document database, like RavenDB or DynamoDB, to store your events, this becomes a lot easier, but it’s still not a good idea. Let’s say that you have an Appointment​Created event to indicate that an appointment was created, and an Appointment​Canceled event to indicate it was canceled. Yes, you can ask your database ‘give me the information from Appointment​Created events where the date is X unless an Appointment​Canceled event exists for that appointment,’ which is still somewhat manageable.

What if we add rescheduling into the equation? ‘Give me the information from Appointment​Created where the date is X, unless an Appointment​Canceled event exists for that appointment, or an Appointment​Rescheduled event exists where the date is no longer X.’ Wait, what about rescheduling to that date? ‘Oh, and include appointments where the latest Appointment​Rescheduled event has the date X.’ You see where I’m going with this, I hope.

Don’t query your state

You might be tempted to replay all the events in your system in memory and querying over those. This will be slow for large-ish systems and while it might work for very small systems, in theory, it’s still not a very good idea. It requires you to have knowledge of all the streams in your system. While most storage solutions would probably be able to answer this question, it’s usually a rather slow operation. A work-around to that is actually maintaining a list of streams, but this is error-prone (if done manually) and not very elegant.

Enter CQRS

CQRS stands for ‘Command-Query Responsibility Segregation’. It’s a pattern (not an infrastructure, as some people refer to it) coined by Greg Young. It requires you to have separate classes for writing data and reading data. This then allows you to have separate models for reading and writing. You could write to a set of properly normalized tables and read from a different set of tables that is entirely denormalized. You could write to a relational database and read from a NoSQL database or, in our case, write events and read from whatever tickles your fancy and suits the application.

It does this, in the words of Greg Young, by ‘creating two objects where there was previously only one.’

Let’s say we have a single repository for our (not Event Sourcing) application that deals with appointments.

interface IAppointmentRepository
{
    void CreateAppointment(
         int appointmentId, 
         DateTimeOffset time, 
         string title);
    void CancelAppointment(int appointmentId);
    Appointment GetAppointment(int appointmentId);
    IList<Appointment> AllAppointments();
}

We simply separate this out into two new repositories. One for writing:

interface IAppointmentWriteRepository
{
    void CreateAppointment(
         int appointmentId, 
         DateTimeOffset time, 
         string title);
    void CancelAppointment(int appointmentId);
}

And one for reading:

interface IAppointmentReadRepository
{
    Appointment GetAppointment(int appointmentId);
    IList<Appointment> AllAppointments();    
}

You should be able to see how this makes it very easy for the implementation of IAppointment​Write​Repository to use, for example, Event Sourcing, while the implementation of IAppointment​Read​Repository uses, for example, a NoSQL database like DynamoDB.

In this situation we talk about a ‘write model’ and a ‘read model’. The write model is the storage used by the implementation of IAppointment​Write​Repository. The ‘read model’ is then, logically, the storage used by the implementation of IAppointment​Read​Repository. Note that these models can be exactly the same. They don’t have to, however.

Command and Queries

The first two letters of CQRS stand for, respectively, Command and Query. Let’s look at what that means. Greg Young’s definition is as follows:

A command is any method that mutates state and a query is any method that returns a value.

This implies that a request to the data store is either a command or a query. Not a little bit of both. This means, by extension, that a command does not return a value. It also means that a query does not mutate state.

Why is this? One of the reasons behind CQRS is that it allows your read and write models to be asymmetric. Another very important reason is that it allows commands to become asynchronous. A command might be put onto a queue and executed at a later stage, which means you might have to wait for quite a long while before the command completes.

One of the most common patterns that would compromise this is using auto-generated identifiers. Something like this:

// returns ID of newly created user
int CreateUser(string userName, string password);

Usually you need the result of this method, for example to redirect the user to his own profile page after an account was created. You have to wait for the Create​User method to return before you can do anything. If creating the user is done asynchronously, this means the user is sitting there, waiting for something to happen.

In the CQRS pattern you would fix this by specifying the ID beforehand (using a GUID, for example). The Create​User method can then return as soon as the request has been validated and it has been put on a queue to be executed. You can then show a page to the user telling them ‘please hold on, we’re processing your request’. Meanwhile you’re polling your read model to see if the account has been created successfully.

Synchronization

If you’re writing to and reading from separate models, how do you go about keeping the data up to date? Let’s assume we’re implementing the IAppointment​Write​Repository from earlier. You could add some code to the end of the Create​Appointment method that calls into another method that updates the read model for you.

That method should really be in another class, to avoid violating the Single Responsibility Principle (which is really the ‘Single Reason for Change Principle’). Let’s call it a Normalizer.

void CreateAppointment(int appointmentId, 
                       DateTimeOffset time, 
                       string title)
{
    var appointment = new Appointment(appointmentId, time, title);
    this.eventStore.Store(appointment.NewEvents);
    this.normalizer.AddAppointment(appointmentId, time, title);
}

There are a number of problems here. First of all, this violates the Open/Closed Principe. When you add other read models, you might have to change this repository as well. Second of all, the class knows too much about how the Normalizer works; a change to its parameters means this class will also have to change. And what if the logic for creating an appointment changes and we also want to store, for example, which attendees have been invited to which appointments?

All in all, this doesn’t sound like a very scalable and maintainable solution. So what is? The answer is: events. We already have events that describe what has happened to an appointment, since they are the source of truth the system is built upon. Why not broadcast those events and have the read models response to those events?

void CreateAppointment(int appointmentId, 
                       DateTimeOffset time, 
                       string title)
{
    var appointment = new Appointment(appointmentId, time, title);
    this.eventStore.Store(appointment.NewEvents);
    this.eventBus.BroadcastEvents(appointment.NewEvents);
}

Obviously we need some infrastructure for this, so let’s create a (simplistic) example.

interface IEventBus
{
    void BroadcastEvents(IEnumerable<object> events);
}

interface IEventHandler<T>
{
    void Handle(T @event);
}

Then our ‘event bus’ enumerates over all the events and notifies all known handlers for that event type. The implementation would then look something like this:

class AppointmentNormalizer: 
    IEventHandler<AppointmentCreated>,
    IEventHandler<AppointmentCanceled>,
    IEventHandler<AppointmentRescheduled>
{
    void Handle(AppointmentCreated @event)
    {
        // add a new appointment to your read model
    }

    void Handle(AppointmentCanceled @event)
    {
        // remove an appointment from your read model
    }

    void Handle(AppointmentRescheduled @event)
    {
        // update an appointment's time
    }
}

This nicely solves most of the problems, except that each method in the implementation of IAppointment​Write​Repository has to store events and then broadcast them. If we combine the two steps into one method, that foregoes a lot of code repetition.

CQRS and Event Sourcing

CQRS on its own is a very powerful pattern, but using it in combination with Event Sourcing is especially potent. Let’s look at a scenario.

We’ve created our little calendar application using Event Sourcing and CQRS. We’ve created a read model that aggregates appointments by month and stores the results in a NoSQL database. Alas, there is a bug in the code that updates the read model, so it doesn’t handle rescheduling across months. What do we do? Well, we start by throwing away the entire NoSQL database. Then we replay all the events in the system to the (bugfixed) read model code, which will rebuild the database from scratch. Problem solved!

This could have probably been solved in a non-Event-Sourcing system by writing a custom piece of code that loops over all the appointments and adjusts the data in the NoSQL database. The beauty of CQRS with Event Sourcing is: you don’t have to write any custom code! This is part of the normal way the system works!

Caveats

There are a number of consequences, depending on your implementation, that you should be aware of when implementing CQRS.

Your storage space requirements will, relatively, explode. Most storage nowadays is pretty cheap, so this is usually not such a problem. Still, it’s worth being aware of it. The complexity of your codebase will also increase significantly.

If your ‘event bus’ is asynchronous, you will have to come to grips with the realities of Eventual Consistency. In a lot of applications this is not really a big issue, but of all the potential problems to be aware of, this is one of the few that might impact users of your application, so it’s the most important one to be aware of. And, unfortunately, also one of the most mind-boggling ones.

It is important to realize that, like with Event Sourcing, the decision to use CQRS with separate models (note the emphasis) should not be made on an application-wide level. You should make this decision for each part of your application separately. For instance, it might be valuable to use separate models for appointments, but not for users. If you use separate models anywhere in your application, it’s not a bad idea to use the ‘separate classes for reading and writing’ part of CQRS in your entire application. If you can, of course.

Further reading

There is a lot of stuff to be found around the web about CQRS and its many wonderful advantages and dangerous pitfalls. Here is some stuff I found useful.

• Jimmy Bogard: Busting some CQRS myths
• Martin Fowler: CQRS
• Udi Dahan: CQRS isn’t the answer - it’s just one of the questions
• Udi Dahan: Why you should be using CQRS almost everywhere...

I was introduced to Event Sourcing by the CQRS Journey series of articles from Microsoft’s Patterns and Practices team. It’s an excellent read about a pattern that offers great flexibility, at a cost. Event Sourcing was mentioned as more of a side note, but I was immediately drawn to its potential. As I’m currently involved in developing an application which uses CQRS and Event Sourcing, I’ll be journaling my understanding of it and its benefits and disadvantages.

What is Event Sourcing?

There are several excellent articles about this subject already, so I might be repeating some information. This is my understanding of the paradigm.

The ‘traditional’ way of persisting the state of your application is to store the state as it currently is. For example, your application might be a calendar, so you want to store appointments. It might look something like this:

This is a table that should look pretty familiar. It might be a table in a relational database, or a set of documents in a key-value store, or even a list of objects held in memory. The point is, it represents the state of system as it is now. Let me ask you a question: how did the system get to this state?

The audit log

To answer that question, you could create an audit log. In addition to creating or updating an appointment, you also store a record in the audit log, describing what happened (an ‘event’). For a single appointment, it might look something like this:

This reveals a whole lot of information that simply isn’t there if you only have the current state of the system. After it was created, the appointment was rescheduled once and its title was changed. The initial end time of 10:30 was there after the appointment was just created, but after it was rescheduled, that information disappeared. Poof.

This kind of information can be pretty useful for users of your application, because they can see what happened and who they should blame. Developers will profit the most from it; it’s so much easier to analyze a problem when you can see why something is in a particular state.

Single source of truth

An audit log beside your current state is useful, but there’s a problem: conflict. If you find yourself in a situation where your current state says A and your audit log says B, you now have two problems. You not only have to diagnose the problem at hand, but you also have to figure out where the discrepancy comes from.

The problem is the fact that you have two sources of ‘truth’ telling you what the state of the system should be. Your application only looks at the current state, so it sees no problem. You as a developer have two sources of truth. Because they conflict, neither can really be trusted.

Enter Event Sourcing

If we eliminate the audit log, we have one source of truth left, but we lose all of the detailed historical information that we value so much. What if we eliminate the current state?

That is the essence of Event Sourcing: instead of storing the current state of a system, you only store the events that lead up to that state. To get the current state, you can ‘replay’ those events in memory.

This current state is only an ‘ephemeral’ representation of events that have happened. It is not persisted. Well, there are snapshots—more on that later—but they are discardable. You can change what events do to the state of your application, but you cannot change the events. They have become the irrefutable truth.

Replaying?

To get the current state of your system, you will have to replay all the events. All the events? Well, not really all of them. Your current state is usually divided into several logical ‘objects’, which would traditionally be rows in a table. If you uniquely identify each object, you only need to replay the events for that object to get that object’s state.

Let’s look at a very simplistic example of the current state for our calendar.

class Appointment
{
    public Guid AppointmentId { get; }
    public DateTime StartTime { get; private set; }
    public DateTime EndTime { get; private set; }
    public string Title { get; private set; }
    public bool IsCanceled { get; private set; }
}

The events look like this.

class AppointmentCreated
{
    public Guid AppointmentId { get; }
    public DateTimeOffset StartTime { get; }
    public DateTimeOffset EndTime { get; }
    public string Title { get; }
}

class AppointmentRescheduled
{
    public DateTimeOffset StartTime { get; }
    public DateTimeOffset EndTime { get; }
}

class AppointmentRenamed
{
    public string Title { get; }
}

class AppointmentCanceled
{
}

Note that the Appointment​Canceled event does not have any properties. Just its type is a meaningful representation of what has happened.

What would replaying those events look like from the point of view of Appointment? Like this.

void ReplayEvent(AppointmentCreated @event)
{
    AppointmentId = @event.AppointmentId;
    StartTime = @event.StartTime;
    EndTime = @event.EndTime;
    Title = @event.Title;
}

void ReplayEvent(AppointmentRescheduled @event)
{
    StartTime = @event.StartTime;
    EndTime = @event.EndTime;
}

void ReplayEvent(AppointmentRenamed @event)
{
    Title = @event.Title;
}

void ReplayEvent(AppointmentCanceled @event)
{
    IsCanceled = true;
}

You start with a blank object. Then you simply call Replay​Event for each event that has happened, in the order they have happened. That’s all you need to recreate the current state.

Modifying state

Since state is represented by the events that have happened in the past, modifying that state is done by appending events. Making this the ‘current state’ object’s responsibility is a convenient way of keeping all the knowledge about the business logic and its rules in one place, which is a philosophy you also see in DDD.

The current state object (or entity object or domain object, however you want to call it) also becomes responsible for its own consistency; it is the only actor creating events about itself and it’s able to use events from the past to allow or disallow operations: your appointment has already been canceled; you cannot cancel it again.

This could look something like this.

class Appointment
{
    public Appointment(
        Guid id,
        DateTimeOffset startTime,
        DateTimeOffset endTime,
        string title)
    {
        if(endTime < startTime)
            throw new EndTimeBeforeStartTimeException();

        AppendEvent(
            new AppointmentCreated(id, startTime, endTime, title)
        );
    }

    public void Reschedule(
        DateTimeOffset startTime,
        DateTimeOffset endTime)
    {
        AppendEvent(
            new AppointmentRescheduled(startTime, endTime)
        );
    }

    public void Cancel()
    {
        if (IsCanceled)
            throw new AppointmentAlreadyCanceledException();

        AppendEvent(
            new AppointmentCanceled()
        );
    }
}

In this example, Append​Event is a method which will append an event to storage for the unique object that the Appointment refers to.

Validation of the parameters is done inside the methods. It makes a method ultimately responsible for the validity and consistency of the data. For brevity, Reschedule does not validate the parameters, but Cancel does check if the operation is allowed given the current state.

Currently, if you attempt to cancel an appointment that has already been canceled, an exception will be thrown. This doesn’t need to be the case. We could skip the check and just store the event anyway. It does not change the fact it has been canceled; we will still set the Is​Canceled property to true when it is replayed. The point here is: you do not always need to maintain and validate against state. Sometimes it is sufficient to ignore superfluous events.

Notice that all that the constructor and the Reschedule and Cancel methods are actually doing is storing an event. They are not directly modifying the state. Why? Well, we already have a method to modify the state based on an event that has happened: the Replay​Event method. So, in addition to storing an event, Append​Event will also immediately replay the event, so its state is up to date for any other operations you might want to perform and we don’t have to write the code to change state twice.

What does it bring to the table?

Let’s establish some ground rules about events.

• Events describe what has already happened, and because we cannot change the past, they are immutable. Events cannot be changed after they are persisted.
• Extending on the previous rule: they cannot be deleted.
• Each event contains all the information needed to represent the state change and to be able to replay it. They should not include computed values if they can be derived from the event, possibly when combined with earlier events.
• An event’s metadata contains its type and when it occurred, among others.
• Events can be uniquely identified by the identity of the object they apply to and an ordinal—also referred to as the revision or version number.
• You do not query over events. They are only used to replay into a representation of the state of the system at a given time.

For an event-sourced object, a request to change state should only result in one of three things:

• Storing an event;
• Throwing an exception (because a business rule would be violated);
• Doing nothing.

Other results, such as directly modifying the object’s state or making a database call, are strongly discouraged because they are side effects. Side effects are usually not represented in the events that are stored, which means you cannot replay them. They also make testing a lot harder.

Considering these rules, let’s look at some obvious, and not so obvious, benefits and disadvantages.

Benefits

Because you only have to read and append events, storage is very easy. The only things you need to be able to do are appending an event and retrieving the list of events for a unique object. You can store events pretty much anywhere: in a table, a key-value store, a directory of files, an e-mail, or almost anywhere you can imagine. Since events are immutable, they are very easy to cache, which enables you to get excellent performance.

You will not need explicit transactions any more, because you are only inserting data, not updating or deleting it; you also only need a single table, collection, set, or however your data store groups data. A transaction is like a lock, so not requiring transactions is kind of like lock-free code: there are no deadlocks and it’s much faster. This assumes that you design your events and group them into unique objects in such a way that you never need to manipulate more than one of them at a time—which is not that difficult to achieve.

You also get an audit log for free, which you can use when debugging to see exactly what has happened and why the system is in a particular state.

If you make sure the operations on your entity class are free from side effects, it basically lives in isolation and the class will have a very low coupling metric. You’ve isolated that part of the business logic from the rest of the system.

If you’ve achieved this, testing your logic can be expressed as ‘given these events have happened, when this operation is requested, then do these events get stored?’ Or, ‘then does this exception get thrown?’ Given, when, then. Does that sound familiar?

You don’t have to deal with the fact that relational databases are built on set algebra, while your objects are not. Event Sourcing sidesteps the object-relational impedance mismatch.

Your source of truth is a bunch of events in a database. After you store them, you can also broadcast them to the rest of the world via a messaging system. It’s very easy to let other applications know what’s happening in your system. You no longer need to query into another system’s data store and duplicate it to find out what has changed. All you need to store is a list of unique object identifiers and their latest known version number.

Disadvantages

Of course there are downsides to every approach, and Event Sourcing is no exception. The most important practical disadvantage, one which you will keep encountering when using Event Sourcing, is this: there is no easy way to see what the persisted state of your application is, and you cannot query over it. The only time when you can see what happens is at run-time. A solution, which is outside of the scope of this post, is using a separate model for reading that responds to the events that happen in your system. The CQRS pattern combines very well with Event Sourcing to accomplish this.

There are also disadvantages originating from the fact that Event Sourcing is an entirely different approach to what most people are used to. It will take some getting used to. It also takes a bit more boiler-plating than ‘regular’ current-state backing. You need to define each event that can take place in your system, so adding new features might be slower than you’re used to. To offset that, your data becomes tons more valuable.

Event Sourcing sounds like a very inefficient mechanism to store state. It is; it requires a lot more storage than current-state backing. It also requires more processing time, simply because you need to retrieve more data to get to the current state. Because you have practically limitless amounts of storage and processing power nowadays, it’s pretty easy to overcome any problems that arise from these ineffiencies. One of those is the use of snapshots.

Snapshots

You might think that when an event stream contains thousands or tens of thousands of events, your system must become slow, because for each operation it needs to replay all of those events. Well, not necessarily.

Remember, the only thing replaying events is doing is modifying some representation of state. It should be idempotent; whether you replay the events once or a hundred times, it should always have the exact same outcome.

After we’ve replayed, let’s say, ten thousand events, we can take a snapshot of the resulting state and store that, along with the identity of the last replayed event. Now, when we want the current state, we simply load the snapshot and replay any event that has happened since the snapshot, of which there should only be a few. If you’ve created your snapshot correctly, the outcome of loading a snapshot and replaying new events should be identical to replaying all of the events.

It doesn’t invalidate the events that happened. If, for some reason, the representation of state has changed, we can simply discard the snapshot and create a new one by replaying all the events at that point.

Testing

I mentioned that testing becomes a lot easier. Let’s make this a little more concrete and show how you test an event-sourced object.

private AppointmentCreated CreateAppointmentCreatedEvent()
{
    return new AppointmentCreated(
        appointmentId: Guid.NewGuid(),
        startTime: DateTimeOffset.Now.AddHours(3),
        endTime: DateTimeOffset.Now.AddHours(4),
        title: "Appointment"
    );
}

private AppointmentRenamed CreateAppointmentRenamedEvent()
{
    return new AppointmentRenamed(title: "Renamed appointment");
}

private Appointment CreateSut(IEnumerable<IEvent> events)
{
    var sut = new Appointment();
    
    foreach (var @event in events)
        sut.ReplayEvent(@event);
    
    return sut;
}

[Fact]
public void Reschedule_AppendsAppointmentRescheduled()
{
    // GIVEN these events have happened
    var events = new IEvent[]
                 {
                     CreateAppointmentCreatedEvent(),
                     CreateAppointmentRenamedEvent()
                 };
    var sut = CreateSut(events);
    
    // WHEN we ask to reschedule
    var newStartTime = DateTimeOffset.Now.AddHours(5);
    var newEndTime = DateTimeOffset.Now.AddHours(6);
    sut.Reschedule(newStartTime, newEndTime);
    
    // THEN does the AppointmentRescheduledEvent get published?
    sut.AppendedEvents.ShouldAllBeEquivalentTo(
        new[]
        {
            new AppointmentRescheduled(newStartTime, newEndTime)
        },
        config => config.RespectingRuntimeTypes()
    );
}

This exposes a few details I haven’t touched upon. They are mostly implementation details; they’re my way of solving some problems, you might choose a different way.

Remember AppendEvent?

Append​Event is a method which will append an event to storage for the unique object that the Appointment refers to.

That’s a little white lie. From the perspective of the Appointment, that is what happens, but it’s not really what happens.

For testability and decoupling, what really happens is that the event gets appended to a list of ‘unsaved’ events called Appended​Events. This enables us to very easily inspect what an object does and it enables the test you see above.

Note that the test can be divided into three logical sections: given some state, when an action is performed, then can we observe this behavior? A language which makes this much more formal is Cucumber (look into it, it’s awesome for behavior testing). Because this post is already getting long in the tooth, you’ll find an example of how to do this in this Gist.

Analogies

Event Sourcing is very powerful way of thinking, but it is not really new. There are event-sourcing systems most of us are familiar with and maybe don’t recognize them as such. Some examples:

• Your bank account is probably stored using Event Sourcing. The current account balance is simply a projection of all the deposits and withdrawals that you made.
• Version control systems. Each commit or change to a file is an event. If you replay all of the events, you get the current state of the source code.
• Most large RDBMSes use Event Sourcing under the hood. There are, simply put, only three events: insert, update and delete. The RDBMS stores the events in the transaction log and then applies them to the tables.

Conclusion

This is my understanding of Event Sourcing. I think it’s great, it’s powerful, but I might be biased because I’m already actively using it in an application under development. I hope you can see its potential. If so, please check out these excellent posts about the subject:

• Rinat Abdullin: Why Event Sourcing?
• Jonathan Oliver: Aggregate Roots and Shared Data
• Rinat Abdullin: Event Sourcing - Versioning
• Greg Young: CQRS and Event Sourcing
• Martin Fowler: Event Sourcing

In future posts I’ll be writing about my experience creating an application using Event Sourcing and CQRS. Have any other advantages or disadvantages? Let me know!

Previously in this series, I looked at ways we can instrument our code with two of the most basic metrics; how often does our code run and how long does it take to execute? After settling on interception, we took a small detour and looked at how hard it is to measure the time it takes to measure how long it takes to execute your code. This time, we’re going to combine all of them and throw another component in the mix: DI frameworks.

DI, abridged

Most of you probably already know what DI stands for and what dependency injection is. Just a small recap, which is also relevant for the article.

Dependency Injection is about decoupling a class from its dependencies. Instead of requiring a class to construct its own dependencies, it is passed them at construction time. This promotes readability, testability and, as a result of those, maintainability.

A consequence of using a DI framework is that it’s involved in the creation and destruction of most objects in your application. It needs to know, for each class it constructs, which dependencies it requires and how to construct those classes in turn. If you think about the amount of work potentially involved, you quickly come to a realization: it had better be fast.

Scope

A lot of DI frameworks are more flexible than most yoga instructors. Some, less so. As such, it’s impractical and unfair to compare all of the possibilities, and it would make the results very hard to interpret. So let’s define what we’ll be benchmarking.

• Resolving (or creating) an instance of Something, which implements ISomething, in such a way that calls to ISomething.Foo are timed, in the most common way for the framework in question;
• Invoking ISomething.Foo.

What we’ll be measuring is the cost of selectively applying a wrapper or interception and what that wrapper or interceptor adds in invocation time.

Frameworks

Since we can’t test all of the DI frameworks out there, I’ve made a—rather arbitrary—selection, based on, amongst others, what my colleagues and I use and want to use and frameworks I’m familiar with. These are the DI frameworks we’ll look at:

• Autofac
• Ninject
• SimpleInjector
• Unity

We’ll also (sometimes implicitly) look at a number of different interception frameworks:

• Castle DynamicProxy
• LinFu
• Unity Interception

The benchmark code is available on GitHub, so you can plug in your framework of choice and see how it does.

Let’s see the code

The code is pretty standard; a single method for each combination of frameworks we want to benchmark, decorated with a [Benchmark] attribute. Let’s see what it would normally look like.

[Benchmark]
void AutofacDecorator()
{
    var builder = new ContainerBuilder();

    builder.RegisterType<Something>()
           .Named<ISomething>("inner");
    builder.RegisterDecorator<ISomething>(
        (ctx, inner) => new TimingSomething(inner), 
        fromKey:"inner");

    using (var container = builder.Build())
    {
        var something = container.Resolve<ISomething>();

        something.Foo();
    }
}

This is the entire pipeline, from registering the class and its decorator, to building the container and actually resolving and invoking.

The problem is that registering dependencies can be a pretty expensive operation, but because it’s something you almost always only do once at the start of your application, it’s not very interesting to benchmark.

Amortizing initialization cost

That’s a header I didn’t think I’d ever write in my free time. One approach to separating these costs is to use BenchmarkDotNet’s [Setup] attribute, which exists specifically for this purpose.

We’ll do container construction and registrations there, which mimics real use—because you wouldn’t build up a new container for each request, right? (Right?!)

IContainer _container;

[Setup]
void Initialize()
{
    var builder = new ContainerBuilder();

    builder.RegisterType<Something>()
           .Named<ISomething>("inner");
    builder.RegisterDecorator<ISomething>(
        (c, inner) => new TimingSomething(inner), 
        fromKey:"inner");

    _container = builder.Build();
}

[Benchmark]
void AutofacDecorator()
{
    var something = container.Resolve<ISomething>();

    something.Foo();
}

The Initialize method gets called only once during test initialization, and the only thing that is actually benchmarked is actually resolving and invoking our Something.

Interception

Our interceptor looks pretty much the same for every framework. It has some optimizations, working under the assumption that we can, one-time, gather a list of all the methods that need to be intercepted. For the sake of simplicity, that initialization has been dumbed down to specifically getting the IFoo.Bar method.

internal class DynamicProxyInterceptor : IInterceptor
{
    private static readonly HashSet<MethodBase> _matchingMethods;

    static DynamicProxyInterceptor()
    {
        _matchingMethods = 
            new HashSet<MethodBase>(new[] {GetMatchingMethod()});
    }

    public void Intercept(IInvocation invocation)
    {
        if(!ShouldInterceptMethod(
            invocation.MethodInvocationTarget))
        {
            invocation.Proceed();
            return;
        }

        var stopwatch = Stopwatch.StartNew();

        invocation.Proceed();

        stopwatch.Stop();
    }

    private static bool ShouldInterceptMethod(MethodBase method)
    {
        return _matchingMethods.Contains(method);
    }

    private static MethodBase GetMatchingMethod()
    {
        var type = typeof(Something);
        var interfaceMap = type.GetInterfaceMap(typeof(ISomething));

        int index = Array.FindIndex(
            interfaceMap.InterfaceMethods, 
            method => method.Name == 
                      nameof(ISomething.Foo));

        return interfaceMap.TargetMethods[index];
    }
}

Benchmark results

I won’t go into the code in full detail, that’s for your own enjoyment. So without further ado, let’s see the results of our benchmark.

Some notes on these results:

• Methods starting with NoDI are not using any DI framework, as a baseline.
• If you want to see the specific versions of frameworks I’m using, see the project’s packages.config.
• Unity (natively) only supports its own interception framework, and that framework is—as far as I can tell—not usable outside of the context of Unity, so that’s why Unity is not combined with Castle DynamicProxy or LinFu and why none of the other frameworks are combined with Unity interception.
• LinFu is only supported (for automatic interception) by Ninject; that’s why it made no sense to include Autofac + LinFu or SimpleInjector + LinFu, for example.

Observations

Unity interception and Ninject are both so slow compared to the competition, they immediately seem uninteresting. Ninject without any interception is barely faster than Autofac with DynamicProxy. As for Unity interception, it takes around 75 times longer to intercept a method call using Unity interception than it does with LinFu.

SimpleInjector is on the other end of the spectrum; its scaled performance compared to the baseline (remember, that’s not using any DI) is still in the single digits, which is impressive!

Going back to LinFu, look at how fast it is! Kudos to its developers; LinFu stand-alone is even faster (admittedly, by a small margin) than Autofac without interception. It’s so fast that if you integrate it with a slow DI framework, it’s almost like getting interception for free.

Other observations:

• Autofac and Unity are ‘meh’ in terms of performance; they don’t excel but are not excessively slow, either.
• None of the DI frameworks add a lot of overhead of their own when using an interception framework.

Conclusion

First up the most obvious conclusion: YMMV. This is by no means a very realistic benchmark. You might encounter that certain frameworks perform a lot worse when they’re loaded down with thousands of registrations, or a lot better when you don’t use the most simple registration possible.

On the other hand, for most applications, time spent in resolving objects is not going to be the bottleneck, so don’t expect it to solve all of your problems. If you want to know if it will, use a profiler and measure first.

That being said, I still think it’s problematic if a DI framework, which is by definition involved in a large part of your application, is ‘slow’. For that reason, I wouldn’t use Unity interception. I would also steer away from Ninject in new projects, unless there is real good reason to use it. Other than that, everything else we tested flies below the radar.

Ease of use

I just want to touch on an aspect which is almost perpendicular to all we’ve been talking about, and that is ease of use. Some integrations between DI frameworks and interception frameworks are easy to use; they require little code and are easy to read and maintain. Others, not so much.

Let’s look at two examples that I find are at opposite ends of the spectrum. First up, Unity.

_container.AddNewExtension<Interception>();

_container.Configure<Interception>()
          .AddPolicy("Metering")
          .AddMatchingRule(new SingleMethodMatchingRule())
          .AddCallHandler(new UnityCallHandler());

This is the piece of code you configure once for your container, which says ‘I want to intercept some services with Unity​Call​Handler’.

_container.RegisterType<ISomething, Something>(
    new Interceptor<InterfaceInterceptor>(),
    new InterceptionBehavior<PolicyInjectionBehavior>()
);

This is a single registration. The Interceptor and Interception​Behavior will have to be repeated for each registration. That is a lot of code to simply say ‘I want to intercept calls to ISomething’.

Now let’s look at SimpleInjector.

_container.InterceptWith<DynamicProxyInterceptor>(
    type => type == typeof(Something)
);

Again, the one-time setup. Note how much simpler it is.

_container.Register<ISomething, Something>(Lifestyle.Transient);

This is the registration of a service. Where’s the part about intercepting? It’s not there, which means you can truly ‘plug in’ interception using the single call to Intercept​With. This might be worth a lot more than performance (although SimpleInjector has little to complain about).

Outcome

I started this series of posts with the premise that I was tasked with adding instrumentation to a project. Given that the project used Unity, my findings were another nail in Unity’s coffin. Long story short, we’re now looking at retro-fitting that project and are using SimpleInjector for future projects.

Oh, and if anybody is integrating LinFu with SimpleInjector, I’d love to know. Drop me a line!

Last time we evaluated a number of ways to add instrumentation to your code while keeping your code maintainable and legible. We landed on using interception and the cliffhanger question from last time was: what’s the overhead? Is it fast enough? In this post, we will start looking at ways to answer this question.

Benchmarking is hard

It’s not easy to measure how fast your code runs benchmark your code. There are misconceptions about how to measure how long something takes and what that means. You might think to capture the current date and time, run something in a loop, and again capture the current date and time. Take the difference, divide it by the number of iterations, and you have a relevant number. And you’d be wrong.

Don’t measure using DateTime

Naively measuring your code’s run-time duration looks like this:

DateTime startTime = DateTime.Now;
LongRunningOperation();
DateTime endTime = DateTime.Now;
TimeSpan elapsed = endTime - startTime;

This is wrong for a number of reasons.

• DateTime does not have enough accuracy to measure small increments;
• Assuming you’re doing this in production code, you might have gone into or out of daylight savings time, giving you huge or negative readings;
• Even if you’re using Date​Time.Utc​Now, the system time might have been adjusted—perhaps by an automatic time synchronization.

There are probably more of these falsehoods programmers believe about time that apply to measuring how long your code took to execute.

They can all be easily fixed by using something designed for the task: System.​Diagnostics.​Stopwatch. It is a highly accurate, monotonically increasing counter which does not suffer from changes to date or time at all. But even Stopwatch does not have an infinite amount of accuracy.

Don’t run your code just once

Let’s try to measure the run-time of a particularly small piece of code:

int x = 1;
var sw = Stopwatch.StartNew();
x++;
sw.Stop();

On my machine (YMMV), for about half the times, the elapsed time is zero, while the rest of the times, I get 300 nanoseconds. Neither of those sound plausible.

Let’s run it a whole lot of times and extrapolate a mean running time.

int x = 1;
var sw = Stopwatch.StartNew();

for (int i = 0; i < 100000; i++)
{
	x++;
}

sw.Stop();

Now I’m getting a total run-time of around 67 000 nanoseconds. But there seems to be a lot of jitter in the results. Here are the results of ten consecutive runs.

74 666,61 ns
51 357,98 ns
36 740,71 ns
79 802,41 ns
76 641,91 ns
41 086,39 ns
82 962,90 ns
117 728,30 ns
41 481,45 ns
40 296,26 ns

My number of iterations is probably still too small. I’d need to do some trial runs to figure out how many iterations I need to reduce the effects of jitter.

Unforeseen consequences

There are a lot of issues that crop up when you try to benchmark your code.

• The JIT compiler might add a delay to the first few calls to your code;
• The garbage collector might randomly pause your code to clean up memory;
• The benchmarking code itself also takes time to execute; you don’t want that included in the test results;
• You might be interested in more statistics than the mean execution time (think median, percentiles, standard deviation...);
• Compilers are pretty good at finding dead code, so you need to make sure your code is not being optimized away;
• Different runtimes might provide different results, so you probably want to see what running in x64 mode does, or when using (or not using) RyuJIT.

There are a thousand other problems you might encounter if you try to write your own benchmarking framework. Benefit from the blood, sweat and tears of other people and use an existing, battle-hardened one. My library of choice is BenchmarkDotNet.

BenchmarkDotNet

Let’s examine what a benchmark looks like when using BenchmarkDotNet. For the sake of example, we’ll be benchmarking string concatenation. I want to compare the speed of string.Concat, string.Format and using a String​Builder. Yes, I am aware that the comparison is totally unfair. Just play along.

class StringConcatenation
{
    [Benchmark]
    public string UsingStringConcat()
    {
        return string.Concat("Hello,", " ", "world!");
    }

    [Benchmark]
    public string UsingStringFormat()
    {
        return string.Format("{0}{1}{2}", "Hello", " ", "world!");
    }

    [Benchmark]
    public string UsingStringBuilder()
    {
        var builder = new StringBuilder();

        builder.Append("Hello,");
        builder.Append(" ");
        builder.Append("world!");

        return builder.ToString();
    }
}

Each method decorated with [Benchmark] is a candidate to be benchmarked, and when you start the process, like so:

BenchmarkRunner.Run<StringConcatenation>();

... BenchmarkDotNet will start running benchmarks. It will go through a number of steps to find the right iteration count, measure its own overhead and warm up the environment.

After a bit of thinking it will produce something like this:

// * Summary *

Host Process Environment Information:
BenchmarkDotNet=v0.9.8.0
OS=Microsoft Windows NT 6.2.9200.0
Processor=Intel(R) Core(TM) i7-6700HQ CPU 2.60GHz, ProcessorCount=8
Frequency=2531249 ticks, Resolution=395.0619 ns, Timer=TSC
CLR=MS.NET 4.0.30319.42000, Arch=32-bit RELEASE
GC=Concurrent Workstation
JitModules=clrjit-v4.6.1586.0

Type=StringConcatenation  Mode=Throughput  GarbageCollection=Concurrent Workstation

             Method |      Median |    StdDev |
------------------- |------------ |---------- |
  UsingStringConcat |  27.7441 ns | 0.5363 ns |
  UsingStringFormat | 125.0597 ns | 1.6980 ns |
 UsingStringBuilder |  53.8680 ns | 1.9694 ns |

Let’s look at the table at the bottom, which, incidentally, is valid GFM.

This is the final performance report. By default it lists the median duration and the standard deviation. We can see from the median duration that in this run, on my machine, using the regular x86 JIT, string.Concat is almost twice as fast as String​Builder, and from the standard deviation that the duration is also much more stable.

I’ve heard and read rumors that String​Builder will be faster than string.Concat if you concatenate more than 5 substrings. Let’s find out.

class StringConcatenation
{
    [Benchmark]
    public string UsingStringConcat()
    {
        return string.Concat("Hello,", " ", "world!", " Something ", "else ", "entirely");
    }

    [Benchmark]
    public string UsingStringFormat()
    {
        return string.Format("{0}{1}{2}{3}{4}{5}", "Hello", " ", "world!", " Something ", "else ", "entirely");
    }

    [Benchmark]
    public string UsingStringBuilder()
    {
        var builder = new StringBuilder();

        builder.Append("Hello,");
        builder.Append(" ");
        builder.Append("world!");
        builder.Append(" Something ");
        builder.Append("else ");
        builder.Append("entirely");

        return builder.ToString();
    }
}

string.Format is still trailing behind, but the gap between string.Concat and String​Builder has shrunk significantly. Let’s see what happens if we add more arguments.

Now things are getting interesting. The two methods are almost as fast, but it’s notable that string.Concat’s standard deviation is much higher than that of String​Builder.

The relative gap between the two now seems to be increasing, whereas the standard deviation is only increasing for string.Concat, and decreasing for String​Builder. Let’s go all out and double the number of arguments.

I ran this one a number of times to be sure, but it looks like String​Builder is finally faster than string.Concat! The conclusion? It takes around 28 substrings to be concatenated for String​Builder to be faster than string.Concat on my machine, in x86 mode, using .NET 4.6.2. I didn’t measure the exact tipping point, but I guesstimate it to be around 25 substrings. Interestingly enough, when using RyuJIT in x64 mode, string.Concat was still winning out.

What does this tell us?

Very little. At most, it tells us that there is little to no reason to use String​Builder over string.Concat for scenarios where you know at compile-time which strings you will be concatenating, and you’re not using loops.

It’s important to realize that benchmarks can give you an indication of performance, but they are no replacement for a run-time metric or a profiler when it comes to finding hotspots. Also, YMMV; when you benchmark a piece of code using a certain runtime on a certain architecture, your results are only valid on that specific environment. Nowhere else.

This, however, is what micro-benchmarks are about. They tell you if algorithm A is faster than algorithm B under specific circumstances, and by how much. When the difference is small, unless you have extremely stringent performance requirements, at most you might say: this won’t hurt us.

Now that we’ve looked at micro-benchmarking, next time we’ll look at actually putting some DI frameworks through their paces, combining it with interception, and trying to explain what we find.

I know I said the next post would be about benchmarking, and it was fully my intention, but then the real world happens. Recently, Microsoft announced the tentative feature-set of C# 7.0. Glancing over it, it looks like a similar improvement to what C# 6.0 brought us; it brought us 12 relatively minor improvements, while C# 7.0 clocks in at around 10 improvements. Let’s look at the ones that stand out.

Out variables

The gist of it is that instead of writing this:

string result;
if (dictionary.TryGetValue(key, out result))
    // it worked!

We can now write this:

if (dictionary.TryGetValue(key, out string result))
    // it worked!

It saves you an extra line of code and keeps you in the flow of typing, because you don’t have to jump back and add a variable. Time will tell if the time savings are worth the possible extra mental load.

Pattern matching

C# is adopting more and more functional paradigms, but that shouldn’t surprise you; it’s a shift that started with the introduction of LINQ. Pattern matching is one of those, and while it’s currently limited in scope, only supporting is and case, there are plans to expand this.

Both case and is pattern matching will increase the number of cases where you can join a variable’s declaration and initialization. For me, that’s less mental load, so it increases readability. It also makes writing code where you’d normally use an as clause a lot more enjoyable to write.

In a supreme case of tunnel vision, I spent an inordinate amount of time wondering if is also supports the when clause, you know, for ‘additional conditions’. I thought to myself: I must download the Visual Studio “15” Preview and find out. This is, of course, how you do ‘additional conditions’:

if ((shape is Rectangle s) && s.Width == s.Length)
   // it's a square!

Tuples

Alright, let’s move past the stupidity and onto the next productivity enhancer. I think tuples are a great solution for when you have to return a very small number of values from a private method, like a query that returns values from a number of related columns. When you construct a tuple with more than four values, I think it’s high time to introduce a DTO, because readability will most likely suffer otherwise.

private (string honorific, string first, string middle, string last, string postNominal) ParseName(string name)

My eyes hurt from reading this. Please, think of my eyes and introduce a DTO when you have too many values to return. What baffles me is that C# technically supports up to 8 (eight!) values in a single tuple. In my very humble opinion, if you use that many values, you need a stern talking to.

Deconstruction

Basically the consuming side’s counterpart of tuples, this feature’s main use case in my eyes is assigning values from a tuple to existing variables.

int x = 0, y = 0;
if (point != null)
    (x, y) = point;

Deconstructing tuples into new variables is only mildly useful, as you can just address the tuple’s members directly without having to create more variables (and possibly more mental load). Deconstructing arbitrary types is definitely only useful when using existing variables.

Local functions

A colleague of mine asked “Can’t you achieve the same thing with a lambda expression?” You cannot. Lambda expressions don’t have a name, so they can’t really self-reference. You can write a solution for that with delegates, but it’s pretty ugly:

void WriteFibonacciSequence()
{
    Func<int, int> fib = null;

    fib = i =>
    {
        if (i == 0)
            return 0;
        else if (i == 1)
            return 1;
        else
            return fib(i - 2) + fib(i - 1);
    };

    for (int i = 0; i < 15; i++)
    {
        Console.WriteLine(fib(i));
    }
}

This is pretty fragile; if someone modifies the value of fib (by accident, of course), your code is broken. Local functions don’t have this problem.

void WriteFibonacciSequence()
{
    int Fib(int i)
    {
        if (i == 0)
            return 0;
        else if (i == 1)
            return 1;
        else
            return Fib(i - 2) + Fib(i - 1);
    };

    for (int i = 0; i < 15; i++)
    {
        Console.WriteLine(Fib(i));
    }
}

Also, lambda expressions and anonymous delegates don’t support the yield return and yield break statements, while local functions do. Now you can finally stop writing separate private generator methods!

Literal improvements

Small stuff, but the addition of binary literals is nice and being able to insert _ as a separator at arbitrary positions might make some values more understandable.

Ref returns and locals

void Foo()
{
   int[] values = { 1, 2, 3, 4, 5, 6 };
   ref int bar = ref GetBar(values);
   bar = 42; // values is now { 1, 2, 3, 4, 42, 6 }
}

ref int GetBar(int[] values)
{
   return values[4];
}

This is the counterpart to ref method arguments, and it completes the full implementation of ‘references to value types’, and it’s probably most useful to high performance, memory critical applications.

Generalized async return types

This is another counterpart to a feature introduced in an earlier language version. In this case it’s async and await. In C# 6.0 you could already await things that are not Task; now you can also return things that are not Task from an async method.

This is a feature aimed at library developers, because it’s basically only plumbing to make sure you are able to return Foo from an async method, provided that you implement the async method builder and the awaitable.

More expression-bodied members

Expression-bodied members are wonderfully useful. The community rallied and implemented them in some of the places were Microsoft didn’t, which is awesome.

Throw expressions

Let’s look at one of the samples Microsoft provides:

public string Name { get; }
public Person(string name) => Name = name ?? throw new ArgumentNullException(nameof(name));

This is wonderfully specific and also doesn’t require any new methods or curly braces. I like it!

Conclusion

C# 7.0 doesn’t make as big a splash as C# 5.0 did, but it contains some very useful features that are bound to enhance your development productivity. I’m looking forward to using this in production code.

C# was already a mixed-paradigm language since the introduction of LINQ, and I’m happy to see Microsoft is continuing down that path. One of the features that didn’t make it which I would have loved to see is non-nullable reference types but that is understandably a much harder problem to solve.

Recently I was tasked with adding instrumentation to a project: measuring how often methods were being called and how long each execution took. I had a couple of options.

• Timing it inside of the methods;
• Use an adapter;
• Use an aspect.

The adapter solution and why not

Inserting timing code directly into the methods would violate separation of concerns, so it was never an option. The adapter solution consists of defining an adapter class which wraps calls with timing code. Let’s say we have a Foo class which implements IFoo:

interface IFoo
{
    void Bar();
}

class Foo: IFoo
{
    public void Bar()
    {
        // do important things here        
    }
}

Our adapter, Foo​Timing​Adapter, would then look something like this:

class FooTimingAdapter: IFoo
{
    private readonly IFoo _internalFoo;

    public FooTimingAdapter(IFoo internalFoo)
    {
        _internalFoo = internalFoo;
    }

    public void Bar()
    {
        var stopwatch = Stopwatch.StartNew();

        _internalFoo.Bar();

        Debug.WriteLine($"IFoo.Bar took {stopwatch.Elapsed}");
    }
}

This is nice; it separates the concern of instrumentation from the actual implementation of the class. The problem, of course, is that when you have a lot of classes to instrument, the collection of adapter classes becomes a burden to maintain.

When you change IFoo, you not only have to change Foo, but also Foo​Timing​Adapter. Doing TDD, as you should, this means you also have to write new, menial, tests for your adapter. This is starting to sound worse by the minute.

Aspects to the rescue

Instrumentation is a clear example of a cross-cutting concern: it is relevant to the program, but it is a different concern than, say, database access. It’s also generally highly reusable, which makes it a fine candidate for an aspect.

The difficulty with aspects is how to integrate them into your code; calling them directly kind of defeats the point. You want to use metadata like a marker interface or an attribute to indicate where an aspect should be invoked. .NET leaves you with a few options.

• Instructing an Intermediate Language (or IL) weaver like Fody where and how to insert calls to your aspect (which involves writing code that emits IL);
• Using an AOP framework like PostSharp, which uses IL-weaving under the hood but provides abstractions and interception points so you don’t have to write IL;
• Using a proxy generation framework like Castle Dynamic​Proxy to intercept calls.

Interception

Interception is all done at run-time. A proxy object is created which routes calls through an interceptor. Unlike the other methods, it’s only possible to intercept interfaces or class methods that are abstract or virtual. If you only want to intercept some calls, you’ll have to do filtering yourself, at run-time.

Alternatives to Interception

IL-emitting code quickly becomes complicated and very hard to understand and maintain, and your aspects’ code becomes invisible when debugging your code. Performance is top notch, however, because the code executed is identical to just inserting the code yourself. You can apply aspects to all of your code, including private and internal methods. Only applying your aspect to some of your code is done compile-time, so it’s cost-free at run-time.

Tools like PostSharp tend to have a large impact on your build process and they often require (expensive) licenses for each developer that compiles your project. In a team of developers, that’s probably everybody. Performance is excellent and it is fairly easy to reason about what your code is doing when debugging. You can easily target individual methods, classes or entire namespaces and assemblies if you want, and this is again done compile-time.

Let’s do interception

Performance-wise and feature-wise, this is easily the worst option. Because it’s highly legible (relatively, of course) and has virtually no impact on your build process—very important when you’re doing TDD—I still consider it to be the best option.

Let’s take a look at a very naive interceptor to time a single method, using Castle DynamicProxy.

class TimingInterceptor: IInterceptor
{
    public void Intercept(IInvocation invocation)
    {
        var methodToInstrument = 
            typeof(IFoo).GetMethod(nameof(IFoo.Bar));
        
        // is this the method we want to instrument?
        if (invocation.MethodInvocationTarget != methodToInstrument)
        {
            // no, so just call the original method
            invocation.Proceed();
            return;
        }

        // yes, time it!
        var stopwatch = Stopwatch.StartNew();

        // call the original method
        invocation.Proceed();

        Debug.WriteLine($"IFoo.Bar took {stopwatch.Elapsed}");
    }
}

This is naive, for a couple of reasons. First of all, it only supports instrumenting a single method. We probably want an attribute to indicate which methods to instrument. Second of all, it does not handle failure of the intercepted method. It might not be important for this purpose, but exception handling is something you have to explicitly do in an interceptor.

Last of all, it uses reflection each time a method is called. A cache would reduce the impact of that, but there is an unavoidable cost from the interception framework which has to create an IInvocation for each call, and it might do some reflection to do that each time a method is called.

Benchmarking time

What’s the overhead of all that reflection and whatever else the interception framework is doing? We could reason about it, but measuring is better. Benchmarks often yield surprising results (and it’s a lot of fun), so next time we’ll dive into benchmarking interception, and looking at how DI frameworks cope with this.