The thing with the change from waterfall to agile is that architects feel their role is being undercut, the team just goes fast and are only paying attention to the Product Owner. The standard answer they seem to get is: ‘then join the team’, but they feel reluctant to do so, and most of the times they can not fully commit (full time). So they pass, and they feel miserable about it, since now this Agile project is going to make mistakes, and can not learn from past experiences and their expertise.

The answer lies in the closer observation of the definition of Agile and Architecture.

Going agile is not the same as going fast, as the Agile Manifesto’s first principle says:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

So not just fast (early), but continuous too! And that’s the whole point of the position of architecture in Scrum/XP/Agile. In order to be able to continue to go fast, we have to make sure we mind our architecture, since architecture is:

The set of design decisions that minimize the cost of development, maintenance and use (my definition, close enough to Grady Booch’s)

So the whole point is that architects make sure that teams can go fast now and ever after, and that the cost of use and maintenance is proportional to the development investment. So architecture is not a stable deliverable or document, much better to speak about architecting; the process of keeping the architecture aligned with the product development and agile process.

Architecting is about identifying stakeholders that might be overlooked by the team and Product Owner, such as Support, Legal and 3rd party vendors. Architecting is about challenging the Product Owner and team to go slow, in order to be able to go fast by high quality standards, such as readability, 95%+ test coverage, early integration and performance testing, refactoring and the boy scout rule.

So architects, make sure you identify and represent stakeholders, and get their points on the Product Backlog (yes, it may contain non-functionals) by working closely with the Product Owner. You are a major stakeholder! Assist the team by making them go slow for a good reason, namely to serve previously overlooked stakeholders. There is a subtle balance here, if in doubt go fast with the Product Owner, and only address the issue when velocity drops because of technical debt.

The obvious implementation is to simply pass the UserDao directly to the report class^[1]:

[code lang=java]
package com.example.fizzbuzz.reports;

import com.example.fizzbuzz.dao.UserDao;
import com.example.fizzbuzz.domain.User;

public class FizzBuzzCsvReport extends CsvReport {

private final UserDao userDao;

public FizzBuzzCsvReport(UserDao userDao) {
this.userDao = userDao;
}

public String[] headers() {
return new String[] { "Fizzy", "Buzzy", "Email" };
}

public String[] row(FizzBuzz data) {
User user = userDao.findById(data.getUserId());
return new String[] {
data.getFizz(),
data.getBuzz(),
user.getEmail()
};
}
}
[/code]

The UserDao is injected into the constructor of the FizzBuzzCsvReport and used in the row method to translate the user id into an email address. Simple and probably what most Java code would look like. Unfortunately, this is not the right solution^[2]. Let’s write a unit test:

[code lang=java]
package com.example.fizzbuzz.reports;

import static org.junit.Assert.*;
import static org.mockito.Mockito.*;
import org.junit.Test;

import com.example.fizzbuzz.dao.UserDao;
import com.example.fizzbuzz.domain.Role;
import com.example.fizzbuzz.domain.User;

public class FizzBuzzCsvReportTest {
@Test
public void test_row_generation() {
UserDao userDao = mock(UserDao.class);
when(userDao.findById("userid")).thenReturn(
new User("userid", "user name", "user@example.com", new Role[] { Role.ADMINSTRATOR }));
FizzBuzzCsvReport subject = new FizzBuzzCsvReport(userDao);

String[] result = subject.row(new FizzBuzz("fizz", "buzz", "userid"));

assertArrayEquals(
new String[] { "fizz", "buzz", "user@example.com" },
result);
}
}
[/code]

That’s quite a bit of overhead just to perform a simple check! Why is it so hard to write the test? We’re even programming to an interface, not an implementation!

The main problem is not that UserDoa is not an abstraction (it is), but that it is the wrong abstraction for this usage. UserDao abstracts over how users are stored^[3], and by passing in the UserDao to the report we unnecessarily couple the report to the details of how users are represented and managed within the rest of our system. Note that using a dynamic language doesn’t really help either. The coupling would be less (no need to agree on the exact type) but the report would still require an object that responds to the findById message with an object that responds to the getEmail message.

So what would be the right abstraction? Let’s go back to the new requirement: “given the user id, add the user’s email address”. What’s the simplest abstraction that could work here? Let’s just use a function:

[code lang=java]
package com.example.fizzbuzz.reports;

import com.google.common.base.Function;

public class FizzBuzzCsvReport extends CsvReport {

private final Function userIdToEmail;

public FizzBuzzCsvReport(Function userIdToEmail) {
this.userIdToEmail = userIdToEmail;
}

public String[] headers() {
return new String[] { "Fizzy", "Buzzy", "Email" };
}

public String[] row(FizzBuzz data) {
return new String[] {
data.getFizz(),
data.getBuzz(),
userIdToEmail.apply(data.getUserId())
};
}
}
[/code]

Now the report is fully decoupled from our application’s user infrastructure and is much easier to test:

[code lang=java]
package com.example.fizzbuzz.reports;

import static org.junit.Assert.*;
import java.util.Collections;
import org.junit.Test;
import com.google.common.base.Functions;

public class FizzBuzzCsvReportTest {
@Test
public void test_row_generation() {
FizzBuzzCsvReport subject = new FizzBuzzCsvReport(Functions.forMap(
Collections.singletonMap("userid", "user@example.com")));

String[] result = subject.row(new FizzBuzz("fizz", "buzz", "userid"));

assertArrayEquals(
new String[] { "fizz", "buzz", "user@example.com" },
result);
}
}
[/code]

We use the handy Functions.forMap method to create a function from a map^[4] and use this in our test. Compared to the previous version, setup boilerplate has been reduced by 57.1%^[5].

Obviously, the actual production code (for example, the controller that let’s the user download the CSV) will still have to adapt the UserDao to the Function interface to make use of the FizzBuzzCsvReport. This is straightforward and only needs to be defined once. With Java 8′s upcoming lambda support this will be even easier. To summarize:

• Prefer simple, well-understood abstractions over home-grown variants (in this case a Function<String, String> versus a UserDao).
• By defining the FizzBuzzCsvReport in terms of a Function<String, String> userIdToEmail we make clear what the report needs and also limits what it can do (principle of least privilege). With the UserDao approach we wouldn’t know what exactly the report is using that DAO for, it could even be deleting users!
• Using an abstraction like Function gives you a huge library of pre-defined tools: adapting maps as functions, using memoization or caching, function composition, etc. Compare this to having to write your own caching adapter for a UserDao!
• Function is just the start. There are many others that are simple and widely applicable.

Footnotes:
[1] Code simplified for explanatory reasons.
[2] There are many other possible solutions for this example, but see footnote 1.
[3] Abstracting over how users are stored was very useful when we replaced LDAP with a web based user management system.
[4] In many languages collections are automatically functions. Scala’s Map and other collections already extend scala.Function1, so you can pass a Map whenever a function is expected. Ruby 1.9′s Array, Hash, Proc (Ruby’s function class), and String classes all respond to the [] method, etc.
[5] 98% of all statistics are made up.

In order to use this game in a presentation Bob, Daniel and I made a Javascript (standalone) version of it which uses variable iterations of shooting at the enemy’s ships. Board layouts are random and you get 40 shots in total to destroy the enemy’s fleet. After each iteration you get feedback about hits and misses. If you use iterations of 1, you are playing the regular battleship-game.

Each shot costs 10.000 and when you sink a ship you get the_ships_size * 50.000 (e.g. the submarine of size 3 will reward you with 150.000). If you keep track of the balance after each iteration, you could also try to get across the idea that stopping after a few iterations might give ‘good enough’ rewards.

It can be downloaded from our GitHub repository as a zip or you can take a look at our code. Just double click on the index.html (in the public folder) to start a game.

Now that Play 2.0 is getting closer to final release I took some time to dive in. Here are my first impressions, using the Scala APIs.

TL;DR

Pros:

• Automatic recompile on refresh gives you extremely quick feedback.
• High modularity makes Play easy to learn, use, and extend.
• Very easy deployment to production environments.
• Type safe templates and routes make it easier to keep your application consistent.
• High performance with support for both synchronous and asynchronous HTTP handling.
• Java and Scala APIs are quite similar, so should allow easy transfer of skills.
• Support for pre-compiled assets including CoffeeScript, LESS CSS, and the Closure JavaScript compiler.
• Sessions are simple cookies, making it easy to scale or perform zero-downtime upgrades.

Cons:

• Support for WAR packaging is not planned until 2.1. This is needed if you need to deploy in a Servlet Container.
• Not backwards compatible with the older Play 1.x framework.
• Custom conf/routes format instead of simple DSL like Scalatra or spray.cc.
• Many APIs rely on singletons (Scala) or static methods (Java), which can make testing challenging. However, fake implementations of the main abstractions are provided (Application, Request, etc).

Out of the box experience

Getting started with Play is straigthforward. Just download the distribution, unpack it, and add it to your $PATH. After that type play new my-app, select your application’s name (defaults to my-app in this example), and type (Scala, Java, or empty). After choosing the Scala application type I ended up with a new directory containing just a few directories and files (with just 19 lines of Scala code and 17 lines of HTML templates).

Starting the application is as easy as typing play run and pointing your browser to http://localhost:9000/. The initially load takes a while as it needs to compile templates and Scala sources. After the page loads you get a nice explanation what to do next and plenty of documentation links:

Controllers and templates

The first thing you encounter in any web framework is to mapping from HTTP requests to your code. In Play this is extremely straightforward. The request method and path are located in the conf/routes file and the indicated method is invoked in your controller. An example route entry is:

GET     /            controllers.Application.index

So whenever the home page is requested the controllers.Application.index method is invoked. Notice that you’ll get a compile time error if this method does not exist or requires parameters. So if you add the following to the conf/routes file:

GET     /:page       controllers.Application.page(page, format ?= "html")

and refresh your browser you’ll quickly see an error message:

Fixing this is easy, just add the missing method to the controller.Application object:

def page(name: String, format: String) = Action {
  Ok(<p>You requested page {name} in format {format}</p>.toString).as(HTML)
}

As you can see a controller action is just a method that returns an Action. The Action in turn returns an Ok response, which translates into a HTTP 200 OK response. What I like here is that the controller method is directly linked from the conf/routes file without me having to remember any mapping conventions. There is also no complicated rendering pipelines that are typically associated with component based web frameworks like JSF or Wicket.

In this action we directly returned a piece of HTML. It is also possible to use templates. Let’s change the index page to link to our new controller action. By changing the index.scala.html template to:

@(message: String)

@main("Welcome to Play 2.0") {
  <a href="@routes.Application.page("1984", "PDF")">page example</a>
}

a new link is rendered that maps to the new route. After refreshing the browser you can now click the link to navigate to your new page action. Extremely straightforward and especially nice is the use of type safe URLs and the light-weight template syntax (just using the @ sign to add some Scala code in your HTML template).

Also nice is that templates are turned into ordinary methods (with the first line of the template becoming the parameter list) that can be called from controllers, other templates, or tests. In the template above the main template is invoked to render layout around the provided link.

Forms

Play also has support for forms, validation, and binding form data to domain objects. Unlike most other web frameworks it is not necessary to alter your domain objects to use them with forms. Specifically, there is no need to add a default constructor or any setters. Here is an example domain object User with a form definition:

import play.api.data._, Forms._, validation.Constraints._

// Domain object
case class User(name: String, age: Int)

// Form definition
val userForm = Form(
  mapping(
    "name" -> nonEmptyText,
    "age" -> number(min = 0, max = 150)
  )(User.apply)(User.unapply))
  
// Form submit action
def register = Action { implicit request =>
  userForm.bindFromRequest.fold(
    errors => BadRequest(views.html.index(errors)),
    user => Redirect(routes.Application.index))
}

The form defines two fields with validation and the mapping from the form fields to the domain object (in this case by using the Scala generated User.apply and User.unapply methods).

The register action uses the form definition to perform validation and bind data from the request to the domain object. In case of errors, the page is rerendered with the validation errors. Otherwise a new user instance is available (and in this case a simple redirect is performed, a real application would probably save the user somewhere). Notice that it is not possible to get a (partially) invalid user object, unlike most other form validation frameworks (like Ruby on Rails, Spring MVC, and Wicket).

To render the form there are some predefined form helpers (also available in Twitter Bootstrap variant). Unfortunately the field names are not statically checked against the form definition:

@(registrationForm: Form[User])

@helper.form(action = routes.Application.register) {
  @helper.inputText(registrationForm("name"))
  @helper.inputText(registrationForm("age"))
  <input type="submit" value="Register" />
}

Databases

Play comes with built-in support for SQL/JDBC databases. The simplest way to use this to first enable the datasource in conf/application.conf:

db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:mem:play"

Then add a schema evolution file conf/evolutions/default/1.sql:

# --- !Ups

create table users (
  name text primary key, 
  age int not null);

# --- !Downs
 
drop table users;

When you refresh the browser Play will notify you that you need to apply the schema migration:

Pressing the ‘Apply this script now!’ button will immediately perform the database migration.

Now that we have a running database it is easy to query using the JDBC API and the Anorm wrapper library:

import play.api.db._, anorm._, play.api.Play.current

def registeredUsers = DB.withConnection { implicit connection =>
  SQL("select name, age from users")().map(row => User(row[String]("name"), row[Int]("age"))).toList
}

Since DB.withConnection/DB.withTransaction gives you a plain JDBC connection it is also easy to use any other Java or Scala tool to handle database persistence.

Deployment

Deploying a Play application is easy. My preference would be to use the play dist command that builds a ZIP archive. Then you only need a Java runtime and database installation on your production server to run your application (everything else is included in the ZIP, including the Scala libraries and a start script).

Another option is to use the play stage command to run the application in-place. Combine this with a git push to Heroku or your own production server and you’re ready to go.

Performance

Play is build on Netty, which provides very high HTTP server performance. Play supports both synchronous and asynchronous IO, so handling many long-lived connections should be possible (and Play provides excellent support for streaming using its Iteratees based data stream support). Integration with Akka 2.0 should give you all the tools you need to build high-performance and highly-scalable applications.

To give you an idea, I had no trouble getting more than 10.000 HTTP requests per second (tested using jmeter) on my mid-2010 MacBook Pro (2.66 GHz Intel Core i7). The rendered HTML page was fairly trivial (just the user registration form as shown above), but it is good to know basic performance is very good.

Conclusion

This was just a first quick look at Play 2.0, but I’m quite impressed with the ease of learning, development, and deployment. Play 2.0 is a worthy addition to the Typesafe Stack and should give Java and Scala developers a very good platform to build web applications on.

It’s all about a shared goal
First I’d like to emphasize the essence of Agile and Scrum. It is all about delivering maximum value to your customer and/or end users. In order to achieve this (rather complex) goal, instead of working in a traditional hierarchical organizational setting in which management dictates and workers act, you work together as a group in order to achieve the best possible results. In fact this means not only the development team should work together as one, but the business and management in the organization should be aligned as well. All involved should strive for this same goal and realize they’re in it together.

Nice, now back to the standup meeting
To determine and synchronize the current status in a sprint Scrum describes to ask each development team member (as taken from the Scrum Guide):

• What has been accomplished since the last meeting?
• What will be done before the next meeting?
• What obstacles are in the way?

At the end of this meeting every team member should be up to date with the team’s current status and should have clearly determined what to do for the rest of the day.

Anti-pattern: the team actually is a group of individuals
In the real world however I see some different behavior and scenario’s. The most fundamental issue is that the team not having a shared goal. This is mostly reflected by developers individually working on backlog items or tasks. There is little or no cooperation going on and no real concern to finish a top class product as a group. The effect this has on a team is that the Daily Scrum is not running smoothly, or even considered a nuisance. The individuality symptom is often reflected through the three default questions asked using a slight twist (however, with a dramatic effect):

• What have you done since the last meeting?
• What will you do before the next meeting?
• What obstacles are in your way?

The effect: the team accepts and embraces the fact of being individuals. Working as individual subgroups is a step forward, but a small one. There are a couple of risks involved when in this situation:

• The team and organization are dependent on individual team members. If such a team member is not in, the team struggles to do the work normally executed by that person. It limits the notion of collective code ownership, given by XP
• Team members are focused on their own tasks with their own concerns and have little focus on the customer / end users
• How can you make sure you will reach your commonly shared goal?

A typical sprint backlog when in this situation will look like the one display below.

A troubled sprint backlog (image created by Henrik Kniberg)

A few tips how to overcome this and work towards a shared goal and shared team responsibility

• First, be sure to have one (!). Without having a goal, everything is acceptable. When having one, make sure this goal is spread and known by everyone involved. Every individual needs to be able to breathe the goal
• It is allowed to have more than one Daily Scrum a day.
  It often comes as a surprise that Scrum allows you to synchronize more than once a day. I especially advice teams that are in a stressful situation – say prepping the Sprint Review meeting – to have frequent short meetings to make sure to deliver a great product Increment. It may be the term Daily Scrum is actually a bit unlucky
• Focus on what you can do together as a team

Try to ask questions like:

• When can we have this item finished?
• What do we need to do to wrap up this item?
• What can we do before the end of the day?
• What do we need to do for the Sprint Review?

I’ve seen asking these questions help teams to achieve shared responsibility towards reaching the end goal.

Remember: no single individual is responsible for reaching the goal.

A few weeks after it closed I started noticing that whenever I was outside grabbing some fresh air (or smoking if you wish ) people still came in, walked the up the stairs to find out that the postal office was closed. This happened every single day for a couple of weeks at least 3 times per cigarette. And these people were not only tourists but also knowledgeable Amsterdammers.

In the first couple of weeks the only thing that changed was that the original postal office sign itself was removed. People still walked the stairs only to find out the postal office was moved. After the first couple of weeks the new owner probably got fed up with people coming up to the new store and asking directions to the postal office:
A new big sign was placed in front of the door that indicated the postal office was moved. At the same time the new postal office next door placed huge banners with their logo.

New big sign.

Postal office next door.

So clearly these Big Visible Charts should be noticed by everyone you think right? Well wrong. While enjoying fresh air I observed some people looked at the sign but ignored it, and some people didn’t even look at the sign and walked straight by it.
When observing this I realized I experienced the same things in projects. Even though the information was there, it was visible and big, it was still ignored. Why did this happen? Why do people ignore this information? Here is my short analysis:

• The sign was not clear. Looking closer at the sign you see a lot of information. It is so much people probably don’t know where to start reading. If this happens people will discard the complete sign.
  
  Sign with information overload

• Habit. Habits are difficult to kill. This probably applies to most Amsterdammers who came into the building expecting the postal office. They knew from their year long experience the postal office is there. They probably discarded the sign as advertisement or something else and just went straight for their goal.
• Pointed to the wrong direction. I noticed several tourist were brought to the location by taxi drivers. The taxi drivers didn’t know it was moved. Without questioning the tourists followed directions and just walked in the building to discover the postal office was gone.

Just when I wanted to blog about these findings something interesting happened. This week they replaced the sign with a newer version. To me this one is much clearer. I posted in front of the door for 4 times already and no one came up to me to ask directions for the new postal office. People stopped to read the sign and actually followed the directions.

Improved sign

How does this relate to my work?
From time to time I experience what can happen if radiated information is not understood, not seen or misinterpreted. This goes from completely ignoring the information, to killing a project after the first sprint because the “radiated” velocity was not enough.
When using Big Visible Charts we should take care on what we want to achieve by radiating this information, otherwise we might not reach the goal we want to achieve. When I started to think about this more consciously I discovered that I actually apply the following guidelines when creating these charts:

• Educate! Big Visible Charts are not enough. People must learn how to read these charts and act upon it. This especially goes for doing Agile projects in traditional environments. People in these settings are typically not used to get that much information in such short time.
• If it is outward-facing information (e.g. for stakeholders) then the readers need to be able to understand it in a couple of seconds. Don’t use a lot of words, but use pictures or simple charts instead.
• If it is inward-facing information (e.g. for team members, product owner) then it can be fine-grained with details.
• You could try to physically separate the outward-facing information charts from the inward-facing information charts.

What guidelines do you apply while creating Big Visible Charts?

PS: After googling I also found some studies about successful (commercial) signs, for instance: http://web1.msue.msu.edu/imp/modtd/33719792.html. This also contains stuff like color schemes and such. Do you know more?

[scala]
// try1.scala
def serialize(string: String) = string.getBytes;
def deserialize(bytes: Array[Byte]) = new String(bytes);
def test(string: String) {
val actual = deserialize(serialize(string))
println(if (string == actual) “OK “.format(string)
else “FAIL expected got “.format(string, actual))
}

test(“”)
test(“foo”)
test(“the quick brown fox jumps over the lazy dog”)
test(“\u2192″)
[/scala]

Save this to a file (try1.scala) or get it from github at erikrozendaal/scalacheck-blog and run it with the Scala “interpreter”:

[bash]
$ scala try1.scala
OK
OK
OK FAIL expected got
[/bash]

Oops, the last test fails on my laptop: FAIL expected <?> got <?>. The error message is not very helpful, but it turns out the default encoding used by Java on Mac OS X is MacRoman, and String.getBytes silently changes any unknown character to a question mark. Now silent corruption may make things easier, but is normally unwanted. Especially when building an event store!

Let’s try again. This time we explicitly use the UTF-8 encoding:

[scala]
// try2.scala
def serialize(string: String) = string.getBytes(“UTF-8″)
def deserialize(bytes: Array[Byte]) = new String(bytes, “UTF-8″)

// [... test code unchanged ...]
[/scala]

Running this example succeeds, even with the default encoding set to MacRoman. But these little interactions showed us that serializing a string to an array of bytes may not be as easy as originally expected. Are there any other strings that fail to serialize and deserialize? And how can we find these?

Fortunately, there is a powerful tool available for this: ScalaCheck. ScalaCheck generates random test examples and uses these to test programmer specified propositions. This sounds more complicated than it is in practice. So let’s try an example:

[scala]
// try3.scala
import org.scalacheck._

def serialize(string: String) = string.getBytes(“UTF-8″)
def deserialize(bytes: Array[Byte]) = new String(bytes, “UTF-8″)

val prop_serializes_all_strings = Prop.forAll { s: String =>
s == deserialize(serialize(s))
}

prop_serializes_all_strings.check
[/scala]

Here we reuse the same string serialization code from the previous example, but instead of manually thinking of examples to test we use ScalaCheck’s Prop.forAll method to create a proposition. You can basically read lines 7 to 9 as: “For all strings s, s must be equal to the result of serializing and deserializing s“. On the last line we tell ScalaCheck to check this proposition. ScalaCheck does so by generating random strings and passing these to our proposition. Download the ScalaCheck jar and try it:

[bash]
$ scala -cp scalacheck_2.8.1-1.8.jar try3.scala
! Falsified after 15 passed tests.
> ARG_0: ?
[/bash]

Not good. It only took 15 tries for ScalaCheck to find a counter example, which ScalaCheck prints on line 3 above. Unfortunately, due to the limits of current font technology unknown characters are not clearly printed. So we know there is at least one counter example, but we do not know yet what it is.

To help us with this, ScalaCheck allows us to label a test case. Labeling our proposition with the integral values of the string contents should provide us with more information:

[scala]
// try4.scala
import org.scalacheck._, Prop._

def serialize(string: String) = string.getBytes(“UTF-8″)
def deserialize(bytes: Array[Byte]) = new String(bytes, “UTF-8″)

def decode(string: String) = “characters = ” + string.map(_.toInt).mkString(“,”)

val prop_serializes_all_strings = Prop.forAll { s: String =>
decode(s) |: s == deserialize(serialize(s))
}

prop_serializes_all_strings.check
[/scala]

Here an invocation to the new function decode (line 7) is added as a label to the prop_serializes_all_strings proposition using the ScalaCheck provided |: operator (line 10).

Running this version a couple of times gives us some ideas about what is going wrong:

[bash]
$ scala -cp scalacheck_2.8.1-1.8.jar deser4.scala
! Falsified after 11 passed tests.
> Labels of failing property:
characters = 56465
> ARG_0: ?
[/bash]

It turns out our proposition fails for strings containing characters in the range \uD800 \uDFFF. Looking at the Unicode reference it turns out these are the leading- and trailing-surrogate ranges defined for UTF-16 encoding. And Java Strings use UTF-16 encoding internally. So it looks like ScalaCheck generates Strings that are not valid UTF-16 Unicode strings, and java.lang.String happily accepts them. [NOTE: ScalaCheck 1.9 no longer generates strings containing leading- and trailing-surrogates. So Strings will never contain characters from the Supplementary Planes, but at least all generated strings are valid UTF-16.]

In our case, we’re only interested in serializing valid Unicode strings. But since we should never silently corrupt data, it would be better to fail with an exception when an illegal string is passed to our serialize implementation. The Java String class does not provide a method for this, so we have to use a CharsetEncoder instead.

Unfortunately, this requires digging into some of the lower-level APIs provided by Java NIO. Here’s the end result:

[scala]
import java.nio.{ByteBuffer, CharBuffer}
import java.nio.charset.Charset
import org.scalacheck._, Prop._

def encoder = Charset.forName(“UTF-8″).newEncoder
def decoder = Charset.forName(“UTF-8″).newDecoder

def serialize(string: String) = {
val bytes = encoder.encode(CharBuffer.wrap(string))
bytes.array.slice(bytes.position, bytes.limit)
}
def deserialize(bytes: Array[Byte]) = decoder.decode(ByteBuffer.wrap(bytes)).toString

def decode(string: String) = “characters = ” + string.map(_.toInt).mkString(“,”)

val prop_serializes_all_strings = Prop.forAll { s: String =>
encoder.canEncode(s) ==> (decode(s) |: s == deserialize(serialize(s)))
}

prop_serializes_all_strings.check
[/scala]

The serialization code has been changed to directly use the Charset encoder and decoder and the proposition is changed to include a guard condition using the ScalaCheck ==> operator: the proposition only holds for strings that can be encoded. Let’s run this:

[bash]
$ scala -cp scalacheck_2.8.1-1.8.jar try5.scala
+ OK, passed 100 tests.
[/bash]

Finally, success! Well, mostly. Occassionally ScalaCheck will fail:

[bash]
$ scala -Dfile.encoding=UTF-8 -cp scalacheck_2.8.1-1.8.jar try5.scala
! Gave up after only 98 passed tests. 500 tests were discarded.
[/bash]

This happens because our guard condition discards strings which cannot be encoded. By default, ScalaCheck will try to generated 100 passing examples, but will not discard more than 500 examples when trying to achieve this. One way to fix this is to add some parameters to the check invocation:

[scala]
prop_serializes_all_strings.check(Test.Params(minSuccessfulTests = 50, maxDiscardedTests = 1000))
[/scala]

Now it becomes very unlikely that ScalaCheck fails to find enough successful tests.

Another approach is to define a custom generator that only generates valid Unicode strings. Generating valid UTF-16 Unicode strings is a little tricky, so the code is a bit long:

[scala]
// [... imports same as before ...]

val UnicodeLeadingSurrogate = ‘\uD800′ to ‘\uDBFF’
val UnicodeTrailingSurrogate = ‘\uDC00′ to ‘\uDFFF’
val UnicodeBasicMultilingualPlane = (‘\u0000′ to ‘\uFFFF’).diff(UnicodeLeadingSurrogate).diff(UnicodeTrailingSurrogate)

val unicodeCharacterBasicMultilingualPlane: Gen[String] = Gen.oneOf(UnicodeBasicMultilingualPlane).map(_.toString)
val unicodeCharacterSupplementaryPlane: Gen[String] = for {
c1 <- Gen.oneOf(UnicodeLeadingSurrogate)
c2 <- Gen.oneOf(UnicodeTrailingSurrogate)
} yield {
c1.toString + c2.toString
}

val unicodeCharacter = Gen.frequency(
9 -> unicodeCharacterBasicMultilingualPlane,
1 -> unicodeCharacterSupplementaryPlane)

val unicodeString = Gen.listOf(unicodeCharacter).map(_.mkString)

// [... serialization code same as before ...]

val prop_serializes_all_strings = Prop.forAll(unicodeString) { s: String =>
decode(s) |: s == deserialize(serialize(s))
}

prop_serializes_all_strings.check
[/scala]

On lines 3-5 we define three character collections that will be used to construct correct Unicode data.

On line 7 we define a generator for generating a single Unicode character from the Basic Multilingual Plane (BMP). It basically picks a random element from the UnicodeBasicMultilingualPlane collection and converts it into a (single character) String.

Lines 8-12 define a generator for Unicode characters from the supplementary planes. It generates a two character string with the first character taken from the leading surrogate range and the second character taken from the trailing surrogate range. The code uses a for-comprehension since generators are allowed to fail (but in our case will never do so).

Lines 14-16 combines these two generators to produce a new generator that generates a BMP character 9 times out of 10, and a supplementary plane character otherwise.

Finally, line 18 uses the previous generator to create a list of random but valid Unicode characters and concatenates this list into a single string. This is the generator we want to use in our proposition. This is shown on lines 22 to 24. Instead of guarding our proposition we now explicitly pass in the generator to use in our call to Prop.forAll. Our proposition now passes without needing to discard any examples, so we can remove the check parameters.

Conclusion

With traditional unit tests it is hard to come up with good data that actually exposes bugs, especially the unexpected kind of bugs that only seem to creep up in production. Using ScalaCheck we can let the computer come up with random data to test our code. ScalaCheck includes standard generators for the usual suspects such as Int, String, etc. It’s also easy to add and use custom generators for existing or new data types.

ScalaCheck is not a replacement for traditional tests. Explicit examples are still useful as documentation and we should also add a test to the above code to ensure an exception is thrown when we try to serialize a String that is not valid Unicode, as our proposition only tests the behavior for valid Strings. But ScalaCheck based testing can help us find those unexpected bugs (really, the most common kind) before they get into production code. Trying to find the silent corruption bug of the first few serialization attempts would be a lot harder in a production environment.

In this first post I’ll introduce you how to setup your machine to be able to rock some Rails code.

1. Rubies please

Instructions how to get Ruby for your environment you can find here. On a Mac it is installed by default. You can easily find it out by executing the ‘ruby’ command in your shell and see what happens.

No worries if this isn’t the latest version available. We will only use it to get you bootstrapped.

2. RVM

RVM (Ruby Version Manager) is a tool to help you to be able to use multiple Ruby environments on one machine. If you have multiple Ruby projects on your machine it is quite handy to be able to let them run in their own (sort of) sandbox. Simply put: if you don’t use RVM all dependencies (gems) will be installed system-wide. You just don’t want that as things will get messy. Given you have successfully installed Ruby, you will now install RVM. This short installation step summary requires you having git installed. If not you can view the full installation guide here.

$ bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

Edit your ~/.bash_profile to contain the following:

[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"

Now start a new shell and you should be able to perform something like

$ type rvm | head -1

This should give you

rvm is a function

Next, and last step, execute

rvm notes

This gives you an overview of what to do next for your environment. On a Mac, be sure to have the right XCode version installed as said in the output.

Now what did this do?
You got the version manager tool installed, but what about the bash_profile stuff? What this small line does is making the rvm command available on your PATH. Handy, you can now run it anywhere you go. Also, it helps you to switch environments easily when navigating through different Ruby projects. We’ll get into that later. Moving along to the next step…

3. Install Ruby within RVM – creating your first RVM environment

Let’s install the latest Ruby version within RVM:

$ rvm install 1.9.2-p180

This is the latest version at the time of writing. You can go here to verify if there exists a newer version.
If you run

$ rvm use 1.9.2

$ which ruby

you should get something similar to the following output:

So…

From now on, all the dependencies or Ruby versions you will install will be placed in the RVM folder that’s located in your home folder. If you install stand alone gems (equivalent of JARs) or upgrade dependencies this will only affect the currently used Ruby installation stored in your home folder. If you mess up, there’s nothing to worry about .

Almost there. Going well so far. You’ve got the latest and greatest Ruby version within RVM. You’re almost ready to rock…

4. Install Rails and create your first application

Rails is a web framework for Ruby. Install it by running

$ gem install rails

Create a new application

$ rails new example

This will create a new folder containing your first Rails application. Plus it has created a lot of stuff for you. It’s out of scope of this post to cover fully what happened. Just remember this generated a skeleton and a kick start for your first application.

5. Bundler (Maven for Ruby)

Bundler helps you to maintain dependencies similar as what Maven does for Java. You cannot start the example application without retrieving the required dependencies. Retrieve the dependencies by:

  $ cd example
  $ bundle install

6. Rock and roll

Your first application is there! It might have been a boring road with nothing much to see. But… you’ve got an application that is ready to start. Give it a kick by

$ rails server

Now open a browser pointing to http://localhost:3000. You should see this screen:

Wrapup

Setting up development environments isn’t the most fun you can have. But doing the above gives you enormous flexibility. By using RVM instead of installing everything system wide makes it easy to switch Ruby versions, be flexible with the dependencies – and not be surprised by versioning issues over different projects. There’s more to RVM than described above, but this is enough to give you a heads up.

If you like this to be the first of a series, please let me know!

I think a code review report should be concise, clear and with concrete actions on how to improve. Since these are also characteristics of an A3, let’s see how they fit together.

A typical A3 consists of the following steps¹:

1. Identify problem
2. Research
3. Root cause analysis
4. Propose counter measures
5. Develop target state
6. Create implementation plan
7. Develop follow-up plan with predicted outcome

Since A3 is a problem solving tool it doesn’t map 1-to-1 to a code review. The nature of a code review is different then a problem solving process (the code review could be input for a real A3 though).
Some parts can be used, so I mapped this to a code review as follows:

This will give you the report in a nice and concise format.

Then starts the next and probably important part of the code review: The follow up. This means discuss the review with your client and the team members. Come up with an implementation and follow-up plan together with the team.

Happy code reviewing!

1. http://www.coe.montana.edu/ie/faculty/sobek/a3/steps.htm

1. a not totally trivial example implemented using JPA and Scala
2. an event sourced implementation using explicit state changes
3. a straightforward translation of the mutable event sourced implementation into an immutable implementation
4. encoding domain knowledge into the type system to make the domain easier to understand and reduce the number of runtime error checks

In this last part we’ll reduce the boilerplate code related to handling events and as a bonus we’ll also make handling validation a bit nicer. But before we take a deep dive into the code, let’s consider the design of the last three Invoice implementations.

Design comparison

Since this part is mostly about reducing boilerplate from the typed implemention of invoice without significantly affecting the design of our domain classes, it makes sense to take a quick look at the design of the three event sourced implementations so far: mutable (part 2), immutable (part 3), and typed/monadic (part 4/5). Which one, if any, is the “best”? Let’s compare them based on XP’s rules of simple design.

So all implementations pass the tests and therefore satisfy the requirements. Splitting the invoice class into three subclasses improved expressiveness and reduced duplication (no need for some runtime checks), at the cost of adding some new methods and classes.

But the main highlight is that we were able to radically redesign and reimplement our domain code with minimal impact on our clients or infrastructure. We still generate exactly the same events as in the first mutable event sourced implementation and only the calling syntax was slightly changed (which usually is contained in a controller or service layer anyway). The reporting side of the application, any views, integration with external systems, etc. are unaffected!

So by having our events as a stable abstraction we have greatly decoupled the components of our application while making each component more cohesive. This gives us confidence that as requirements change we can keep our domain clean and simple, without needing to compromise our implementation with respect to durability or reporting needs.

So now that we have this out of the way, let’s get back into the code.

Composing event handlers

Let’s take a look at the applyEvent method in the DraftInvoice class from part 4:

def applyEvent = {
  case event: InvoiceRecipientChanged => applyRecipientChanged(event)
  case event: InvoiceItemAdded => applyItemAdded(event)
  case event: InvoiceItemRemoved => applyItemRemoved(event)
  case event: InvoiceSent => applySent(event)
  case event => unhandled(event)
}

It’s basically checking the type of the event and then dispatching to the correct event handler using a case-block. Each event handler is simply a function that takes an event of the correct type and returns the appropriate response. We could try to use some reflection magic to remove the need for this method, but that feels like cheating. Let’s try to if we can compose our typed event handlers into the generic applyEvent method instead.

Fortunately, Scala has a built-in trait called PartialFunction that has the useful method orElse that does exactly what we need. We just need to turn our event handling functions into partial functions to make this work.

We do this by making the type of our handlers explicit using a new class EventHandler and adding an implicit conversion from our handler type to partial functions. The partial function checks the event’s type and invoke the handler if the type is correct:

protected class EventHandler[Event, +Result](callback: Event => Result) {
  def apply(event: Event) = // [... code omitted ...]

  def applyFromHistory(event: Event) = callback(event)
}

protected def handler[A, B](callback: A => B) = new EventHandler(callback)

implicit protected def handlerToPartialFunction[A, B](handler: EventHandler[A, B])(implicit m: Manifest[A]) =
  new PartialFunction[AnyRef, B] {
    def isDefinedAt(event: AnyRef) = m.erasure.isInstance(event)

    def apply(event: AnyRef) = handler.applyFromHistory(event.asInstanceOf[A])
  }

Here an instance of EventHandler can be created using the handler method. The EventHandler class has a applyFromHistory method which simply passes the event to the provided callback.

The handlerToPartialFunction takes an EventHandler and an implicit Manifest and turns it into a partial function that takes an event of type AnyRef (Scala’s equivalent of Java’s Object class). When the partial function is invoked it downcasts the event and delegates to the EventHandler‘s applyFromHistory method. But the partial function is only defined when the provided event’s type matches the type expected by the event handler!

Using these definitions we can now update our typed event handlers and easily compose them for use with applyEvent, using the standard PartialFunction.orElse method:

def applyEvent = applyRecipientChanged orElse applyItemAdded orElse applyItemRemoved orElse applySent

private def applyRecipientChanged = handler {event: InvoiceRecipientChanged =>
  copy(event :: uncommittedEvents, recipient_? = event.recipient.isDefined)
}

Now the applyEvent method is just a single line listing all the applicable event handlers, without all the boilerplate. The typed event handlers are slightly changed to include the call to the handler factory method.

Extracting uncommitted events

The other part of boilerplate code was related to storing and managing the list of uncommitted events. Each aggregate root subclass needed to provide an accessor for the uncommittedEvent field and implement the markCommitted method. Each event handler then had to ensure the new event was prepended to the list of uncommitted event, which is somewhat error-prone.

The first step to fixing this is to remove the list of uncommitted events from the aggregate root and to start explicitly passing it into our methods, which then returns the updated list together with the updated invoice. This will certainly clean up the invoice subclasses, such as the PaidInvoice listed below.

class PaidInvoice extends Invoice {
  def applyEvent = unhandled
}

Unfortunately it makes methods like pay and applyPaymentReceived rather ugly, and we’re not even talking yet about the client code which now needs to manually manage the list of uncommitted events:

def pay(uncommittedEvents: List[Any]) = 
  applyPaymentReceived(InvoicePaymentReceived(id, new LocalDate), uncommittedEvents)
  
private def applyPaymentReceived(event: InvoicePaymentReceived, uncommittedEvents: List[Any]) =
  (event :: uncommittedEvents, new PaidInvoice)

Fortunately, monads can help us with that. But first we’ll make our types explicit (a recurring theme of this series).

Let’s look at the types we have now. The pay method above has the type List[Any] => (List[Any], PaidInvoice). Let’s call the type of pay a Behavior which returns a Reaction when triggered by passing it a list of events. A reaction can either be Accepted for success or Rejected when failed:

trait Reaction[+T]
case class Accepted[+T](events: List[Any], result: T) extends Reaction[T]
case class Rejected(message: String) extends Reaction[Nothing]

trait Behavior[+A] {
  protected def apply(events: List[Any]): Reaction[A]

  // [... code omitted ...]

  def reaction = apply(Nil)

  def changes = reaction.asInstanceOf[Accepted[_]].events

  def rejected = reaction.asInstanceOf[Rejected].message
}

The Behavior class defines an abstract method apply that takes the current list of uncommitted events as argument and should implement the specific behavior we want. It also adds a reaction method that invokes the behavior with the empty list. The changes and rejected methods are just there for convenience.

We’ll also define a few additional methods to create some useful behaviors:

object Behaviors {
  def behavior[T](callback: List[Any] => Reaction[T]) = new Behavior[T] {
    protected def apply(events: List[Any]) = callback(events)
  }

  def accept[T](result: T) = behavior(events => Accepted(events, result))

  def reject(message: String) = behavior(_ => Rejected(message))

  def record(event: Any) = behavior(events => Accepted(event :: events, ()))

  def guard(condition: Boolean, message: => String) = if (condition) accept() else reject(message)
}

The behavior method lets us easily create a new behavior by providing a callback, instead of having to define a new anonymous subclass implementation every time.

Accept doesn’t modify the list of uncommitted events but simple returns a specific result, reject forgets about any uncommitted events and returns an error message, record records the provided event and returns an uninteresting value, and guard rejects when the condition does not hold and returns an uninteresting value otherwise.

Now we’ll just need two more pieces to complete the puzzle. The first thing we need to be able to do is to compose two behaviors into a single new behavior. We do this by first triggering the first behavior, and if successful, passing the result from the first behavior into the second behavior. This is the monad bind operation, which is called flatMap in Scala and we make it part of our Behavior trait:

trait Behavior[+A] {
  protected def apply(events: List[Any]): Reaction[A]

  def flatMap[B](next: A => Behavior[B]) = behavior {events =>
    this(events) match {
      case Accepted(updatedEvents, result) => next(result)(updatedEvents)
      case Rejected(message) => Rejected(message)
    }
  }
  
  // [... code omitted ...]
}

With all of this in place, we can define the EventHandler.apply method which is used by our invoice implementation to call event handlers when not reloading from history:

protected class EventHandler[Event, +Result](callback: Event => Result) {
  def apply(event: Event) = record(event) flatMap (_ => accept(callback(event)))

  // [... code omitted ...]
}

The apply method simply records the event and then accepts whatever the result of the event handler’s callback is. The return value of record is ignored, since it is of type Unit and not interesting.

So let’s take a look at the DraftInvoice.send method of our new Invoice.scala:

def send: Behavior[SentInvoice] =
  guard(readyToSend_?, "recipient and items must be specified before sending") flatMap {_ =>
    val now = new LocalDate
    applySent(InvoiceSent(id, sentDate = now, dueDate = now.plusDays(14)))
  }

// [... code omitted ...]

private def applySent = handler {event: InvoiceSent => new SentInvoice(id, event.dueDate)}

This send method’s only changes are the return type (Behavior[SentInvoice]) and the use of the guard method to perform validation. Again flatMap is used to sequence the behavior. So if the guard fails the applySent is never performed. The implementation of applySent is now also cleaned up and no longer has to worry about recording the event as this is taken care of by EventHandler.apply.

Client code is now unfortunately a bit more complicated, since it needs to deal with the monad:

"draft invoice" should {
  val invoice: DraftInvoice = Invoice.loadFromHistory(Seq(InvoiceCreated(1)))

  "support adding invoice items" in {
    val updated = invoice.addItem("Food", "2.95") flatMap (_.addItem("Water", "1.95")) flatMap (_.removeItem(1))

    updated.changes must contain(InvoiceItemAdded(1, InvoiceItem(1, "Food", "2.95"), "2.95"))
    updated.changes must contain(InvoiceItemAdded(1, InvoiceItem(2, "Water", "1.95"), "4.90"))
    updated.changes must contain(InvoiceItemRemoved(1, InvoiceItem(1, "Food", "2.95"), "1.95"))
  }

  "not be ready to send" in {
    invoice.send.rejected must beEqualTo("recipient and items must be specified before sending")
  }
}

Here the need to sequence the different behaviors using flatMap is ugly. Fortunately, this can be improved by using a more convenient name (bind or >>=) and in many cases, a service will only invoke a single method on an aggregate at a time, so the need for sequencing behaviors may be rare. Validation checking has improved, since there is no longer a need to catch exceptions, improving readability and making it easier to combine multiple validation results into one.

Conclusion

This series of blog posts has taken us through a whirlwind tour of modeling a simple invoice example. Starting out with a straightforward JPA implemention we quickly moved to event sourcing that made it possible to implement an immutable domain model. This allowed us to raise the level of abstraction of our code, increase clarity, and increase type safety. Finally we used some functional programming techniques to reduce boilerplate. Even with the additional classes and event handling methods, the monadic version of Invoice is only slightly larger than the mutable event sourced invoice. The test code is significantly smaller, since it no longer needs to test for various run-time checks, as the compiler takes care of that.

But the most significant advantage of event sourcing is the ability to change the implementation of the domain in radical ways, making it possible to keep up with changing business requirements and allowing us to keep improving the domain as our knowledge and understanding improves. All this with very little impact on the rest of the system and no need to perform database migrations.

Furthermore, our reporting and querying needs are also decoupled from the domain. Now we can easily use an RDBMS, document store, lucene index, graph database, and/or anything else that is most appropriate for our specific querying and reporting needs, without any impact on our domain model.

Finally, critical business data has become much more durable, as we only add events, and never update or delete. Full historical data is maintained, which potentially holds a great amount of business value. We now also have access to full audit logs for regulatory or debugging purposes, etc. This is incredibly valuable!

The design space for CQRS, event sourcing, and immutable domain models is still wide open. It will be very interesting to see how this evolves, and when and where these techniques are applicable. Certainly it makes it possible to apply standard functional programming techniques to “inherently” mutable business domains, with all the goodness that entails.