Other parts

• Part 1 – Introduction
• Part 2 – Consistency
• Part 3 – Redis Event Store
• Part 4 – Conflict Resolution
• Part 5 – Refactoring and Transactions
• Part 6 – Users, Authentication, Authorization

Code

You can find the code associated with this part at github on the part-6 branch.

Since quite some time has past since the previous part some other upgrades were made as well:

• Scala 2.10.0 and Play 2.1-RC1 were released. The code has been updated accordingly. This mainly reduced the amount of code needed to work with JSON.
• The stable version of Redis is now 2.6.7, which supports Lua scripts. So the 2.4.x Redis Watch/Multi/Exec event store implementation was removed.

User Accounts

Most applications need some level of user authentication and authorization support. Since we’re building an event sourced example application we’ll first define the events we need for managing users and active sessions:

[code lang="scala"]
sealed trait UserEvent extends DomainEvent {
def userId: UserId
}
case class UserRegistered(userId: UserId, email: EmailAddress, displayName: String, password: Password) extends UserEvent
case class UserProfileChanged(userId: UserId, displayName: String) extends UserEvent
case class UserEmailAddressChanged(userId: UserId, email: EmailAddress) extends UserEvent
case class UserPasswordChanged(userId: UserId, password: Password) extends UserEvent
case class UserLoggedIn(userId: UserId, token: AuthenticationToken) extends UserEvent
case class UserLoggedOut(userId: UserId) extends UserEvent
[/code]

From the event definitions it is quite clear what user related functionality our system supports. Besides registering, logging in and logging out, users can also change their profile information (display name), email address, and password.

To ensure we always hash passwords securely and never accidentally display them a custom Password class is defined, which uses the scrypt password hashing algorithm:

[code lang="scala"]
case class Password private (hash: String) {
require(hash.startsWith("$s0$"), "invalid password hash")

def verify(password: String): Boolean = SCryptUtil.check(password, hash)

override def toString = ""
}
object Password {
def fromHash(hash: String): Password = Password(hash)
def fromPlainText(password: String): Password = Password(SCryptUtil.scrypt(password, 1 << 14, 8, 2))

implicit val PasswordFormat: Format[Password] = valueFormat(fromHash)(_.hash)
}
[/code]

Similar classes are defined for EmailAddress and AuthenticationToken. The authentication token is stored in the user's session so we can lookup the current user when an HTTP request is made. This is implemented as part of the Users class:

[code lang="scala"]
case class Users(
private val byId: Map[UserId, RegisteredUser] = Map.empty,
private val byEmail: Map[EmailAddress, UserId] = Map.empty,
private val byAuthenticationToken: Map[AuthenticationToken, UserId] = Map.empty) {

// [... code omitted ...]

def findByAuthenticationToken(token: AuthenticationToken): Option[RegisteredUser] =
byAuthenticationToken.get(token).flatMap(byId.get)

// [... code omitted ...]
}
[/code]

Now that we not only store posts but users as well, the global state of the application is now captured in the ApplicationState class, which simply combines the Posts and Users classes and dispatches incoming updates to its parts:

[code lang="scala"]
case class ApplicationState(posts: Posts = Posts(), users: Users = Users()) {
def update(event: DomainEvent, revision: StreamRevision) = event match {
case event: PostEvent => copy(posts = posts.update(event, revision))
case event: UserEvent => copy(users = users.update(event, revision))
case _ => sys.error(s"unknown event: $event")
}

def updateMany(events: Seq[(DomainEvent, StreamRevision)]) = events.foldLeft(this) {
case (state, (event, streamRevision)) => state.update(event, streamRevision)
}
}
[/code]

The User trait is defined as follows:

[code lang="scala"]
sealed trait User {
def displayName: String

/**
* @return `Some(registeredUser)` if this is a registered user,
* `None` otherwise.
*/
def registered: Option[RegisteredUser] = None

// [... authorization code ommitted ...]
}
[/code]

By not having to worry about mapping classes to a (relational) database we can freely define our classes to match our application's needs. In this case there are four different implementations of this trait. They all have a displayName and registered method in common, but are used in different situations:

[code lang="scala"]
/**
* A deleted or non-existent user.
*/
case class UnknownUser(id: UserId) extends User {
def displayName = "[deleted]"
}

/**
* A user that we know the name of, but nothing else. Consider a comment posted
* by a guest (where they only have to specify their name).
*/
case class PseudonymousUser(displayName: String) extends User

/**
* A visitor to the site is represented by the guest user object.
*/
case object GuestUser extends User {
def displayName = "Guest"

// [... authorization code ommitted ...]
}

/**
* A registered user who may have an active session identified by the
* `authenticationToken`.
*/
case class RegisteredUser(
id: UserId,
revision: StreamRevision,
email: EmailAddress,
displayName: String,
password: Password,
authenticationToken: Option[AuthenticationToken] = None)
extends User {

override def registered = Some(this)

// [... authorization code ommitted ...]
}
[/code]

The current user

In most applications there is a current user. We want this user to be available to all our controller actions and views. We also want to store the current user's id as part of a commit to our event store, so that we can easily trace who performed a specific action.

Instead of reimplementing this in every controller action we'll add some helper methods, so that controller actions can stay focused. Play! makes it easy to do so using action composition.

First a new trait is defined so that we can decouple the controller from the raw memory image API. An instance of this trait will be passed to each controller when constructed:

[code lang="scala"]
/**
* Actions available to controllers that make use of a memory image.
*/
trait ControllerActions[State, -Event] extends Controller { outer =>
/**
* The type of blocks that only need to query the current state of the
* memory image.
*/
type QueryBlock[A] = State => ApplicationRequest[A] => Result

/**
* The type of blocks that wish to read and modify the memory image.
*/
type CommandBlock[A] = State => ApplicationRequest[A] => Transaction[Event, Result]

/**
* Runs the given `block` using the current state of the memory image.
*/
def QueryAction(block: QueryBlock[AnyContent]): Action[AnyContent]

/**
* Runs the given `block` using the current state of the memory image
* and applies the resulting transaction.
*/
def CommandAction(block: CommandBlock[AnyContent]): Action[AnyContent]

// [... code omitted ...]
}
[/code]

As you can see two kinds of actions are provided:

1. Query actions - used when you want to access the current user and application state, but do not need to modify it.
2. Command actions - used when you want to generate and commit new events.

Each action expects a callback block as a parameter. The block will be invoked with the current memory image state and an ApplicationRequest, which wraps the standard Play! Request and adds additional user related information:

[code lang="scala"]
trait CurrentUserContext {
/**
* The current authenticated user or the guest user.
*/
def currentUser: User
}

trait UsersContext {
/**
* All known users and sessions.
*/
def users: Users
}

/**
* Context for use as an implicit parameter to views.
*/
trait ViewContext extends CurrentUserContext {
def flash: Flash
}

/**
* Extend Play's `Request` with user context information.
*/
class ApplicationRequest[A](
request: Request[A],
val currentUser: User,
val users: Users)
extends WrappedRequest(request) with ViewContext with UsersContext
[/code]

So instead of passing the Play! Request to views we now pass it a ViewContext, so that we can easily control which information is available to a view (in this case, just the current user and the flash message information).

The implementation of this trait uses a MemoryImage based on the ApplicationState class and can be found at MemoryImageActions.scala. Here's the implementation of the QueryAction method:

[code lang="scala"]
/**
* Implements `ControllerActions` using the (global) memory image containing
* the `ApplicationState`.
*/
class MemoryImageActions
(memoryImage: MemoryImage[ApplicationState, DomainEvent])
extends ControllerActions[ApplicationState, DomainEvent] {

override def QueryAction(block: QueryBlock[AnyContent]) = Action { request =>
val state = memoryImage.get
block(state)(buildApplicationRequest(request, state))
}

// [... CommandAction omitted ...]

private def buildApplicationRequest[A](request: Request[A], state: ApplicationState) = {
val currentUser = request.session.get("authenticationToken")
.flatMap(AuthenticationToken.fromString)
.flatMap(state.users.findByAuthenticationToken)
.getOrElse(GuestUser)

new ApplicationRequest(request, currentUser, state.users)
}
}
[/code]

As you can see a QueryAction is implemented as a Play! Action that reads the current state of the memory image and uses it to build the ApplicationRequest with the current user. If there is no authentication token the GuestUser is used instead. The provided block is then invoked. The CommandAction is similar, but uses the MemoryImage.modify to also commit the generated events.

Finally, we do not want to expose the entire application state to each controller. The UsersController only needs to see the users, while the PostsController only needs to see the posts. The view method on ControllerActions takes care of that:

[code lang="scala"]
trait ControllerActions[State, -Event] extends Controller { outer =>
// [... action methods omitted ...]

/**
* Only expose part of the state of the memory image using the provided
* function `f`.
*/
def view[S, E <: Event](f: State => S) = new ControllerActions[S, E] {
def QueryAction(block: QueryBlock[AnyContent]) = outer.QueryAction { state => request =>
block(f(state))(request)
}
def CommandAction(block: CommandBlock[AnyContent]) = outer.CommandAction { state => request =>
block(f(state))(request)
}
}
[/code]

Controllers that use the memory image can now simply have an instance of ControllerActions passed in at construction time to define actions. For example, the PostsController.index action now becomes

[code lang="scala"]
object PostsController
extends PostsController(Global.MemoryImageActions.view(_.posts))
class PostsController(actions: ControllerActions[Posts, PostEvent]) {
// Import the action and controller methods directly.
import actions._

/**
* Show an overview of the most recent blog posts.
*/
def index = QueryAction { posts => implicit request =>
Ok(views.html.posts.index(posts.mostRecent(20)))
}

// [... other code omitted ...]
}
[/code]

Uniqueness of email addresses

One common requirement is that email addresses must be unique across all users. In an application backed by a relational database this is as easy as adding a unique constraint^[1], but an event store does not provide anything similar. We could use the email address as the user's event stream identifier, but that makes it hard for a user to change their email address without creating a brand new event stream. So what we'll do instead is to store a map of email address to user ids and fill this map on demand. For unit testing the following implementation works fine:

[code lang="scala"]
val claimedEmailAddresses = collection.mutable.Map.empty[EmailAddress, UserId]
def claim(email: EmailAddress, requestedUserId: UserId): UserId =
claimedEmailAddress.getOrElseUpdate(email, requestedUserId)
[/code]

In other words, the first time an email address is used a new user id is assigned. Afterwards, the same user id is always returned. For the production implementation we use a Redis hash together with the HSETNX command:

[code lang="scala"]
class RedisEmailRegistry(jedis: Jedis, redisKey: String) {
def claim(email: EmailAddress, requestedUserId: UserId): UserId = {
val result: Long = jedis.hsetnx(redisKey, email.toString, requestedUserId.toString)
result match {
case 0L =>
val existingUserId = jedis.hget(redisKey, email.toString)
UserId.fromString(existingUserId).getOrElse(sys.error(s"cannot parse user id: $existingUserId"))
case 1L =>
requestedUserId
case _ =>
sys.error(s"unexpected Redis return value: $result")
}
}
}
[/code]

So when we register a new user we first map the email address to a user id and then generate the UserRegistered event. If the email address is already taken the event store will return a conflict, since we always expect the event stream to be empty for a new user:

[code lang="scala"]
class UsersController(
actions: ControllerActions[Users, UserEvent],
claimEmailAddress: (EmailAddress, UserId) => UserId) {
import actions._

// [... other actions omitted ...]

def register = CommandAction { state => implicit request =>
val form = registrationForm.bindFromRequest
form.fold(
formWithErrors =>
abort(BadRequest(views.html.users.register(formWithErrors))),
registration => {
val (email, displayName, password) = registration
val userId = claimEmailAddress(email, UserId.generate)
Changes(
StreamRevision.Initial,
UserRegistered(userId, email, displayName, password): UserEvent)
.commit(
onCommit = Redirect(routes.UsersController.registered),
onConflict = _ => BadRequest(views.html.users.register(form.withGlobalError("duplicate.account"))))
})
}
[/code]

Authors and authorization

When it comes to security authentication is often the "easy" part, mainly limiting itself to user registration and logging in. Authorization is often much harder, since it affects the entire application. Now that we have the concept of a user we change blog posts and comments to include the user id of the author or commenter. We then implement authorization rules to ensure only the author can delete a blog post, etc. We add the authorization methods the User trait:

[code lang="scala"]
trait User {
// [... code omitted ...]

def canAddPost: Boolean = false
def canAddComment(post: Post): Boolean = false
def canEditPost(post: Post): Boolean = false
def canDeletePost(post: Post): Boolean = false
def canDeleteComment(post: Post, comment: Comment): Boolean = false

// [... code omitted ...]
}
[/code]

By default a user is not authorized to do anything, except for a RegisteredUser or a GuestUser:

[code lang="scala"]
case object GuestUser extends User {
// [... other code omitted ...]

override def canAddComment(post: Post) = true
}

case class RegisteredUser(/* ... code omitted ... */) extends User {
// [... other code omitted ...]

override def canAddPost = true
override def canAddComment(post: Post) = true
override def canEditPost(post: Post) = post.isAuthoredBy(this)
override def canDeletePost(post: Post) = post.isAuthoredBy(this)
override def canDeleteComment(post: Post, comment: Comment) =
post.isAuthoredBy(this) || comment.isAuthoredBy(this)

}
[/code]

As you can see a guest user can only add comments to a post. Registered users can add posts, comments, and can edit or delete posts and comments authored by themselves.

Now controllers and views can easily invoke the correct authorization method to see if the user is allowed to see a certain button or to perform a certain action. Here's is an example when adding a post (from PostsController), which uses the AuthenticatedCommandAction to require a logged in user:

[code lang="scala"]
def edit(id: PostId, expected: StreamRevision) = AuthenticatedCommandAction {
user => posts => implicit request =>
posts.get(id).filter(user.canEditPost) map { post =>
// [... code omitted ...]
} getOrElse {
abort(notFound)
}
}
[/code]

Event authorization

Unfortunately, it is easy to forget to add the correct authorization check to a controller action, since any action is allowed by default. And an attacker only has to find a single mistake to break the security of an application. With event sourcing we can optionally add an additional layer of security: we can check the events being committed against the authorization level of a user. To do this we add one additional method to the User trait:

[code lang="scala"]
trait User {
// [... code omitted ...]

def authorizeEvent(state: ApplicationState): DomainEvent => Boolean =
event => false
}
[/code]

This method takes the current ApplicationState and returns a function that checks if this user is allowed to commit a specific domain event. The default implementation always forbids any change.

For guest users and registered users we override this method. Here's the code for the guest user:

[code lang="scala"]
case object GuestUser extends User {
// [... code omitted ...]

override def authorizeEvent(state: ApplicationState) = {
case event: UserRegistered => true
case event: UserLoggedIn => true
case event: CommentAdded => state.posts.get(event.postId).exists(this.canAddComment)
case _ => false
}
}
[/code]

Now we have everything to implement the CommandAction helper method from the MemoryImageActions class:

[code lang="scala"]
class MemoryImageActions(memoryImage: MemoryImage[ApplicationState, DomainEvent])
extends ControllerActions[ApplicationState, DomainEvent] {
// [... code omitted ...]

override def CommandAction(block: CommandBlock[AnyContent]) = Action { request =>
memoryImage.modify { state =>
val applicationRequest = buildApplicationRequest(request, state)
val currentUser = applicationRequest.currentUser
val transaction = block(state)(applicationRequest)
if (transaction.events.forall(currentUser.authorizeEvent(state))) {
transaction.withHeaders(currentUser.registered.map(user => "currentUserId" -> user.id.toString).toSeq: _*)
} else {
Transaction.abort(notFound(request))
}
}
}

// [... code omitted ...]
}
[/code]

As you can see it first builds the ApplicationRequest, just like the QueryAction method. But it then inspects the transaction returned by the provided block to see if the current user is authorized to commit all the generated events. If so, it adds the current user id as a header to the commit and applies it to the memory image. Otherwise, the transaction is aborted with a 404 Not Found response.

The main advantage of this level of authorization is that events must be explicitly whitelisted. So when new functionality is added it will simply not work until the whitelist is updated. This is the opposite of views and controllers, where new functionality works by default and authorization checks must be explicitly added. A step that is easy to forget!

Summary

Adding users to our example application caused some major re-design to manage authentication and authorization. Compared to this event sourcing was just a minor detail, the same kind of re-design would have been needed if users were stored in a traditional database. Event sourcing does provide an additional layer of security by allowing us to authorize committed events. We can also fully audit all changes, as every commit includes the committing user's id.

We have now build a fairly complete application. In the next part we'll start making use of our event sourced model to integrate with the outside world. We'll do this by reacting to the events as they are committed to the event store.

In search for a good representation I didn’t want to use an average grade. All individual grades are too important to get lost in an average. Also a few 1′s and 6′s could give a very wrongful average of 3,5. So all numbers are important, but they need to be compressed in a ‘faster’ information radiator.

What I tried, was to make a weather ‘backcast’ out of it. I found some weather icons on the interwebz and translated the overall histogram to a description of our teams weather conditions. So for example, “Cloudy with here and there some sunshine” (a few 3 and 4′s and one 2) or “Sunny with some local thunderstorms” (mostly 2′s and one 5). In this way every observer can see what the happiness of this team is and if they want to get more detailed information, they have to ask a team member. This way we can make sure the correct context is given along with the current happiness.

Of course it is a good idea to put the actions that will improve the scores also on the board. You can add: “When we want the sun to shine tomorrow, we need the following improvements:” and then the list of actions. So they are visible for everyone, even (especially) for the ‘innocent’ bystander.

Do you have any improvements?

While reading blogs about Agile games, I came across this link of Karl Scotland.
Because I always like combining programming with building something that gets across Agile principles (our Battleship game is also a good example) this seemed like a good new project for our Fridays at Zilverline HQ. Bob Forma and me started this and within a few days we came up with this (You can check the source code here)

Each feature has a random generated effort which you can see on the ‘sticky’. Each day (if you press ‘Run 1 day’) for each stage (Analysis, Code, etc.) productivity is generated of 4, 6 or 8 hours. The stage then looks if there is ready work to be found in its preliminary stage, if there is, that work is pulled to ‘in progress’ of course the ‘Work in progress Limit’ is respected at all times. The velocity is then divided between the features that are ‘in progress’. Once a feature has zero remaining effort, it is moved to the ‘ready’ column of that stage.

If you click on a feature, the text on it changes to ‘Workdays’ which is time this feature spend in ‘in progress’ and ‘Waitdays’ which is time it has spend in a ‘Ready’ column. I also added Totaldays which is a sum of these two. If you use ‘RESTART’ the board gets reset and you can experiment with the work in progress limits and their effect on flow.

We hope you like it and feel free to ask for new options in the comments (or branch and add them yourself)!

Other Parts

• Part 1 – Introduction
• Part 2 – Consistency
• Part 3 – Redis Event Store
• Part 4 – Conflict Resolution
• Part 5 – Refactoring and Transactions
• Part 6 – Users, Authentication, Authorization

Code

You can find the code associated with this part at github on the part-5 branch.

Refactoring

The main changes we want to make before improving the current commit code to handle transient conflicts is to move it out of the PostsController, so that we use it from other parts of the application. But the current implementation is tied directly to Posts and PostEvents. So first we remove the following dependencies from this method:

1. PostEvent conflict resolution.
2. How to derive the PostId from the PostEvent.
3. How to turn a PostId into a event stream identifier (currently just using toString)

To solve the first problem we introduce a new trait ConflictsWith:

[code lang=scala]
/**
* Compares committed events against an attempted events to check for
* conflicts.
*/
trait ConflictsWith[-Event] {
/**
* Checks each committed event from `conflict` for conflicts with the `attempted` events.
* Any committed events that conflict are returned.
*/
def conflicting[A <: Event, B <: Event]
(conflict: Conflict[A], attempted: Seq[B]): Option[Conflict[A]]
}
object ConflictsWith {
/**
* Builds a new `ConflictsWith` based on the `conflicts` predicate.
*/
def apply[Event](conflicts: (Event, Event) => Boolean) = new ConflictsWith[Event] {
override def conflicting[A <: Event, B <: Event]
(conflict: Conflict[A], attempted: Seq[B]): Option[Conflict[A]] =
conflict.filter(a => attempted.exists(b => conflicts(a.event, b)))
}
}
[/code]

ConflictsWith is a type class that defines how to resolve conflicts for a certain type of events. By making it a separate trait (instead of a function) the Scala compiler can automatically pass implementations as an implicit parameter where needed. The definition of the Conflict class has also changed slightly, to make it easier to filter the events while preserving meta-information.

The apply method of the ConflictsWith companion object is helpful to create instances of ConflictsWith based on a simple predicate. The predicate must check if two events conflict or not. The PostEvent conflict resolution implementation becomes:

[code lang=scala]
implicit val PostEventConflictsWith: ConflictsWith[PostEvent] = ConflictsWith {
case (a: PostCommentEvent, b: PostCommentEvent) => a.commentId == b.commentId
case (_: PostCommentEvent, _) => false
case _ => true
}
[/code]

We use Scala’s convenient partial function syntax to define all the cases, without having to explicitly use the match keyword.

To solve the second and third problem, and to support different type of event streams in a single event store, we introduce another trait EventStreamType:

[code lang=scala]
/**
* Event stream type information.
*/
trait EventStreamType[StreamId, Event] {
/**
* Convert a stream identifier to a string. Used by the event store to persist
* the stream identifier.
*/
def toString(streamId: StreamId): String

/**
* Extract stream identifier from `event`.
*/
def streamId(event: Event): StreamId

/**
* Cast `event` to the `Event` type.
*
* @throws ClassCastException if `event` is not of type `Event`.
*/
def cast(event: Any): Event
}
object EventStreamType {
def apply[StreamId, Event]
(writeStreamId: StreamId => String,
eventToStreamId: Event => StreamId)
(implicit manifest: Manifest[Event]) =
// [...]
}
[/code]

This trait captures the relationship between an event stream identifier type and the type of events that can be stored in the associated stream. It also provides methods to extract the event stream identifier from an event and to turn an event stream identifier into its string representation. Finally it allows casting a value of any type to the type of the event. With these two traits defined we can move the commit method from the PostsController into the MemoryImage class. But first we look into using the event store for different event stream types.

Multiple event stream types

Now that we’ve associated an event streams identifier type with the event type using the EventStreamType trait, we can make our event store contra-variant. In other words, when we have an event store of type EventStore[DomainEvent] we can use it as an EventStore[PostEvent], if PostEvent is a subtype of DomainEvent.

But we can only do this by changing event store methods that return events to take into account the EventStreamType, otherwise the Scala compiler will not allow us to treat a EventStore[DomainEvent] as a EventStore[PostEvent] (since reading from such a store could return DomainEvents, while the caller would only expect PostEvents). So the definition of CommitReader becomes:

[code lang=scala]
/**
* Reads commits from the event store.
*/
trait CommitReader[-Event] {
// [...]

/**
* Reads all commits `since` (exclusive) up `to` (inclusive). Events are
* filtered to only include events of type `E`.
*/
def readCommits[E <: Event]
(since: StoreRevision, to: StoreRevision)
(implicit manifest: Manifest[E]): Stream[Commit[E]]

/**
* Reads all commits from the stream identified by `streamId` that occurred
* `since` (exclusive) up `to` (inclusive).
*
* @throws ClassCastException the stream contained commits that did not
* have the correct type `E`.
*/
def readStream[StreamId, E <: Event]
(streamId: StreamId,
since: StreamRevision = StreamRevision.Initial,
to: StreamRevision = StreamRevision.Maximum)
(implicit descriptor: EventStreamType[StreamId, E]):
Stream[Commit[E]]
}
[/code]

Other methods in the event store traits are similarly changed. The methods can be used much like before, except that you’ll have to provide the expected event type (readCommits) or that Scala will look for an appropriate instance of EventStreamType (readStream):

[code lang=scala]
implicit val PostEventStreamType: EventStreamType[PostId, PostEvent] =
EventStreamType(streamId => streamId.toString, event => event.postId)

val myCommits = eventStore.reader.readStream(myPostId)
[/code]

Here the Scala compiler knows the type of myPostId and will infer that the events must be of type PostEvent^[1], based on the existence of PostEventStreamType. So Scala will correctly infer the type of myCommits to be Stream[Commit[PostEvent]], so you will not have to do any type casting.

Commit parameters

Another change is that we introduce a new Changes type that combines the three parameters provided to the event store committer’s tryCommit method (streamId, expected stream revision, and the event to append). The interface is:

[code lang=scala]
/**
* Represents the changes that can be committed atomically to the event store.
*/
sealed trait Changes[Event] {
type StreamId
def eventStreamType: EventStreamType[StreamId, Event]

def streamId: StreamId
def expected: StreamRevision
def events: Seq[Event]

def withExpectedRevision(expected: StreamRevision): Changes[Event]
}
object Changes {
// [...]

def apply[StreamId, Event]
(expected: StreamRevision, event: Event)
(implicit streamType: EventStreamType[StreamId, Event]): Changes[Event] =
apply(streamType.streamId(event), expected, event)
}
[/code]

So besides the three parameters it also captures the type of the stream identifier and the associated EventStreamType instance, so that only events of the correct type can be committed to the event stream. The withExpectedRevision returns a new copy of the changes with an updated expected StreamRevision, which is useful for retrying a commit in the case of resolved conflicts.

With all these changes in place we can get back to solving transient transaction conflicts.

Atomic memory image modifications

Going back to our application’s storage architecture the following picture emerges:

Remember that in the blog post application there is one event store, each with many memory images (one per application server), all concurrently committing new events, based on the actions performed by the users. We already detect the conflicts that can occur when users work with (potentially) outdated information by comparing the expected stream revision to the actual revision (see the previous part for more details). Let’s call these kind of conflicts user conflicts.

But by adding the blog comment functionality we introduced a potential conflict in the interaction between the memory image and the event store. In the add comment action of PostsController we first read the current state of a blog post and use it to assign a CommentId to the new comment. We then commit a CommentAdded event. If multiple users happen to add comments to a blog post simultaneously, an unresolvable conflict will occur: both new comments will have the same id. This is a transaction conflict^[2][3].

In the current version of the application the commit fails and we’ll notify the user that someone else added a comment with the same id. The user must then resubmit the comment, which will then succeed. But why ask the user to retry an action for these kind of transient conflicts, when computers are much better at doing repetitive work automatically?

What we have to do is to automatically rerun the code that reads the memory image and then generates the event. So we will replace the MemoryImage‘s tryCommit method with a new modify method:

[code lang=scala]
class MemoryImage[State, Event] /* [...] */ {
// [...]

def modify[A](body: State => Transaction[Event, A]): A = ...
}
[/code]

The body parameter is a function that uses the current state of the memory image and returns a Transaction value. This transaction value contains all the information we need to try to commit to the event store. If it turns out that a transient transaction conflict occurred, we will rerun the body function and retry the commit. When the transaction succeeds we return the value returned by body.

The possible values for a Transaction are shown below:

[code lang=scala]
/**
* The transaction to commit to the event store when modifying the memory image.
*/
sealed trait Transaction[+Event, +A] {
/**
* Maps the result of this transaction from `A` to `B` using `f`.
*/
def map[B](f: A => B): Transaction[Event, B]
}
object Transaction {
/**
* Transaction result that completes with `onAbort` when run,
* without committing anything the event store.
*/
def abort[A](onAbort: => A): Transaction[Nothing, A] =
new TransactionAbort(() => onAbort)

implicit def ChangesOps[Event](changes: Changes[Event]) = new ChangesOps(changes)
class ChangesOps[Event](changes: Changes[Event]) {
/**
* Transaction result that will commit the `changes` to the event store.
*/
def commit[A](onCommit: => A, onConflict: Conflict[Event] => A)
(implicit conflictsWith: ConflictsWith[Event]):
Transaction[Event, A] =
new TransactionCommit(changes, () => onCommit, onConflict, conflictsWith)
}

implicit def OptionTransactionOps[Event, A](m: Option[Transaction[Event, A]]) = new OptionTransactionOps(m)
class OptionTransactionOps[Event, A](value: Option[Transaction[Event, A]]) {
/**
* Turns an `Option[Transaction[Event, A]]` into `Transaction[Event, Option[A]]`.
*/
def sequence: Transaction[Event, Option[A]] = value match {
case None => abort(None)
case Some(transaction) => transaction.map(Some(_))
}
}
}
[/code]

The TransactionCommit and TransactionAbort (not listed) are simple case classes to hold parameters until we are ready to commit (or abort) the transaction. The ChangesOps enriches the Changes class with a new commit method that instantiates a TransactionCommit (Scala 2.10 will offer a nicer syntax for “extending” existing classes with new methods using implicit classes). Values of type Option[Transaction[Event, A]] are also extended with the sequence method, which flips the optionality to return a Transaction[Event, Option[A]]

To modify the memory image safely, we can now write code like:

[code lang=scala]
memoryImage.modify { posts => Changes(/* ... */).commit(/* ... */) }
[/code]

The flow chart below describes the full MemoryImage modify algorithm, including user and transaction conflict resolution:

That’s quite a few steps! Let’s go through each step in the left column:

1. Read the current state of the memory image.
2. Pass the state to the provided transaction body.
3. Is the result of the transaction an abort? If yes, return onAbort result and stop.
4. Try to commit the Changes to the event store using the expected stream revision.
5. Did the commit succeed? If yes, return the onCommit result and stop.
6. Was there an update to the event stream that was not yet applied to the memory image state used to run this transaction? If yes, we have a transaction conflict and go back to step 1 to retry.
7. Is there an unresolvable user conflict? If yes, return the result of the onConflict handler and stop.
8. Try to commit again, but now use the actual stream revision.
9. Did the commit succeed? If yes, return the onCommit result and stop. If not, we have a transaction conflict, so go back to step 1 to retry.

This entire process is implemented in MemoryImage.modify. The method is a bit larger than I would like, but it seems to be one of those cases that don’t get any more understandable when split up into smaller pieces.

Now that we have basic transaction support in place we can modify the controller methods to work with this. Here’s the post edit action together with the updatePost helper method:

[code lang=scala]
object edit {
// [...]
def submit(id: PostId, expected: StreamRevision) = Action { implicit request =>
updatePost(id) { post =>

postContentForm.bindFromRequest.fold(
formWithErrors =>
abort(BadRequest(views.html.posts.edit(id, expected, formWithErrors))),

postContent =>
Changes(expected, PostEdited(id, postContent): PostEvent).commit(
onCommit =
Redirect(routes.PostsController.show(id)).flashing("info" -> "Post saved."),
onConflict = conflict =>
Conflict(views.html.posts.edit(id, conflict.actual, postContentForm.fill(postContent), conflict.events))))

} getOrElse {
notFound
}
}
}

/**
* Runs the transaction `body` against the post identified by `postId` and
* returns the result, if it exists. Otherwise `None` is returned.
*/
def updatePost[A](id: PostId)
(body: Post => Transaction[PostEvent, A]): Option[A] =
memoryImage.modify { _.get(id).map(body).sequence }
[/code]

As you can see, if the form validation fails, we abort the transaction and return a BadRequest as usual. Otherwise we return a commit result with the PostEdited event and the appropriate onCommit and onConflict handlers.

Summary

With the addition of the modify method to the MemoryImage we now have a place to put domain logic. Since the example application’s logic is still extremely simple, we can just keep it inside the PostsController. As your application gets more complicated, you should extract the domain logic into separate functions and classes.

The event store (and memory image) now also support multiple types of event streams. We’ll use that in the next part to add user accounts to the example application. We’ll also look into ensuring uniqueness of the email addresses associated with these user accounts.

I have used the following retrospective practice for several years now:

Ask your team members to be quiet for 5 minutes and write down a number between 1 and 6 on a piece of paper. The 1 usually stands for ‘something too good to be true’ and the 6 stand for ‘total disaster’. So if you want to find out how people feel about being part of a certain team, 1 stands for ‘I want to be on this team for the rest of my life’ and 6 stands for ‘I want to get the hell out of here, today!’. Also ask them to think about what has to happen to get your score up by one point (so towards 1):

Mark: So Bob what is your score?
Bob: A 5, our code base is a disaster.
Mark: Ok, what has to happen to make this a 4?
Bob: Time to refactor during the sprint.

Repeat this for all team members and write them in a histogram, it will look something like this:

1
2 XXX
3 X
4
5 XX
6

What change would improve your score?

5: Code-base is a disaster -> Time to refactor during the sprint.
3: Nice team, project can go much faster -> More automated testing for iPhone.
2: Traveling to work takes a lot of time, everything else is very cool! -> 1 Day to work from home.
etc.

I use writing down a score to avoid socially acceptable answers and use the silence to ‘force’ members to think about their grade and what to change. It seems to be very difficult for groups to keep silent for 5 minutes, but I really try to make a point of this. While silent, people don’t get distracted and can give the topic serious thought. It usually takes some time before you have disconnected from your daily work and are taking a step back to think about your opinion.

The numbers 1 and 6 are usually exaggerations because we tend not to give a 1 or a 6 anyway. Also there is no ‘middle number’. If you would use 1 to 5, it is very safe to give a 3. Although this might be comfortable, I think it is better to be a little bit optimistic/pessimistic (e.g. 3 or 4 on a 6 point scale).

Remember that there is no good or bad score. The most important part is that the team members realize what is needed to give a better grade and the possibility for anyone to address these problems. The second most important part is that after this session, everyone knows what the opinion is of his or her colleagues, people will have a better understanding of each other.
There are also no bad improvement suggestions. Even things that seem unrealistic or just silly have to be written down. Things that seem silly to one person, might be normal to another. Things that might be unrealistic now, might be attainable tomorrow. Respect people by taking every suggestion serious.

A weak point of this exercise is that it is difficult to keep everybody’s attention during the discussion. A suggestion of a colleague of mine, was to let team members ask the ‘What has to happen to improve your grade?’ question to each other (still one at a time).

Of course you can use different themes, ‘How likely do you think it is that this functionality will delight our customers’, ‘Will this Scrum-thingy work in this organization’, etc.

Was this post helpful?
1: Totally! I will never read another blog.
6: Not at all, why didn’t my spam filter block this?

Other Parts

• Part 1 – Introduction
• Part 2 – Consistency
• Part 3 – Redis Event Store
• Part 4 – Conflict Resolution
• Part 5 – Refactoring and Transactions
• Part 6 – Users, Authentication, Authorization

Events

To add the new functionality to our application we will first define the new events and supporting data types. Notice that by focusing on the events we are actually thinking about the behaviour of the application first, instead of just the data model. In such a simple application as this blogging application it’s a rather subtle distinction, but when your application is more complicated events can help you to get a good understanding of what your application is supposed to do. Here are the event definitions for adding and deleting comments:

[code lang="scala"]
case class CommentId(value: Int)
object CommentId {
implicit val CommentIdFormat: Format[CommentId] = valueFormat(apply)(unapply)
implicit val CommentIdOrdering: Ordering[CommentId] = Ordering.by(_.value)
}

case class CommentContent(commenter: String, body: String)

// [...]

sealed trait PostCommentEvent extends PostEvent {
def commentId: CommentId
}
case class CommentAdded(postId: PostId, commentId: CommentId, content: CommentContent) extends PostCommentEvent
case class CommentDeleted(postId: PostId, commentId: CommentId) extends PostCommentEvent

object PostEvent {
// [...]

implicit val CommentContentFormat: Format[CommentContent] = objectFormat("commenter", "body")(CommentContent.apply)(CommentContent.unapply)

implicit val PostEventFormat: Format[PostEvent] = typeChoiceFormat(
"PostAdded" -> objectFormat("postId", "content")(PostAdded.apply)(PostAdded.unapply),
"PostEdited" -> objectFormat("postId", "content")(PostEdited.apply)(PostEdited.unapply),
"PostDeleted" -> objectFormat("postId")(PostDeleted.apply)(PostDeleted.unapply),
"CommentAdded" -> objectFormat("postId", "commentId", "content")(CommentAdded.apply)(CommentAdded.unapply),
"CommentDeleted" -> objectFormat("postId", "commentId")(CommentDeleted.apply)(CommentDeleted.unapply))
}
[/code]

Since comments are always part of a blog post, we’ll can use a simple sequential comment identifier. The first comment of a post will have id CommentId(1), the second one CommentId(2), etc. Obviously, we could also generate UUIDs for comments (and it would actually simplify the code), but by working with a sequential identifier we’ll slowly introduce some “domain” logic into our example application. We’ll also use CommentId as a key in a sorted map, so we need to define the Ordering (line 4), which is based on the underlying numeric value.

CommentContent is a simple container for the name of the commenter and the comment text.

The CommentAdded and CommentDeleted events are both subtypes of the PostCommentEvent trait, which in turn extends PostEvent. This makes it easier to distinguish comment related events which will be useful for resolving conflicts automatically.

To store the events in the event store, we also define and extend the necessary Format instances.

Models

Now that we have defined the new events, we can adjust our view models to keep track of comments as part of the post class:

[code lang="scala"]
case class Post(
id: PostId,
revision: StreamRevision,
content: PostContent,
nextCommentId: CommentId = CommentId(1),
comments: SortedMap[CommentId, CommentContent] = SortedMap.empty)

case class Posts(byId: Map[PostId, Post] = Map.empty, orderedByTimeAdded: Seq[PostId] = Vector.empty) {
// [...]

def update(event: PostEvent, revision: StreamRevision): Posts = event match {
// [...]
case CommentAdded(id, commentId, content) =>
modify(id) { post =>
post.copy(
revision = revision,
nextCommentId = CommentId(commentId.value + 1),
comments = post.comments.updated(commentId, content))
}
case CommentDeleted(id, commentId) =>
modify(id) { post =>
post.copy(
revision = revision,
comments = post.comments - commentId)
}
}

private[this] def modify(id: PostId)(f: Post => Post) =
this.copy(byId = byId.updated(id, f(byId(id))))
}
[/code]

The Post class is modified to keep track of the next available CommentId (starting at 1) and also tracks the comments, using a SortedMap^[1] from CommentId to CommentContent. Remember that the representation of the view model is not tied to any kind of database schema, so we can easily change it whenever we need to. It will be automatically rebuild whenever we restart the application!

The update method also has to be changed to match against the new events and update the Post class accordingly. Updating nested, immutable data structures is a bit more involved than the mutable equivalent, so we use a simple helper method to take care of the first two levels of nesting. There are more general solutions (PDF) to this problem, but for now the modify method will do.

Views and controller

Next we add the required routes to conf/routes:

[code]
POST /posts/:postId/comments ⏎
controllers.PostsController.comments.add(postId: PostId, r: StreamRevision)
POST /posts/:postId/comments/:commentId/delete ⏎
controllers.PostsController.comments.delete(postId: PostId, r: StreamRevision, commentId: CommentId)
[/code]

Listing, deleting, and adding comments is all part of the show.scala.html template.

[code lang="html"]

@if(post.comments.nonEmpty) {

Comments

@for((commentId, commentContent) <- post.comments) {

}
}

Add a comment:

@helper.form(action = routes.PostsController.comments.add(post.id, post.revision)) {
@globalErrorsPanel(form)
@conflictsMessagePanel(conflicts)

}
[/code]

As you can see, the template is quite straightforward. The only thing that might be unfamiliar is the invocation of the conflictsMessagePanel template, which we’ll get back to later.

Finally, we need to implement two new controller methods, which can be found in the comments singleton object inside of PostsController:

[code lang="scala"]
// [...]

private[this] def withPost(postId: PostId)(found: Post => Result)(implicit request: Request[_]) = {
posts().get(postId).map(found).getOrElse(notFound(request))
}

object comments {
val commentContentForm = Form(mapping(
"commenter" -> trimmedText.verifying(minLength(3)),
"body" -> trimmedText.verifying(minLength(3)))(CommentContent.apply)(CommentContent.unapply))

def add(postId: PostId, expected: StreamRevision) = Action { implicit request =>
withPost(postId) { post =>
commentContentForm.bindFromRequest.fold(
formWithErrors => BadRequest(views.html.posts.show(post, formWithErrors)),
commentContent =>
commit(expected, CommentAdded(postId, post.nextCommentId, commentContent))(
onCommit = Redirect(routes.PostsController.show(postId)).flashing("info" -> "Comment added."),
onConflict = (actual, conflicts) => Conflict(views.html.posts.show(post, commentContentForm.fill(commentContent), conflicts))))
}
}

def delete(postId: PostId, expected: StreamRevision, commentId: CommentId) = Action { implicit request =>
withPost(postId) { post =>
def deletedResult = Redirect(routes.PostsController.show(postId)).flashing("info" -> "Comment deleted.")
post.comments.get(commentId) match {
case None => deletedResult
case Some(comment) =>
commit(expected, CommentDeleted(postId, commentId))(
onCommit = deletedResult,
onConflict = (actual, conflicts) => Conflict(views.html.posts.show(post, commentContentForm, conflicts)))
}
}
}
}

// [...]
[/code]

The code is a bit more complicated than the basic post actions, since adding and deleting comments requires the presence of a blog post instance. The withPost helper method is used to read the required post from the memory image, or render a 404 Not Found result if the post is not present.

In the case of the add method we then validate the submitted form (line 14). If incorrect, we rerender the page with the error messages (line 15). Otherwise we commit a new CommentAdded event using the provided content and the next available comment id (line 17). If the commit succeeds without conflict, we redirect the user to the blog post with the added comment (line 18). In case there is a conflict, we rerender the form but add some information on the conflicts that occurred (line 19). The delete action is very similar.

This basically completes the addition of some new functionality. The effort required is comparable to a traditional database backed application, which is good to know. But there is one fly in the ointment we need to fix before we can call it a day…

Conflict resolution

Now that we can add comments to post you’ll soon discover that multiple concurrent users will quickly get a conflict. In part 2 we added conflict detection and rendered a placeholder page whenever a conflict occurred. Now that conflicts are more likely to occur, we need to be a bit smarter about resolving these conflicts.

The basic idea is that comment events usually do not conflict, even when they apply to the same post and therefore the same event stream. We can capture this knowledge in a method like this:

[code lang=scala]
def conflictsWith(committed: PostEvent, attempted: PostEvent) =
(committed, attempted) match {
case (a: PostCommentEvent, b: PostCommentEvent) => a.commentId == b.commentId
case (_: PostCommentEvent, _) => false
case _ => true
}
[/code]

This method states that any two PostCommentEvents only conflict when they affect the same comment (line 3). Furthermore, any new blog post related event does not conflict with an already committed PostCommentEvent (line 4). This allows the author to edit the blog post without getting conflicts on added or removed comments. Any other event combination is considered to conflict (line 5). So if you add a comment while someone edited the post, the system will give you a warning and ask you to resubmit your comment, if it is still applicable.

Notice that the main goal of this method is to decide whether we should ask the user for confirmation when a conflict occurs, or whether we should just proceed with the commit even though changes were made while the user was working on their request. This is rather subjective and the details will vary depending on your domain, your users, etc.

So now that we have a way to decide if two events conflict or not we need to modify our commit method to take this into account. This method is defined in the PostsController:

[code lang=scala]
/**
* Commits an event and applies it to the current state. If successful the
* provided `onCommit` callback is invoked and its result returned. Otherwise
* the `onConflict` is callback is invoked and its result returned.
*/
private[this] def commit
(expected: StreamRevision, event: PostEvent)
(onCommit: => Result,
onConflict: (StreamRevision, Seq[PostEvent]) => Result): Result = {
def resolveConflict(committed: Seq[PostEvent], attempted: PostEvent) = {
val conflicting = committed.filter(PostEvent.conflictsWith(_, attempted))
if (conflicting.isEmpty) Right(attempted)
else Left(conflicting)
}

@tailrec def run(expected: StreamRevision, event: PostEvent): Result =
memoryImage.tryCommit(event.postId.toString, expected, event) match {
case Right(commit) =>
onCommit
case Left(conflict) =>
resolveConflict(conflict.conflicting.flatMap(_.events), event) match {
case Right(event) => run(conflict.actual, event)
case Left(conflicting) => onConflict(conflict.actual, conflicting)
}
}

run(expected, event)
}
[/code]

The new commit method implementation first defines a resolveConflict helper method (line 10), which takes a list of already committed, potentially conflicting events and uses the PostEvent.conflictsWith method to see if there are any real conflicts. If there are none, the attempted event is returned. Otherwise the conflicting events are returned.

The run method (line 16) runs in a (tail-recursive) loop. It tries to commit against the expected revision of the event stream. If there is no conflict, it returns the result of the provided onCommit callback. Otherwise it tries to resolve the conflicts. If this succeeds, it tries invokes itself again^[2], but now with the latest known event stream revision as the expected revision. If the conflict cannot be resolved, the result of the onConflict callback is returned instead.

Finally, line 27 simply kicks off the entire process using the provided revision and event.

With this in place actual conflicts should be quite rare. But we can still do better than just showing a generic “there was a conflict” error page. This is the job of the conflictsMessagePanel.scala.html template. This template shows a human readable version of the conflicts that occurred:

Summary

Besides adding support for conflict resolution, the implementation of the blog post comment functionality was quite straightforward. In a blogging application conflicts are probably quite rare, so it may not make sense to build in extensive UI support for this, but having this as an example hopefully gives you some idea of what is possible. In more collaborative applications this kind of functionality is much more interesting and you may prefer to immediately push updates directly to the client, instead of waiting for the client to submit a form. An example of this is Pivotal Tracker or Apache Wave.

If your application has extreme availability requirements, similar conflict resolution can also help you deal with recovering from network partitioning. Or applications that need to be able to run in disconnected mode. In these cases you will need to write an event stream function that merges divergent event streams. Version control systems are examples of this, and can be a source of inspiration, although they usually don’t have intention revealing events to work with.

But the main point is that the level of conflict resolution you need depends on your users and your application. Event sourcing gives you a great tool to make conflicts easier to resolve, without necessarily complicating your application if you do not need this kind of functionality.

In the next part we’ll take a look at another kind of concurrency conflict that can occur in the current application when two or more users committing events to the same event stream nearly simultaneously.

In this part we’ll develop an event store implementation on top of Redis, a key-value store with support for additional data structures (such as lists and hashes), publish/subscribe, and transactions. If you’re more interested in adding new application functionality, you can safely skip this part and go to part 4 – conflict resolution.

Other Parts

• Part 1 – Introduction
• Part 2 – Consistency
• Part 3 – Redis Event Store
• Part 4 – Conflict Resolution
• Part 5 – Refactoring and Transactions
• Part 6 – Users, Authentication, Authorization

Running the application

So we already have an event store specification and a reference implementation. Implementing a Redis-based event store should be pretty straight forward, right? Well, it mostly is, but before we dive into the details let’s take a look at what we’ll be able to do once everything is up and running.

First, make sure you have a working Redis installation (you can use the redis-cli command to connect to Redis and check if everything is working). Redis 2.6 introduced support for server-side Lua scripts which the event store implementation uses automatically if available. Otherwise, the event store falls back to using the WATCH/MULTI/EXEC commands, so Redis 2.4 also works, although with a performance penalty when it comes to committing events.

If you use a Mac OS X with homebrew you can simple run brew install redis to install 2.4.15 or brew install redis --devel to install 2.6.0-RC5.

Now checkout the part-3 branch of the project, adopt conf/application.conf to match your Redis configuration, and start the application using play run or sbt run. Any blog posts you add, edit, or delete will now have the resulting events committed to Redis. If you restart the application the events are replayed on start-up, so you will no longer lose your data.

Start the Scala console (play console or sbt console) and paste the following Scala code (adjust the connection settings as needed).

[code lang=scala]
import events._, eventstore._
val es = redis.RedisEventStore[PostEvent]("blog", "localhost", 6379)
es.publisher.subscribe(StoreRevision.Initial)(println)
[/code]

This code connects to the event store and subscribes the println method. You’ll see that first all historical commits are printed, and as you use the application you’ll notice that new commits are immediately printed as well. This is one of the major differences with more traditional database backed systems: an event sourced system is open for extension and can communicate to the outside world what is going on right now, while a database based system is usually much more of a black box.

Now start a second instance of the application (play 'run 9001' or sbt 'run 9001'). This second instance also connects to the event store to replay and subscribe to the events being committed. The memory images of both instances stay synchronized as you make changes, while the Scala console keeps printing events as they are committed (make sure you close the event store before exiting the console, otherwise the process stays connected and will fail to terminate). The picture below shows the general setup:

So by using the Redis event store we can run multiple instances of the application, each running on a different server. Each commit from any of the application servers is pushed to all connected servers. This provides fault-tolerance and scalability at the application server level, and also makes it possible to upgrade application servers without affecting users (by doing rolling updates).

Event serialization

When data is stored in an external system we need to define some kind of serialization format. There is a wide variety of serialization formats and libraries available. For this example application we’ll use JSON. This may not be the most compact or fastest format, but is easy to use and read, which is great for this example. Play! defines the Format trait for JSON serialization. We just have to provide an implementation of this trait for each data type that we want to serialize.

Although Play! predefines various JSON format implementations for many of the standard Scala classes, it does not include any helpers to make it easy to define formats for our own types. But it is easy to supply our own helpers, which you can find in the JsonMapping object.

The objectFormat methods are used to define the JSON format for case classes such as PostContent:

[code lang=scala]
implicit val PostContentFormat: Format[PostContent] =
objectFormat("author", "title", "body")(PostContent.apply)(PostContent.unapply)
[/code]

Here we simply list the JSON field names and the Scala generated apply and unapply methods to construct and deconstruct PostContent instances. Formats for class hierarchies like PostEvent are a bit more difficult to get right, but easy enough to define with the typeChoiceFormat helper:

[code lang=scala]
implicit val PostEventFormat: Format[PostEvent] = typeChoiceFormat(
"PostAdded" -> objectFormat("postId", "content")(PostAdded.apply)(PostAdded.unapply),
"PostEdited" -> objectFormat("postId", "content")(PostEdited.apply)(PostEdited.unapply),
"PostDeleted" -> objectFormat("postId")(PostDeleted.apply)(PostDeleted.unapply))
[/code]

This may not be quite as convenient as using a reflection based serialization library, but avoids having to deal with various JVM class loader problems and other reflection related troubles.

The formats are defined as implicit values in the companion objects of the classes that we want to serialize so that the Scala compiler can find these formats and automatically pass them as parameters where needed, so that we don’t have to. (Yes, the type “checker” is filling in the blanks for you.)

Writing tests for correct serialization and deserialization is rather tedious, but very important. Fortunately, ScalaCheck makes writing these kind of tests almost trivial:

[code lang=scala]
"Post events" should {
"convert to and from JSON" in forAll(eventsForMultiplePosts.arbitrary) { events =>
Json.fromJson[List[PostEvent]](Json.toJson(events)) must_== events
}

"parse example Post Added event" in {
val event = PostAdded(PostId(UUID.fromString("5ab11526-477b-43b9-8fe6-4bb25a3dfcc6")), PostContent(author = "Author", title = "Title", body = "Body"))
val json = """{"type":"PostAdded","data":{"postId":"5ab11526-477b-43b9-8fe6-4bb25a3dfcc6","content":{"author":"Author","title":"Title","body":"Body"}}}"""

Json.fromJson[PostEvent](Json.parse(json)) must_== event
}
}
[/code]

For the serialization tests I typically test using a few examples, but the main testing happens by defining the property fromJson(toJson(events)) == events. ScalaCheck will randomly generate data to verify that this property holds, so we can be confident serialization works as expected.

The Redis event store implementation

The Redis event store implementation uses the Jedis client library. For reading and committing we use a JedisPool connection pool. To use a connection and return it to the pool we’ve implemented the loan pattern in the withJedis method:

[code lang=scala]
class RedisEventStore[Event]
(name: String, host: String, port: Int, config: Config)
(implicit val eventFormat: Format[Event])
extends EventStore[Event] {

val jedisPool = new JedisPool(config, host, port)

def withJedis[A](f: Jedis => A): A = {
val jedis = jedisPool.getResource
try {
f(jedis)
} finally {
jedisPool.returnResource(jedis: BinaryJedis)
}
}

// [...]
}
[/code]

Reading events

To store the commits we’ll use two Redis data structures:

1. A Redis hash that maps commit ids to serialized commits. The commit id is equal to the commits store revision. We use a hash instead of a list as Redis lists are O(n) when accessing an arbitrary element, while hashes provide O(1) random element access.
2. One Redis list for each event stream. Each element in the list is a commit id.

This means that if we want to read all past commits we simply need to read each hash value starting with commit id 1 up to the commit id equal to the size of the hash. The code for this can be found in the reader implementation of RedisEventStore.scala:

[code lang=scala]
override object reader extends CommitReader[Event] {
override def storeRevision = withJedis { jedis =>
StoreRevision(jedis.hlen(CommitsKey))
}

override def readCommits(since: StoreRevision, to: StoreRevision) = {
val current = storeRevision
if (since >= current) Stream.empty else {
val revisionRange = (since.value + 1) to (to.value min current.value)
doReadCommits(revisionRange.map(_.toString))
}
}

// [...]
}
[/code]

The current store revision is equal to the size of the commits hash (HLEN). The commits are all read in order using the doReadCommits method:

[code lang=scala]
val KeyPrefix = name + ":"
val CommitsKey: String = KeyPrefix + "commits"

def doReadCommits(commitIds: Seq[String]): Stream[Commit[Event]] = {
val chunks = commitIds.grouped(ChunkSize).map(_.toArray)
chunks.flatMap { chunk =>
val serializedCommits = withJedis { _.hmget(CommitsKey, chunk: _*) }
serializedCommits.asScala.par.map(deserializeCommit)
}.toStream
}

def deserializeCommit(serialized: String) =
Json.fromJson[Commit[Event]](Json.parse(serialized))
[/code]

The list of commit ids to read is first split into chunks, each at most ChunkSize (10,000) elements in size. Each chunk is then read using a single get multiple values from hash (HMGET) command. By reading many commits at a time we greatly reduce the number of process context switches. The commits are then deserialized using the deserializeCommit method, which is just a simple wrapper around the Play! JSON deserializer. Since deserialization is a CPU intensive process we use Scala’s parallel collections (par) to get a nice speed-up on a multi-core machine.

The resulting commits are then transformed into a lazy Stream. This way we do not have to read all the commits into memory at once, but we’ll read them on-demand, one chunk at a time.

Committing

There are two implementations of the EventCommitter interface. The RedisWatchMultiExecEventCommitter uses the Redis WATCH/MULTI/EXEC commands and is compatible with Redis 2.4 and higher. But it’s slower and more complicated than the RedisLuaEventCommitter that we’ll describe below.

The RedisLuaEventCommitter uses a Lua script to atomically commit events in Redis. The script is fairly similar to the commit implementation of the FakeEventStore:

[code lang=scala]
trait RedisLuaEventCommitter[Event] { this: RedisEventStore[Event] =>
val TryCommitScript: String = """
| local commitsKey = KEYS[1]
| local streamKey = KEYS[2]
| local timestamp = tonumber(ARGV[1])
| local streamId = ARGV[2]
| local expected = tonumber(ARGV[3])
| local events = ARGV[4]
|
| local actual = tonumber(redis.call('llen', streamKey))
| if actual ~= expected then
| return {'conflict', tostring(actual)}
| end
|
| local storeRevision = tonumber(redis.call('hlen', commitsKey))
| local commitId = storeRevision + 1
| local commitData = string.format('{"storeRevision":%d,"timestamp":%d,"streamId":%s,"streamRevision":%d,"events":%s}',
| commitId, timestamp, cjson.encode(streamId), actual + 1, events)
|
| redis.call('hset', commitsKey, commitId, commitData)
| redis.call('rpush', streamKey, commitId)
| redis.call('publish', commitsKey, commitData)
|
| return {'commit', tostring(commitId)}
""".stripMargin

// [...]
}
[/code]

The RedisLuaEventCommitter must be mixed into the RedisEventStore class, so we specify this using a self type (line 1). This gives us access to all RedisEventStore methods.

The first section of the Lua script reads the command arguments (line 3-8). The second section checks if the expected stream revision matches the actual stream revision. If not, a conflict is returned (line 10-13).

Otherwise the current store revision is determined. The commit id is set to the next store revision and the serialized commit JSON is generated (line 15-18). Then three commands are executed to:

1. Store the serialized commit in the commits hash (line 20).
2. The commitId is appended to the commits of the affected stream (line 21).
3. The serialized commit is published to all subscribers (line 22). The name of the publication channel is the same as the key for the commits hash, but this is a rather arbitrary implementation detail.

When starting the event store the above Lua script is uploaded to Redis (using SCRIPT LOAD) and the resulting script identifier (the SHA-1 hash of the script contents) is saved:

[code lang=scala]
val TryCommitScriptId = withJedis { _.scriptLoad(TryCommitScript) }
[/code]

The implementation of the tryCommit is now quite straightforward. We simple need to invoke the Lua script using the EVALSHA command and translate the results to Scala:

[code lang=scala]
object committer extends EventCommitter[Event] {
override def tryCommit(streamId: String, expected: StreamRevision, event: Event): CommitResult[Event] = {
// Prepare parameters.
val timestamp = DateTimeUtils.currentTimeMillis
val serializedEvents = Json.stringify(Json.toJson(Seq(event))(Writes.seqWrites(eventFormat)))

// Invoke Lua script.
val response = withJedis { _.evalsha(TryCommitScriptId, 2,
/* KEYS */ CommitsKey, keyForStream(streamId),
/* ARGV */ timestamp.toString, streamId, expected.value.toString, serializedEvents)
}

// Parse response.
try {
response.asInstanceOf[java.util.List[_]].asScala match {
case Seq("conflict", actual: String) =>
val conflicting = reader.readStream(streamId, since = expected)
Left(Conflict(streamId, StreamRevision(actual.toLong), expected, conflicting))
case Seq("commit", storeRevision: String) =>
Right(Commit(StoreRevision(storeRevision.toLong), timestamp, streamId, expected.next, Seq(event)))
}
} catch {
case e: Exception =>
throw new EventStoreException("Error parsing response from Redis TryCommit script: " + response, e)
}
}
}
[/code]

Subscriptions

Redis has built-in support for publish/subscribe, which we will use to notify each subscriber when new events are committed. But before we actually subscribe to Redis, we first need to replay historical commits. The reason to not subscribe before replaying is that replaying might take some time (if you have many historical commits) and we do not want any new commits to queue-up while we’re still processing the older commits:

[code lang=scala]
override object publisher extends CommitPublisher[Event] {
import reader._

override def subscribe(since: StoreRevision)(listener: CommitListener[Event]): Subscription = {
@volatile var cancelled = false
@volatile var last = since
val unsubscribeToken = UUID.randomUUID.toString

executor.execute(new Runnable {
private def replayCommitsTo(to: StoreRevision) {
if (last < to) {
Logger.info("Replaying commits since " + last + " to " + to)
readCommits(last, to).takeWhile(_ => !closed && !cancelled).foreach(listener)
last = to
}
}
// [...]
}
}
}
[/code]

The last variable contains the store revision of the last commit we passed to the listener so far. It is initialized to the since parameter. We then start a new thread to perform the actual commit replay and Redis subscription. This is consistent with our fake event store implementation.

The cancelled flag is used to help with unsubscribing when the subscription is cancelled. In addition to this, the unsubscribeToken is sent to the subscriber as well so that the subscriber wakes up when required.

After replaying (line 7 below) the historical commits we perform the actual subscription using a new Redis connection (line 12):

[code lang=scala]
override def run {
val currentRevision = storeRevision
if (last > currentRevision) {
Logger.warn("Last " + last + " is in the future, resetting it to current " + currentRevision)
last = currentRevision
} else {
replayCommitsTo(currentRevision)
}

val jedis = new Jedis(host, port)
try {
jedis.subscribe(Subscriber, ControlChannel, CommitsKey)
} finally {
jedis.disconnect
}
}
[/code]

The Subscriber object implements the JedisPubSub interface:

[code lang=scala]
object Subscriber extends JedisPubSub {
[/code]

When the subscription is confirmed we check if we missed any commits that happened while we were performing the initial replay, but before completing the subscription:

[code lang=scala]
override def onSubscribe(channel: String, subscribedChannels: Int) = channel match {
case ControlChannel =>
// We may have missed the cancellation token while subscribing, so check the flag.
if (closed || cancelled) unsubscribe
case CommitsKey =>
// We may have missed some commits while subscribing, so replay missing if needed.
replayCommitsTo(storeRevision)
case _ =>
Logger.warn("message received on unknown channel '" + channel + "'")
}
[/code]

Once we’re subscribed we’ll receive a notification for each commit:

[code lang=scala]
override def onMessage(channel: String, message: String) = channel match {
case ControlChannel =>
if (message == CloseToken || message == unsubscribeToken) {
unsubscribe
}
case CommitsKey =>
val commit = deserializeCommit(message)
if (last.next < commit.storeRevision) {
Logger.warn("missing commits since " + last + " to " + commit.storeRevision + ", replaying...")
replayCommitsTo(commit.storeRevision)
} else if (last.next == commit.storeRevision) {
listener(commit)
last = commit.storeRevision
} else {
Logger.warn("Ignoring old commit " + commit.storeRevision + ", since we already processed everything up to " + last)
}
case _ =>
Logger.warn("message received on unknown channel '" + channel + "'")
}
[/code]

We deserialize and forward any new commit to the listener. In some case we may receive old commits, which we ignore. In case we miss any commits, we replay the missing ones. The control channel is monitored for event store closing or subscription cancellation messages.

This completes the implementation of the Redis event store. For full details, check the eventstore.redis package.

Setup and configuration

Now that we have multiple event store implementations we need to provide some configuration options. This is done in the controllers.Global object in the onStart method. The actual configuration can be found in the conf/application.conf file.

Since our event store exposes a blocking API (for ease of implementation) we also increase the Play! thread-pool sizes^[1].

Redis durability

The primary concern of an event store is to ensure events are written to durable storage, so no events get lost. To configure Redis for maximum reliability of written data we need to edit the redis.conf file to:

• Turn off saving snapshots by commenting out all the save settings.
• Turn on appendonly mode (appendonly yes).
• Sync to disk on every transaction by setting appendfsync always.
• (Optional) disable automatic rewriting of the append-only file (auto-aof-rewrite-percentage 0). We’ll only ever be adding new data to Redis, so rewriting will not result in a smaller append-only file anyway.

Performance

To test and compare the performance of the various event store implementations the following setup was used:

• The application and Redis run on deler (dutch for denominator), a 2010 dual-core 2.66GHz Core i7 with hyper-threading and a 7200-RPM HDD. The server was started using the start-server.sh script.
• The performance test client runs on noemer (dutch for numerator), a 2012 quad-core 2.6GHz Core i7 with hyper-threading. The benchmark was started using the run-benchmark.sh script with the following arguments: http://deler.local:9000 50 500000.

The server and client machine were directly connected with a 1 Gigabit ethernet cable and were otherwise idle. This setup ensures that neither the client nor the network was a bottleneck so that we can fully test the performance of the server code. Both computers were running the Java SE 7u5 JDK.

The benchmark itself consists of a warmup phase (running 20,000 iterations of adding/listing/editing/reading/deleting blog posts) to ensure the server JVM has ample opportunity to optimize the generated code, and then 50 concurrent clients completed a total of 500,000 iterations (100,000 iterations for the Redis WATCH/MULTI/EXEC event store implementation) of benchmarking. Note that this is a full-stack benchmark, including the network, HTTP protocol, request routing, the event store, and template rendering.

The results are summarized below (first number is total iterations per second, second number is the 99th percentile latency in milliseconds):

As you can see there is only a slight difference between having no event store at all or the fake event store. The switch to the Lua-based Redis event store causes a much bigger change. Committing events now requires us to wait for a disk write, and also adds the overhead of process switching and commit serialization and deserialization. Still, the performance is not bad at all. When falling back to the WATCH/MULTI/EXEC instructions commit performance drops again. With the Lua implementation Redis was still able to write multiple commits to disk using a single fsync^[2], without Lua each commit needs to wait for a full fsync to complete due to the use of optimistic locking.

When a more complex page is rendered (listing the last 20 blog posts) the performance differences become negligible. It seems the bottleneck here is the speed at which we can render the page. I believe the Play! 2.0 template engine implementation could use a bit more optimization here.

After completing a benchmark run there are 1,060,000 events in the event store, almost 1 Gigabyte of data. How much time does it take to replay these events on startup? On deler it takes about 26 seconds to fully restore the memory image from the committed events (about 40,500 commits per second). Noemer is more than twice as fast, since it has more and faster processors to perform event deserialization, the primary bottleneck.

Summary

Now that we have a working event store implementation our application could go into production, except for the obvious lack of features. There are obviously some enhancements that could be made to the event store:

• Automatic recovery from Redis connection failures.
• Adding arbitrary headers to a commit. These can be used to store metadata like client IP, application server node, authenticated user, etc.
• Performance metrics, such as latency from commit to notification. Raise alerts when latency gets to high.
• Etc.

But for now this event store implementation is good enough.

If you feel that we needed to do a lot of work to get to this place, you’re partly right. But we did gain complete control and understanding of the basic infrastructure of our application! Compare this with having to understand SQL, database schemas, JDBC, your favorite Object-Relational Mapper, transaction managers, etc.

Of course, it still remains to be seen how easy or hard it is to add new application functionality. So in the next parts we’ll focus on adding new features, and some of the problems (with solutions) that you will quickly encounter when using event sourcing.

Other Parts

• Part 1 – Introduction
• Part 2 – Consistency
• Part 3 – Redis Event Store
• Part 4 – Conflict Resolution
• Part 5 – Refactoring and Transactions
• Part 6 – Users, Authentication, Authorization

The code

You can find the code at github in the part-2 branch. Use git clone -b part-2 https://github.com/zilverline/event-sourced-blog-example.git to checkout the correct branch. If you already retrieved the previous part, then using git pull and git checkout part-2 should do the trick.

Living in a distributed world

Something you’ll quickly have to learn when you develop web applications is that the internet is a truly distributed environment. You’ll often run your application across multiple servers (for performance and/or fault-tolerance), while users access your application from different devices all across the globe. All of these servers and devices will have a different view of the world. In a distributed world there is no such thing as a single, global truth. As soon as a web page is rendered by a server and the page is displayed by the client, the data on the page is already out of date. If you do not handle this correctly, some surprising things may happen.

An easy way to see this is to start editing a blog post, then use another browser window to delete the same blog post, and then submit the original edit form. In the application from part 1 this results in a NoSuchElementException while processing the event to update the current state. In the Rails getting started application you get a ActiveRecord::RecordNotFound. Other frameworks that I know of aren’t any better at handling this. Back button usage and double-clicking also causes many of the same problems. And the more collaberative your application is, the more likely it is that these inconsistencies and conflicts will occur for real.

Many current web applications deal with this by relying on a combination of ad-hoc solutions, prayer, and luck^[1]. You just have to hope that the occasional lost update, exception or other glitch won’t cause too much trouble.

Write consistency

Since events are forever, we cannot afford to let these glitches go undetected. Otherwise we could get strange, meaningless event sequences, such as having a blog post being edited after it has been deleted! So one major responsibility of an event store is to detect these conflicts and prevent storing inconsistent events.

To implement this our event store is going to keep track of multiple event streams. Each event stream has an associated stream revision, which is defined to be equal to the number of commits for that particular event stream. This makes the stream revision a gapless, strictly increasing sequence.

When you commit additional events to an event stream, you will have to specify the stream revision that you expect the stream to currently have. If the expected stream revision is lower than the current stream revision, there is a conflict. This approach is also known as optimistic concurrency control.

We use multiple event streams so that commits that go to different event streams do not conflict: each event stream defines a consistency boundary. In our example application we’ll map each blog post to its own event stream. So editing or deleting two different posts never cause conflicts.

Read consistency

There is another problem that we have to solve, related to using a memory image. In a database oriented application the central database is considered to give a single, up-to-date, consistent view of the current state of the application. Database transactions are used to control concurrency conflicts (but make sure you choose the correct isolation level!). Since the database is a single, centralized component any read after a transaction commit will return the correct data (at least until you introduce master-slave replication or caching).

With a memory image the situation is slightly different. After changes are successfully committed to the event store, the memory images of every application server is not immediately updated! The newly committed events must first be transferred to the application servers and applied to the memory image before the effects of the change becomes visible:

So to avoid having a fast client not seeing the results of the action they just performed, we’ll also keep track of the event store revision, which is defined to be equal to the total number of commits to the event store. Again, this produces a gapless, strictly increasing sequence. Whenever a client is redirected to see the results of a commit, we’ll use the store revision to check if our memory image is up-to-date.

There are a couple of ways to implement this:

• Don’t track the store revision at all. The application servers will usually be notified of the change before the client completes the HTTP redirect, so the client will usually see the changes it just made. This may be good enough in situations where highly volatile data is the norm, such as financial markets. For a blog posting application, users expect their changes to be immediately visible, so we need to look at other solutions.
• Only run a single application server or ensure all clients always access the same application server (for example, by using sticky sessions, or having one primary server and another server just for failover purposes). The client should always see the result of its actions if we update the server’s memory image before responding.
• Save the store revision as part of the user’s session state, such as a cookie. Before accessing the memory image ensure it is up-to-date with respect to the revision saved in the cookie or session. Unfortunately, using mutating cookies can lead to race conditions.
• Pass the most recent store revision as a parameter in the redirect URL. This means that every POST action needs to take care of correctly passing in the new store revision whenever a commit succeeds.
• Read the current event store revision whenever the memory image is accessed. Wait until at least this many commits have been applied to the memory image before rendering the response. This is the easiest to implement, so we’ll use this. It is however not the most performant!

The Event Store API

If the above got you thoroughly confused, don’t worry too much. The event store implementation is actually not that complicated. To make the first implementation easier to understand we will not worry about durability yet. In the next part of this series we’ll build an implementation that uses an actual database (Redis). We’ll then compare the fake implementation to the Redis implementation to check that both behave the same.

Store and stream revisions

The store and stream revisions are represented by the StoreRevision and StreamRevision classes and can be found in EventStore.scala. They simply wrap a Long value. The main reason not to use plain Longs is to avoid accidental mixups. Plain numbers that are related but actually mean different things are hard to keep straight otherwise!

Conflicts and commits

Committing events to the event store uses the EventCommitter interface:

[code lang=scala]
/**
* Commits events to an event store.
*/
trait EventCommitter[Event] {
def tryCommit(streamId: String, expected: StreamRevision, event: Event): CommitResult[Event]
}
[/code]

The tryCommit method will attempt to commit the given event to the event stream. The result type is defined as follows:

[code lang=scala]
/**
* The result of a commit attempt is either a `Conflict` or a successful `Commit`.
*/
type CommitResult[+Event] = Either[Conflict[Event], Commit[Event]]

/**
* A successful commit to `streamId`.
*/
case class Commit[+Event](
storeRevision: StoreRevision,
timestamp: Long,
streamId: String,
streamRevision: StreamRevision,
events: Seq[Event])

/**
* The conflict that occurred while trying to commit to `streamId`.
*/
case class Conflict[+Event](
streamId: String,
actual: StreamRevision,
expected: StreamRevision,
conflicting: Seq[Commit[Event]])
[/code]

In other words, tryCommit returns either a Conflict or a Commit. The conflict will contain the actual stream revision and all conflicting commits (any commit that happened since the expected stream revision). When a commit succeeds, it will contain the committed event as well as some metadata, such as the timestamp of the commit.

You may have noticed that the Commit class allows for multiple events in a single commit. This can often be useful if you want a group of events to be treated atomically by subscribers. Another reason for this is that as your application evolves some events may no longer be of interest. These events can then be filtered out without altering the store or stream revisions the commit is part of. In other words, you may have commits that no longer hold any events!

Event notification

To rebuild the memory image and receive commits as they happen the CommitPublisher interface is defined:

[code lang=scala]
/**
* Publishes successful commits to subscribers.
*/
trait CommitPublisher[Event] {
/**
* Notifies `listener` of all commits that happened `since`. Notification happens asynchronously.
*/
def subscribe(since: StoreRevision)(listener: Commit[Event] => Unit): Subscription
}

/**
* A subscription that can be cancelled.
*/
trait Subscription {
def cancel(): Unit
}
[/code]

After subscribing to the event store all commits that happened after the since store revision will be passed to the listener callback. The event store will ensure that commits are send to the listener in-order and without any gaps or duplicates. Notice that notification of commits happens asynchronously.

Reading events and the event store API

Two more interfaces are defined that complete the event store API:

[code lang=scala]
/**
* Reads commits from the event store.
*/
trait CommitReader[Event] {
def storeRevision: StoreRevision
def readCommits(since: StoreRevision, to: StoreRevision): Stream[Commit[Event]]
def streamRevision(streamId: String): StreamRevision
def readStream(streamId: String, since: StreamRevision, to: StreamRevision): Stream[Commit[Event]]
}

/**
* The event store API.
*/
trait EventStore[Event] {
def reader: CommitReader[Event]
def committer: EventCommitter[Event]
def publisher: CommitPublisher[Event]
def close(): Unit
}
[/code]

The CommitReader interface allows access to stored commits (using Scala’s lazy Streams to avoid having to load all commits in memory at once) and the EventStore interface simply combines the other interfaces and a close method into a single unit.

The fake event store implementation

The FakeEventStore implementation can be found in FakeEventStore.scala. It uses a simple Vector (Scala’s immutable equivalent to Java’s ArrayList) to store all events committed to the store and a map of streamId to Vector for the commits associated with each event stream. All commits are stored in their original order. Scala’s Software Transactional Memory (STM) is again used to ensure consistency:

[code lang=scala]
class FakeEventStore[Event] extends EventStore[Event] {
// [...]
private[this] val commits = Ref(Vector.empty[Commit[Event]]).single
private[this] val streams = Ref(Map.empty[String, Vector[Commit[Event]]]).single

override object reader extends CommitReader[Event] {
override def storeRevision = StoreRevision(commits().size)

override def readCommits(since: StoreRevision, to: StoreRevision): Stream[Commit[Event]] = {
commits().slice(
(since.value min Int.MaxValue).toInt,
(to.value min Int.MaxValue).toInt).toStream
}
// [...]
}
// [...]
}
[/code]

The CommitReader implementation just uses regular Scala collection manipulation functions. Notice that the implementation of storeRevision is precisely the definition of a store revision as discussed in "read consistency" above! The streamRevision and readStream methods (not shown) are implemented similarly.

The EventCommitter tryCommit method is also straightforward:

[code lang=scala]
override def tryCommit(streamId: String, expected: StreamRevision, event: Event): CommitResult[Event] = {
require(Txn.findCurrent.isEmpty, "the fake event store cannot participate in an STM transaction, just like a real event store")

atomic { implicit txn =>
val actual = streamRevision(streamId)

if (expected < actual) {
val conflicting = readStream(streamId, since = expected)
Left(Conflict(streamId, actual, expected, conflicting))
} else if (expected > actual) {
throw new IllegalArgumentException("expected revision %d greater than actual revision %d" format (expected.value, actual.value))
} else {
val commit = Commit(storeRevision.next, DateTimeUtils.currentTimeMillis, streamId, actual.next, Seq(event))
commits.transform(_ :+ commit)
streams.transform(streams => streams.updated(streamId, streams.getOrElse(streamId, Vector.empty) :+ commit))
Right(commit)
}
}
}
[/code]

First a check is done to ensure that no STM transaction is currently active, to make sure that we correctly mimic a real event store. Then the expected stream revision is checked against the actual revision. If there is a mismatch a conflict or error is reported. If they match, a new Commit is instantiated and appended to the commits and streams collections. The new commit is then returned. All this is done inside a STM transaction to ensure atomicity.

The final part of the event store implementation deals with notification and is the most tricky. STM again helps to keep complexity under control:

[code lang=scala]
private[this] val closed = Ref(false).single
private[this] val executor = Executors.newCachedThreadPool

override object publisher extends CommitPublisher[Event] {
override def subscribe(since: StoreRevision)(listener: Commit[Event] => Unit): Subscription = {
val cancelled = Ref(false).single
val last = Ref(since).single

executor.execute(new Runnable {
@tailrec override def run {
// Wait for new commits or subscription termination.
val pending = atomic { implicit txn =>
if (closed() || cancelled()) None else {
val pending = commits().drop(last().value.toInt)
if (pending.isEmpty) retry
else Some(pending)
}
}
pending match {
case None => // Stop.
case Some(commits) =>
// Notify listener and go back to run.
commits.foreach { commit =>
listener(commit)
last() = commit.storeRevision
}
run
}
}
})

// Return a subscription instance that can be used for cancellation.
new Subscription {
override def cancel() = cancelled.set(true)
override def toString = "Subscription(" + last() + ", " + cancelled() + ", " + FakeEventStore.this + ")"
}
}
}
[/code]

The closed reference is used to communicate that the event store has been closed. The executor thread pool is used to run each subscription on its own background thread^[2] so that we can correctly mimic a real event store.

When a subscription is made, we create two more references: cancelled to communicate that this subscription has been cancelled and last to keep track of the last commit that has been passed to the listener.

A notification thread is then started that continuously checks if there are any new commits that the listener needs to be notified of. If there are none, the STM transaction is retried. Retry internally uses blocking to avoid needless looping to check for new event store commits.

If the event store is closed or the subscription is cancelled the subscription thread terminates.

The tests for the event store can be found in EventStoreSpec.scala. The tests are implemented as a trait and the FakeEventStoreSpec simply extends this trait. This allows us to use the same tests for all event store implementations.

Memory Image

The memory image itself is now implemented in its own class. It basically wraps an EventStore while allowing access to the current state.

[code lang=scala]
/**
* A `MemoryImage` tracks an underlying event store and uses the provided
* `initialState` and `update` to project the committed events onto the
* current state.
*/
class MemoryImage[State, Event] private
(eventStore: EventStore[Event])
(initialState: State)
(update: (State, Commit[Event]) => State)
extends EventCommitter[Event] {
private[this] val state = Ref(initialState)
private[this] val revision = Ref(StoreRevision.Initial)

/**
* The current state of the memory image with at least all commits applied
* that have been committed to the underlying event store.
*/
def get: State = {
val minimum = eventStore.reader.storeRevision
atomic { implicit txn =>
if (revision() < minimum) retry
else state()
}
}

/**
* Commits an event to the underlying event store. The memory image will be
* updated if the commit succeeds.
*/
override def tryCommit(streamId: String, expected: StreamRevision, event: Event): CommitResult[Event] =
eventStore.committer.tryCommit(streamId, expected, event)

override def toString = "MemoryImage(%s, %s)".format(revision.single.get, eventStore)

// Subscribe to the underlying event store and apply every commit to the
// current state using the provided `update` function.
eventStore.publisher.subscribe(StoreRevision.Initial) { commit =>
atomic { implicit txn =>
require(revision().next == commit.storeRevision, "expected: " + revision().next + ", got " + commit.storeRevision)

state.transform(s => update(s, commit))
revision() = commit.storeRevision
}
}
}
[/code]

The memory image takes three constructor parameters: the event store, the initial state, and the update function that takes the current state and a commit to produce the updated state. The memory image also implements the EventCommitter interface and simply passes any commit attempts to the underlying event store.

The get method first checks if the current memory image is up-to-date. If it isn’t, it blocks until the required number of commits have been applied. Otherwise it simply returns the current state. This ensures the required level of read consistency.

When the memory image is instantiated it also subscribes to the underlying event store and uses the provided update function to ensure the current state reflects all commits received from the event store.

The Posts Controller

Now that we have a (fake) implementation of the event store, we can start adjusting the PostsController to use it to store events. Before we simply had a posts reference to the latest data and a commit method to commit events. We’ll re-implement these to use the memory image:

[code lang=scala]
class PostsController(memoryImage: MemoryImage[Posts, PostEvent]) extends Controller {
/**
* The current blog posts from the memory image.
*/
def posts(): Posts = memoryImage.get

/**
* Commits an event and applies it to the current state. If successful the
* provided callback `f` is applied to the `commit`. Otherwise a conflict
* result is returned.
*/
private[this] def commit(expected: StreamRevision, event: PostEvent)
(f: Commit[PostEvent] => Result): Result = {
memoryImage.tryCommit(event.postId.toString, expected, event) match {
case Left(conflict) => Conflict(todo())
case Right(commit) => f(commit)
}
}
[/code]

As you can see the commit method parameter list was expanded to include the expected stream revision and a callback used to render an HTTP response if the commit succeeds. However, if a conflict is detected an HTTP 409 (Conflict) status code is returned. Giving the client detailed information on the conflict is not implemented yet, so we simply render a todo page.

The controller actions that generate events are all related to HTTP POST requests and need to be adjusted to handle the new expected stream revision parameter and conflict response. Since only the client knows the revision used to render the page by the time the POST request is submitted, the expected stream revision is added as a new URL parameter. Listed below are the show and submit actions related to the "edit post" page:

[code lang=scala]
def show(id: PostId) = Action { implicit request =>
posts().get(id) match {
case Some(post) => Ok(views.html.posts.edit(post.id, post.revision, postContentForm.fill(post.content)))
case None => NotFound(notFound(request, None))
}
}

def submit(id: PostId, expected: StreamRevision) = Action { implicit request =>
postContentForm.bindFromRequest.fold(
formWithErrors => BadRequest(views.html.posts.edit(id, expected, formWithErrors)),
postContent =>
commit(expected, PostEdited(id, postContent)) { commit =>
Redirect(routes.PostsController.show(id)).flashing("info" -> "Post saved.")
})
}
[/code]

Compared to the previous version there are two main changes:

1. The revision of the post is passed to the "edit post" view.
2. The expected revision is added as a parameter to the form submit action, so will need to be passed in as part of the URL. This is configured in the conf/routes file.

Since Play! 2 view templates and URLs are strongly typed it is also impossible to forget to pass this additional expected revision parameter, something that can happen easily with many other solutions, such as hidden "version" form fields.

The blog posts models

Since we need to know the current revision of a blog post when rendering the edit or delete actions we need to keep track of this in our model classes. This is done by adding a new field to the Post class and setting this based on the stream revision associated with the event:

[code lang=scala]
def update(event: PostEvent, revision: StreamRevision): Posts = event match {
case PostAdded(id, content) =>
this.copy(byId = byId.updated(id, Post(id, revision, content)), orderedByTimeAdded = orderedByTimeAdded :+ id)
case PostEdited(id, content) =>
this.copy(byId = byId.updated(id, byId(id).copy(revision = revision, content = content)))
case PostDeleted(id) =>
this.copy(byId = byId - id, orderedByTimeAdded = orderedByTimeAdded.filterNot(_ == id))
}
[/code]

This concludes our tour of the code. There were only minor changes to the code related to the application functionality, as most of the code is related to introducing the event store as a piece of application infrastructure.

Performance

Introducing the fake event store causes about a 3%-10% drop in performance compared to having no event store at all. Writes were affected the most, while read performance is almost the same. This is to be expected, since writes now use a background thread to update the memory image, while reads only add a quick check to the memory image to see if it is up-to-date with respect to the event store. Of course, we’re still not persisting the events to durable storage so we are mostly CPU bound. Writing to disk will obviously introduce new bottlenecks and we’ll go into a more detailed performance analysis in the next part.

Summary

With the introduction of the fake event store implementation our application’s architecture is now in place. The fake event store has the same behavior as a real event store (except for durability) and is very useful in combination with automated testing. The tests for the fake event store will also be used to check that the real event store behaves correctly. No architectural changes will be needed to our application when this new event store is available.

In the next part we’ll use Redis to implement an event store that will actually write the events to durable storage and will be used to replay the events to restore the memory image when the application server is started.

Other Parts

• Part 1 – Introduction
• Part 2 – Consistency
• Part 3 – Redis Event Store
• Part 4 – Conflict Resolution
• Part 5 – Refactoring and Transactions
• Part 6 – Users, Authentication, Authorization

Memory Image and Event Sourcing

The example application is build using a Memory Image. This means that the current state of the application is not stored in a database, but kept in main memory instead. This immediately raises the question of how our state is going to survive application restarts or crashes. Since we cannot save the entire state of our application on every change, we’ll need to keep something like a durable transaction log, just like databases do. To implement this every change our application makes is first represented as a domain event. These domain events are stored as our application’s transaction log, using an event store.

After a restart, we can replay the saved domain events to rebuild the in-memory data structure. This concept is known as event sourcing. Keeping this durable record of domain events also means we will continue to have access to all historical information, which is very useful for auditing, troubleshooting, or data mining purposes.

Keeping our current state in memory has some obvious advantages:

• No need to perform mapping between the in-memory model and a database model. This allows you to quickly evolve your application without worrying about database schema migrations.
• No need to access the database to answer a query or render a view. The database should no longer be the first scalability bottleneck you hit.

There are some disadvantages too:

• Your data must fit in memory. Fortunately, you can start out with a pure in-memory implementation and later refactor your biggest data into a database. Being able to safely delay major architectural decisions until you really need to decide is core to agile architecture and development.
• You must be able to safely and efficiently manage this memory. Garbage collection pause times can become problematic. With the OpenJDK Java VM you can probably get up to 10 Gigabyte or so, and Azul’s Zing JVM will go all the way to 512 Gigabyte. The latter should be enough to handle tens of millions of blog posts.

Running the application

The code for this part can be found on https://github.com/zilverline/event-sourced-blog-example. You can use git clone -b part-1 https://github.com/zilverline/event-sourced-blog-example command to clone and checkout the code that matches the contents of this part.

To run the example you need to install either Play! 2.0.2 (or later) or sbt 0.11.3 or later. If you have a Mac and use homebrew you can simply install these using brew install play or brew install sbt.

After installing execute play run or sbt run to start the application in development mode. After downloading all required dependencies and compilation, the application should be available on http://localhost:9000/.

Click around a bit to see how everything works. The functionality is pretty minimal, so this shouldn’t take long.

The domain events

The domain events (and supporting classes) for blog postings are defined in PostEvents.scala. Since the application is (still) extremely simple, the events basically model adding a new blog posts, and editing or deleting an existing one. These basically mimic the typical create/update/delete actions, but are named in terms our users understand. This way we also capture the intent of the user, not just the effects of the action they just performed. Later we’ll add more events, such as Post Published and Comment Added.

[code lang=scala]
sealed trait PostEvent {
def postId: PostId
}
case class PostAdded(postId: PostId, content: PostContent) extends PostEvent
case class PostEdited(postId: PostId, content: PostContent) extends PostEvent
case class PostDeleted(postId: PostId) extends PostEvent
[/code]

Notice that events are named in the past tense. This can be a bit confusing when writing code that actually generates the events, but at all other times events have always happened in the past so this naming convention makes sense.

The events are implemented using Scala case classes. The Scala compiler will translate these into ordinary JVM classes but will add the fields specified in the constructor, accessors, field based equals/hashCode implementations, and a factory method so that you do not need to use the new keyword when instantiating an object. Case classes can also be used with Scala’s match expression.

In Scala the type is written after the name of a variable or parameter. Explicit types are usually only required on parameters, as the Scala compiler can usually figure out the type in other cases by itself.

The events all extend the PostEvent trait (which is translated into a JVM interface). The PostEvent trait is marked as sealed so that it can only be extended in the same source file. This allows the Scala compiler to do compile-time analysis of match statements to ensure all possible cases are covered. This is very useful when new events are added later!

The events also use two support classes to represent the blog post’s identifier and content, respectively:

[code lang=scala]
case class PostContent(author: String, title: String, content: String)

case class PostId(uuid: UUID)
object PostId {
def generate(): PostId = PostId(UUID.randomUUID())

def fromString(s: String): Option[PostId] = s match {
case PostIdRegex(uuid) => catching(classOf[RuntimeException]) opt { PostId(UUID.fromString(uuid)) }
case _ => None
}

private val PostIdRegex = """PostId\(([a-fA-F0-9-]{36})\)""".r
}
[/code]

The PostContent class is a simple case class with three fields, while the PostId class has a companion object with a generate factory method and fromString parse method.

Since events make up the durable record of everything that happened in our domain, they need to be stable. Stability can be achieved by getting the design right (hard!) and ensuring the events related definitions have very few dependencies on other code. See Stability (PDF) by Robert C. Martin for a great explanation of managing stability and dependencies within a program.

Keeping track of the current state

Although events are very useful to track transactional and historical information, they cannot easily be used to determine the current state. So in addition to capturing the events, we also need to derive the current state from these events. We do not necessarily need to track every piece of data, only what we need for our application to make decisions (by creating new events) or to show information to the user (by rendering views, or sending out emails, etc). The state needed for the blogging application is implemented in Post.scala:

[code lang=scala]
/**
* A specific blog post with its current content.
*/
case class Post(id: PostId, content: PostContent)

/**
* The current state of blog posts, derived from all committed PostEvents.
*/
case class Posts(byId: Map[PostId, Post] = Map.empty, orderedByTimeAdded: Seq[PostId] = Vector.empty) {
def get(id: PostId): Option[Post] = byId.get(id)
def mostRecent(n: Int): Seq[Post] = orderedByTimeAdded.takeRight(n).reverse.map(byId)

def apply(event: PostEvent): Posts = event match {
case PostAdded(id, content) =>
this.copy(byId = byId.updated(id, Post(id, content)), orderedByTimeAdded = orderedByTimeAdded :+ id)
case PostEdited(id, content) =>
this.copy(byId = byId.updated(id, byId(id).copy(content = content)))
case PostDeleted(id) =>
this.copy(byId = byId - id, orderedByTimeAdded = orderedByTimeAdded.filterNot(_ == id))
}
}
object Posts {
def fromHistory(events: PostEvent*): Posts = events.foldLeft(Posts())(_ apply _)
}
[/code]

Since the model classes are in-memory only, we’re free to use regular Scala data structure classes to implement whatever query capabilities we need. There is no need to worry about Object-Relational Mapping, etc. Here we use a map to track each blog post by its identifier and a simple Seq (ordered collection or list) to keep track of the order that posts were added. Notice that this entire model is represented using immutable values, which allows us to safely share the current state between multiple concurrent requests.

The Posts.get method simply looks up a post by its identifier. The Posts.mostRecent method takes the last n added post identifers, reverses the result (so the most recently added post is first), and translates each identifier into a post using the byId map.

The Posts.apply method implements updating the current state based on an event. It basically matches on the type of event and updates its state accordingly. Posts.fromHistory builds up the current state by folding a sequence of events using the Posts.apply method. As with all immutable structures, an update results in a new copy of the original state to be returned with the necessary changes applied. Fortunately we do not need to copy everything to apply changes, only the parts that are changed need to be copied. This makes immutable data structures efficient enough to be practical.

Another advantage of using an in-memory model is that it can be thoroughly tested, with no need to start or stub a database. PostsSpec.scala uses both manually written examples and randomly generated data to test the model. Using the randomly generated date we can run hundreds of tests in less than a second.

Putting it all together: the UI

The UI pulls everything together into a working application. In this case it is a standard Play! 2.0 Scala application and the main work is done by the PostsController.

In this example application the PostsController keeps a Software Transactional Memory (STM) reference to the current state of the application. This reference is our only piece of mutable data in the entire application!

[code lang=scala]
object PostsController extends Controller {
/**
* A Scala STM reference holding the current state of the application,
* which is derived from all committed events.
*/
val posts = Ref(Posts()).single
[/code]

By using an STM reference any controller method can always access the current state simply by reading from the reference. New events are applied to the current state using the commit method:

[code lang=scala]
/**
* Commits an event and applies it to the current state.
*/
def commit(event: PostEvent): Unit = {
posts.transform(_.apply(event))
Logger.debug("Committed event: " + event)
}
[/code]

The posts reference is updated by using the transform method. This ensures the update occurs in a concurrency safe manner. In this version of the application the events are not yet saved to durable storage, but to help you see what is happening while running the application the event is printed to the debug log instead. Later we’ll implement saving the committed events to durable storage before updating the current state. The commit method returns Unit, which is similar to void in Java and provides no useful information to the caller.

The PostsController.index method is invoked by Play! to render a list of posts (the mapping from URL to a controller method is configured in the routes file). The index method returns a Play! action which generates an HTTP response from an HTTP request. In this case the response is an HTTP OK (200) containing the most recent 20 posts as rendered by the index template. Notice the use of the parentheses to read the current value of the posts reference:

[code lang=scala]
/**
* Show an overview of the most recent blog posts.
*/
def index = Action { implicit request =>
Ok(views.html.posts.index(posts().mostRecent(20)))
}
[/code]

To change an existing post we need to implement a GET to render an HTML form, and a HTTP POST to validate the form and update the post. The postContentForm defines the mapping between the HTML add/edit form and the PostContent class using a Play! form:

[code lang=scala]
/*
* Blog content form definition.
*/
private val postContentForm = Form(mapping(
"author" -> trimmedText.verifying(minLength(3)),
"title" -> trimmedText.verifying(minLength(3)),
"content" -> trimmedText.verifying(minLength(3)))(PostContent.apply)(PostContent.unapply))
[/code]

The show method first tries to look up the specified post by its id. If successful it fills the form with the current contents of the post and renders the views.html.posts.edit template. Otherwise it returns an HTTP 404 (Not Found) response:

[code lang=scala]
def show(id: PostId) = Action { implicit request =>
posts().get(id) match {
case Some(post) => Ok(views.html.posts.edit(id, postContentForm.fill(post.content)))
case None => NotFound(notFound(request, None))
}
}
[/code]

When the user has made the modifications and hits the save button the submit method is invoked. First we bind the HTTP POST parameters to the form (bindFromRequest) and then perform validation (fold). If form validation succeeds it commits a new PostEdited event and redirects the browser to the PostsController.show action to show the updated post. Otherwise the method rerenders the HTML form together with the validation errors:

[code lang=scala]
def submit(id: PostId) = Action { implicit request =>
postContentForm.bindFromRequest.fold(
formWithErrors => BadRequest(views.html.posts.edit(id, formWithErrors)),
postContent => {
commit(PostEdited(id, postContent))
Redirect(routes.PostsController.show(id)).flashing("info" -> "Post saved.")
})
}
[/code]

Note that our “domain logic” is implemented directly in the controller. This is fine for simple applications like this example, but a rich domain model is usually introduced when the logic to generate the events becomes less straightforward.

Performance

Like we discusses above one of the advantages of an in-memory model is that there is no need to talk to a database just to render a view. Even though this application has not been performance tuned, it is still interesting to get a basic idea of the performance. I ran some trivial benchmarks on my laptop (an early 2010 MacBook Pro with 2.66 GHz dual core i7 with hyper-threading).

Just rendering the blog posts index view runs at about 5,500 GETs per second (with the three example blog posts you see when you start the application). This was measured with Apache JMeter using 25 concurrent client threads running on the same machine as the server.

Just submitting the blog post edit form runs at 4,200 HTTP POSTs per second or so.

In both tests the server is basically CPU bound, since we’re not performing any disk I/O. It will be interesting to see how well we can do with an actual event store implementation. The server was running on JDK 1.7.0u5 with the Garbage First GC (JVM option -XX:+UseG1GC).

Conclusion

The example application shows a simplified implementation of the event sourcing and memory image principles. Domain events are generated and used to apply updates to the memory image, which is then used to render views, without the need for a traditional database.

Hopefully you’ve noticed that none of the code is very complex. Each part (events, model, controller actions) simply focuses on a single task, without any of the “magic” that is so common with many web- or CRUD-frameworks. Even the most complicated method (Posts.apply) is a rather straightforward translation of events into updates to the current state and is easy to thoroughly test.

But the events are not yet committed to durable storage, which is necessary to build a production-ready application. In the next part we’ll see what is needed to capture the events correctly so that we can start writing the events to durable storage.

The thing with the change from waterfall to agile is that architects feel their role is being undercut, the team just goes fast and are only paying attention to the Product Owner. The standard answer they seem to get is: ‘then join the team’, but they feel reluctant to do so, and most of the times they can not fully commit (full time). So they pass, and they feel miserable about it, since now this Agile project is going to make mistakes, and can not learn from past experiences and their expertise.

The answer lies in the closer observation of the definition of Agile and Architecture.

Going agile is not the same as going fast, as the Agile Manifesto’s first principle says:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

So not just fast (early), but continuous too! And that’s the whole point of the position of architecture in Scrum/XP/Agile. In order to be able to continue to go fast, we have to make sure we mind our architecture, since architecture is:

The set of design decisions that minimize the cost of development, maintenance and use (my definition, close enough to Grady Booch’s)

So the whole point is that architects make sure that teams can go fast now and ever after, and that the cost of use and maintenance is proportional to the development investment. So architecture is not a stable deliverable or document, much better to speak about architecting; the process of keeping the architecture aligned with the product development and agile process.

Architecting is about identifying stakeholders that might be overlooked by the team and Product Owner, such as Support, Legal and 3rd party vendors. Architecting is about challenging the Product Owner and team to go slow, in order to be able to go fast by high quality standards, such as readability, 95%+ test coverage, early integration and performance testing, refactoring and the boy scout rule.

So architects, make sure you identify and represent stakeholders, and get their points on the Product Backlog (yes, it may contain non-functionals) by working closely with the Product Owner. You are a major stakeholder! Assist the team by making them go slow for a good reason, namely to serve previously overlooked stakeholders. There is a subtle balance here, if in doubt go fast with the Product Owner, and only address the issue when velocity drops because of technical debt.