If you plan to attend one of these events, or if you are just curious to see what Azure can offer, sign up for a free 90-day Azure trial. If you are a student, you should also sign up for Dreamspark, which includes a free Windows Store developer license. (You need the Windows Store developer license to send push notifications to a Windows 8 client.)

Sponsors

Pluralsight was kind enough to provide an Annual subscription to their online developer training library of over 450 video courses, a $299 value, as a raffle prize.

Telerik also provided a DevCraft Complete license for the raffle!

Please note that you must be present to receive one of the above prizes during the raffle at the end of the camp.

Web API is much more than a repository. And yes, it is indeed a protocol mapping layer. As Uncle Bob once noted, a web or api front end is just a mapping layer and is not really your application.

In many cases, however, one could argue that a web-api-as-repository is a fairly solid use case. OData is a great example. However, I was thinking of yet another argument I’ve heard for dynamic languages: when you are just going from web to database and back, you are not really working with types.

In that spirit, I set out to write a simple Web API using SQL and JSON with no explicit class definitions. You can see the results in this gist:

I used Dapper to simplify the data access, though I just as well could have used Massive, PetaPoco, or Simple.Data. Mostly I wanted to use SQL, so I went with Dapper.

I also model bind to a JObject, which I immediately cast to dynamic. I use an anonymous object to supply the values for the parameters in the SQL statements, casting the fields from the dynamic object to satisfy Dapper.

All in all, I kinda like this. Everything is tiny, and I can work directly with SQL, which doesn’t bother me one bit. I have a single class to manage my data access and API translation, but the ultimate goal of each method is still small: retrieve data and present it over HTTP. That violates SRP, but I don’t mind in this case. The code above is not very testable, but with an API like this I’d be more inclined to do top level testing anyway. It’s just not deep enough to require a lot of very specific, low-level testing, IMHO.

Also, note again that this is just retrieving data and pushing it up through an API. This is not rocket science. An F# type provider over SQL would give a good enough sanity check. Why bother generating a bunch of types?

Which brings up another point for another post: what would I do if I needed to add some logic to process or transform the data I retrieved?

As a future exercise, I want to see what it would take to cap this with the Web API OData extensions. That could be fun.

1
2
3
4
5

let endpoint = "WSHttpBinding_Test"
let username = ""
let password = ""
let client = new MyService.DataServiceClient(endpoint)
client.ClientCredentials.Windows.ClientCredential <- new NetworkCredential(username, password)

Unfortunately, the issue persisted. I then noticed a little message noting that I was missing an assembly reference. Turns out you must also open System.IdentityModel. Problem solved.

NOTE: If anyone knows of an https hosted WCF service in the wild, I’d be happy to share the full code.

1. Look for a Main.fs or Manifest.fsx file.
2. If a Manifest.fsx file doesn’t exist, generate the Manifest.fsx from #if INTERACTIVE, #r and #load references
3. Do this for all referenced files so that the Manifest.fsx lists the dependencies for the primary script
4. Search for references without a path in the local directory, then the GAC, and finally NuGet

Why an alternate build system?

Today I saw a question about docs for F#x on the F#x mailing list. The first reply, from Art, referenced Tomas’ FSharp.Formatting project. I hadn’t seen Tomas’ project in some time, so I went to check it out. The last I looked at it, FSharp.Formatting provided a Markdown parser. Tomas has extended it to cover a much larger feature set, including a version of literate programming. Literate programming was the ultimate goal of my halted focco project. As I had always intended of pulling in several of Tomas’ other projects to finish focco, and FSharp.Formatting now provides many of those features, I will now be moving what I can from focco over to FSharp.Formatting.

That was what I had originally planned to write about. Instead, worn out from the holidays, I lost focus and instead started thinking too much about true literate programming. (Disclaimer: I have not investigated Tomas’ implementation in any detail, so some of what follows may be inaccurate.) Knuth’s idea of “literate” programming allow programmers to write for people and use a tool to strip out the code and re-organize it for the computer. This sort of thing is more necessary in languages such as F# and Pascal you must first declare a procedure or function before you can use it. Neither focco nor FSharp.Formatting currently addresses this.

Before literate programming in F# is really possible, we need a build system that can work more directly from raw files. Well, that’s not entirely true, but I’ve long wished to see less XML and more F# in my F# builds.

Is this really a necessity to literate programming in F#?

No, of course not. I just found myself thinking through all this and thought I would write it down for posterity. If I really thought it necessary, I would have instead started writing some code to feel this out. Instead, I’m planning to get some sleep and try to focus and get F# project support added to FSharp.Formatting. Once that is done, we’ll see about allowing a mechanism to allow authors to list their code in an arbitrary manner and then reassemble it. And after that, maybe a look at a new build system is in order.

And no, I haven’t forgotten about FAKE. If nothing else, I would probably make this an optional extension via FAKE. Then again, you can already do some of this with a single .fsx script several loose .fs files, and some assemblies.

While pondering nested resources in Web API yet again, I realized where I think this term originates. Ruby on Rails used the term “RESTful routing” to describe their approach to building Controllers along the line of resources and using a convention to route controllers in a hierarchy based on the controller name. The goal, I think, was to correctly model and mount resources at unique URIs. However, you get what I would call “pretty URLs” for free.

If you use the term “RESTful URLs,” please stop. While RESTful Routing makes sense, RESTful URLs are just nonsense. Do use “RESTful Routing.” Do use “Pretty URLs.” Just don’t confuse your terms. Thanks!

Here’s part 1:

I’m glad I’m only getting back to this series now. I’ve had an opportunity to build many more web applications and have a much better appreciation for the poor terminology used to define web applications. For starters, this MV? business silly. We’ll get to that.

I know I’m a bit of an extremist in some things. Specifically, I like things to mean what they mean. When we abuse terms, we don’t communicate well. REST. There, I said it. I feel better. Stop using the term. Most people have a wrong idea of what it means b/c of all the silliness that has been done in its name. I don’t claim to know exactly myself. I don’t think it’s possible to rescue the term from the abuses heaped upon it. There, you see? I’m an extremist.

Now that we’ve covered that, on to MVC. I’m not sure who decided this was an accurate description for what happens on the server-side of the web, but it’s just flat wrong. As noted previously, HTTP uses a functional interface. It’s an IO-bound Request -> Response function. Can you use patterns on either side to help maintainability? Certainly! Just don’t confuse things. Let’s start with Views.

Views

What is a view?

The [view or viewport] is responsible for mapping graphics onto a device.
A viewport typically has a one to one correspondence with a display surface
and knows how to render to it. A viewport attaches to a model and renders
its contents to the display surface. In addition, when the model changes,
the viewport automatically redraws the affected part of the image to reflect
those changes. […] there can be multiple viewports onto the same model and
each of these viewports can render the contents of the model to a different
display surface.

If a view was merely a serialization of a model, this would make sense for building web applications. Unfortunately, there’s a problem. The definition suggests that the view automatically updates whenever the model changes. How do you do that with HTTP? HTTP doesn’t define any mechanism for hooking up observation of a server model. Before you say JavaScript, consider first the current use of View, or even UI. People commonly mean HTML. HTML is not a UI. HTML is a serialization format. The client (normally a browser) must interpret that HTML. Many of you will remember when that wasn’t so standard.

Can we achieve MVC today? Possibly. You might be able to leverage web sockets to reach across a client/server architecture such as that presented by HTTP. However, you are more likely to find that “MVC” on the server is just limiting. You are typically better off building a sort of restricted data access service, a.k.a Web API (subtle hint). There’s really no point in trying to enrich a serialization format to make it work more like true MVC across the client and server.

Controllers

This is no different than routing. Instead of calling your Router a Controller, you split them up. However, most frameworks really just use the router as a top level dispatcher and the controller as a lower-level dispatcher. Otherwise, I’d say web frameworks stay a lot closer to the original meaning than a lot of the other MV? patterns. (Hence the ?, of course.)

Models

This really is the crux. HTML is a model. I noted this last time. It’s just a serialization of data you want displayed. It happens to be a lot richer, but it’s still just a data model. HTML is a great way to bootstrap an application that otherwise uses JavaScript as a model serialization format. If you want to disagree, ask why HTML5 removes the presentation elements. Why has layout and style moved to CSS? CSS and the browser define the actual rendering. In a no-script web application, you don’t have to build a view. You get it for free.

Conclusion

So what? Am I just ranting that I don’t like how people abuse terms? Possibly. However, I think this goes deeper. When you allow the slippery slope, you get caught on it, as well. It’s inevitable. The bigger, lurking danger is that we start to confuse useful patterns and use them in the wrong places. Many people use MVC frameworks today to build web APIs. However, that’s not MVC. So if you then switch to a desktop app to write MVC applications, you are either confused or delighted to find that it’s so much richer.

I don’t know what I would call what we build for the web; I know I wouldn’t call it MVC. In my experiments with Frank, I’ve found that writing simple functions and nesting them with rules makes a very easy mechanism for building web APIs. I think that would essentially just be a Command pattern. Simple, elegant, and very easy to understand. YMMV.

As a preview, In this post I’ll describe the inherent functional nature of Web API. You have looked into Web API at all, you’ve probably seen the Contact Manager sample application, originally created by the infamous “2C”. (Kudos to Dan Roth on the Web API team for keeping it alive!) The controller-based goodness baked into Web API hides a lot of the functional nature, so I will understand if you think I’m a little crazy for my statement above.

See? Not so much? Okay, how about in F#?

1
2
3
4

type MyMessageHandler() =
  inherit System.Net.Http.HttpMessageHandler()
  override x.SendAsync(request, cancellationToken) =
    (* Implementation ... *)

Eh, let’s try again.

1
2
3
4
5
6
7

val app : HttpRequestMessage -> Async<HttpResponseMessage>
let app request = async { return response }

let makeHandler app =
  { new System.Net.Http.HttpMessageHandler() with
      override x.SendAsync(request, cancellationToken) =
        Async.StartAsTask <| app request }

Here I use several unique features of F# to emphasize the point. The type signature should be familiar from my last post. I then create a handler using F#’s object initialization syntax, which creates an instance from a base class using the app from above. I think I’ve made my point that, at its heart, the HttpMessageHandler is functional in its core.

Okay, let’s translate back to C#-land. We don’t have object initialization syntax, but we can create pure, functional approaches for C#. Also, we’ll use DelegatingHandler, as that’s the actual type you’ll need to inherit in order to use it in Web API. (I apologize now for making your eyes bleed with the type signatures.)

Why is this core?

I’ve mentioned this is core, but if you’ve only used Web API with controllers, you may never have interacted with HttpMessageHandlers. HttpMessageHandlers are at the root of everything. HttpClient, HttpServer, and everything else that uses these types all inherit HttpMessageHandler. If you look deep into Web API, you find this consistency of HttpMessageHandler at the core. What does that mean? It means “it’s [functions] all the way down.”

A quick example

You can see this in action in the AspNetWebApi.Conneg sample application.

Okay, so now we have our “function.” How would you hook this up? That’s also quite simple. HttpApplication now has a GlobalConfiguration courtesy System.Web.Http. This provides you access to an instance of HttpConfiguration, which provides you all the extensibility points for managing your Web API. The Self Host option let’s you create your own. Attaching a handler is dead simple:

The point

By no means am I trying to convince you to give up all the ease of built-in model binding and conneg. I only want to drive home the point that the core of Web API is really just functions. As I mentioned previously, HTTP exhibits a very strong functional side. Web API really addresses this head on and provides a very solid core for building web applicaitons, whether using a more MVC-style or functions directly.

Try it out. For really simple applications, you are likely to find that these are all you need. You may also appreciate that you can implement your own routing and other mechanisms directly within a composed set of functions. We’ll explore this more as we continue in this series, so stay tuned.

I hope you find it interesting. I certainly have, as it’s helped me solidify some ideas about the approach I’ve been taking lately. Stay tuned for Part 2, as well as the other posts on functional HTTP.

The Basics

At the bare minimum, HTTP supports an interface in which a client submits an HttpRequestMessage and responds with an HttpResponseMessage. This contract can be represented as a function with the following signature:

1

HttpRequestMessage -> HttpResponseMessage

I could try to just leave it here and say, “See, it’s a function!” but that would be cheating a bit. Nevertheless, the essential contract for HTTP is a simple function. That’s at least something.

Summary

Let’s dive a bit deeper. What precisely might one mean by “functional programming paradigm”? Typically, you’ll see some variation on the following list (which is by no means exhaustive):

• Declarative - what not how
• Pure functions - a function with no side-effects such as I/O or mutating global state
• First-class and higher-order functions - functions can be created and passed as parameters just like other values
• Referential transparency - an expression can be replaced with its value without changing program behavior
• Memoization - a performance enhancing technique made possible by referential transparency

Aligning the above items with ideas found in HTTP, we get:

• Declarative -> HttpRequestMessage headers
• Pure functions -> Safe, idempotent HTTP methods such GET and HEAD
• First-class and higher-order functions -> Hypermedia controls and content negotiation
• Referential transparency and Memoization -> Cache control mechanisms

Discussion

Declarative HTTP

This is the easiest to demonstrate. Request headers are nothing other than a means to declare what you want (e.g. Request Line) and set expectations (e.g. Accept). The entire HttpRequestMessage exists as a means of expressing intent. As a client, you have absolutely no chance of instructing the server on how to process your request. Even RPC-style calls must abide by this constraint, at least as it concerns the request message.

Pure Functions

Some may argue this point due to the prevalence of non-conforming web applications that allow side effects on safe, idempotent methods. Nevertheless, HTTP as it is defined specifies that GET and HEAD methods should be safe. This is quite important for other attributes of HTTP, such as the ability to cache representations, which I’ll discuss a bit further down.

First-class and Higher-order Functions

I will acknowledge that this is my weakest point. However, the fact that both the client and server are able to communicate “callable” options appears very close to the very mechanism used when supplying callbacks or continuations in functional programming. It’s close enough for me. You are free to disagree.

Referential Transparency and Memoization

Referential transparency, which is supported by the presence of pure functions (see above), allows us to safely support caching, known as memoization in functional programming. In functional programming languages, memoization allows us to offset the cost of immutability by calling a function once and then re-using the original value rather than continuously invoking the function over and over. It’s especially useful for expensive operations. In much the same way, HTTP provides the ability to cache representations at intermediaries to speed the process of returning a response.

What About Non-Safe-Idempotent and Non-Idempotent Methods?

Very few functional languages are considered pure, Haskell arguably the most popular. F#, OCaml, Lisp, Scala, JavaScript and others all support mutation. Truly, you would be unable to do very much without the ability to change state somewhere. Thus, these other methods are both useful and in no way against the idea that HTTP supports the functional style. Much as even Haskell can eventually persist data in mutable storage such as a database, HTTP supports controlled mutation (albeit without explicit monads; count your blessings).

Conclusion

I think the above arguments support my original idea pretty well, but I’m curious what you think. Also, I certainly am using this as a means of drawing attention to a project I’ve been working and churning on, related to both F# and Web API. In upcoming posts, I’ll introduce Frank, which is finally approaching a fairly stable api.

Why should you start with a web api rather than just building a web site/app that has HTML and JavaScript all working together? Frankly it’s because you can’t properly decouple the api from the serialization format well enough. When you are designing an api for the web, you (should be) thinking in terms of resources and representations. That’s representations in the plural form. You may not know exactly what forms you’ll need, but you should consider that you will eventually have many. When you start with a web site/app with HTML in mind, you’ve coupled yourself to a single format, and extracting that out later could (will) be difficult. Don’t just take my word for it. Mike Amundsen has an excellent post on the right way to think about these things.

There are certainly some instances where this may be overkill, but I hate rewriting software unless it really is a prototype or just an exploratory attempt to get something up. In those cases, go for the quick and dirty. If you are working on something you want to last a long time, however, you owe it to yourself to consider the evolvability of your app by focusing on api design. You’ll then be able to take advantage of a number of client options. Certainly, supporting the growing number of clients is one of the biggest challenges continuing to face developers as we head into 2012.

Let’s suppose you agree with me on this point. How do you go about building a really solid api design? I don’t think I could articulate it better than Darrel Miller already has. His goals for good apis are suitable both for internal teams and external customers. Who wouldn’t love gaining visibility b/c a customer was able to accelerate their business by using your api in an unforseen way that drives additional business for your own company? How nice is it to knock out not just one project but several b/c you are able to leverage existing platforms for new projects? We’re doing that at Logos, in large part because we moved to MVC and took a more service-oriented approach to building our apps. The number of new projects has grown tremendously, but we are also able to respond much more quickly b/c the services are ready for consumption.

I’ll be continuing to discuss this topic in future posts. In the meantime, check out Mike’s book, Building Hypermedia APIs with HTML5 and Node. It uses HTML5 and Node to illustrate, but the concepts are excellent and portable to other platforms. I also highly recommend REST in Practice as an excellent resource for understanding the fullness of what HTTP offers for building apis. Enjoy!

Before I go any further, I want to make very clear that I’m a fan of Rails and think that it is an incredibly useful framework for building certain applications. I think the same is true of ASP.NET MVC. I’ve used and use both. However, I don’t think that means these are the best approach for tackling HTTP.

Chad’s post was very good. I agree with almost all of his points. In particular, I think he hit several very important points, the biggest of which was his emphatic quote:

INHERITANCE IS FAIL FOR MOST DESIGNS, BUT NEVER MORE SO THAN IN WEB/MVC FRAMEWORKS!

Then from his list of criteria for a “perfect” web framework:

• Completely compositional: 100% agree
• [Resource]-based: I take a slightly different direction here. We talk about “models”, but if you are really thinking about HTTP, you should be thinking in terms of resources. They are similar in many ways, notably in the sorts of “behaviors” (in FubuMVC terms) that should be composed with the “model.” Honestly, it’s the “model” term itself that seems totally overloaded or at least at the wrong layer when talking about a framework. If you have a model, you’re probably really dealing with something more like DDD than the HTTP application layer.
• Not have any notion of controllers: Agree. In MVC-style frameworks, they are sort of necessary, but if you break away from MVC, you’ll find there are better alternatives.
• Action-based: I’m all about using functions, or delegates, as 'Request -> 'Response handlers. I also agree that these should generally be kept simple, relying on compositional helpers, middlewares, or whatever you want to call them to perform model-binding and response generation. Granted, this is slightly different than what Chad is talking about, and I’ll cover my thoughts on this in more detail below.
• Minimized code-in-view: templates are helpful, but the ability to add a lot of logic is just asking for trouble. I really like the tags-as-code approach taken by WebSharper and Wing Beats.
• Independent of other frameworks: Definitely limit other dependencies and play nice with others.
• Testing as a first-class citizen: At least allow whatever the consumer writes to easily write real unit tests. Generally, the only reason you can’t do this is related to hosting, so make sure the hosting model is not necessary to test the application.

Okay, so I obviously skipped some items. I don’t think the other items are bad; I just don’t think they are necessary for a “perfect” web framework.

• Total adherence to principles: This sounds really good. I don’t disagree, really, but exactly which principles? Principles sometimes change. Some make sense in one approach to software development and are sort of non-issues in others. Look at the description of SOLID and its relevance to F#. You’ll note that many of the principles are sort of non-issues.
• Statically typed: I don’t really see this as a requirement. HTTP is inherently dynamic. You could build a useful, fairly complex HTTP application using only dictionaries. Reliance upon type serialization is misguided. How do you generate accurate Atom or HTML from a static type? You don’t. You need a template or some other custom output. At the same time, I have grown to like static type-checking, espeically in F#, and F# 3.0’s Type Providers prove you can do a lot, even type check regular expressions. I just don’t see this as a strict requirement.
• Based on IoC: I love IoC in C# applications. I can’t imagine life without it. However, I’ve found that I don’t even think about IoC when writing F# applications. It’s just not a concern. Higher-order functions, computation expressions, and other functional-style approaches generally render IoC a non-issue. So again, I don’t really see this as a requirement but a case of “use the right tool for the job.”
• Mostly model-based conventional routing: This is a totally fine solution. I take issue with having any single prescription for routing. There are so many ways to handle this, and in the end it doesn’t matter. If you stick to a resource-oriented approach, which perhaps is the goal Chad is getting at, then your routing should be pretty straightforward and simple. One thing many routers miss is the connection between different methods assigned to a single uri. So far as I know, OpenRasta and Web Api are the only frameworks that return the appropriate HTTP status codes for missing methods on an uri or unmatched Accept header, among other things. FubuMVC may allow this through a convention.
• One Model In, One Model Out: When using MVC, I generally like this approach. However, I again don’t see this as a requirement. I’m not convinced a model is always necessary. Again, HTTP is quite dynamic, and it’s sometimes easier to work directly with a request or pull out several different “objects” from a request to help generate a response. Though perhaps 'Request and 'Response are models and the point is moot. :)

Now for the biggest divergence: I generally prefer to be explicit rather than lean on conventions. In C#, I typically go for conventions because of the amount of noise required by the C# syntax. With F#, I’m able to be explicit without suffering syntax noise overload. The same is true for many dyanmic languages like Ruby and Python. So I disagree with both the Conventional top-to-bottom and Zero touch points. Again, I’m not fully opposed, but I definitely don’t see these as requirements at the framework level. Sure, include it as an option, especially if, as in FubuMVC, you can redefine the conventions to suit your needs. This could just mean use the right tool for the job, or it could mean you should take a simpler approach and build your own conventions as wrapper functions.

If you think this sounds like a prescription to write a lot more code, I’ll show a comparison:

Two extra lines (in this instance) and it’s likely with a more complex example you wouldn’t see much of a difference in terms of line count. I doubt this is true with C#, so don’t take this as a statement that this is the “one, true way.” I like conventions when using C#, just as I like IoC in C#.

In any case, definitely check out FubuMVC as an alternative to MVC. There’s a lot to like, and the FubuMVC team has done an incredible job building a terrific platform. I’ll try to highlight a few others in the upcoming weeks, but while you’re looking at Fubu, also checkout OpenRasta and ServiceStack.

This is a slight over-generalization. I want to stress that I don’t think type systems are bad, whether static or dynamic. I’ve just suddenly realized that, should you extend a type system with tools to define pre- and post-variants and invariants, as well as performance or probability requirements, as some unit tests do, you end up with a very nice assertion mechanism that could replace all the different tools currently used in programming.

As a means of testing out this idea, I’ve started investigating the Lively Kernel and want to see what might happen should I add a contracts.coffee layer with a mechanism for validating the program upon saving changes to a script.

OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things. It can be done in Smalltalk and in LISP.

The Iteratee module used in this post is part of the FSharpx library and provides a set of types and functions for building compositional, input processing components.

A fold, by any other name

A common scenario for I/O processing is parsing an HTTP request message. Granted, most will rely on ASP.NET, HttpListener, or WCF to do this for them. However, HTTP request parsing has a lot of interesting elements that are useful for demonstrating problem areas in inefficient resource usage using other techniques. For our running sample, we’ll focus on parsing the headers of the following HTTP request message:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

let httpRequest : byte [] = @"GET /some/uri HTTP/1.1
Accept:text/html,application/xhtml+xml,application/xml
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Host:stackoverflow.com
If-Modified-Since:Sun, 25 Sep 2011 20:55:29 GMT
Referer:http://www.bing.com/search?setmkt=en-US&q=don't+use+IEnumerable%3Cbyte%3E
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.4 (KHTML, like Gecko) Chrome/16.0.889.0 Safari/535.4

<!DOCTYPE HTML PUBLIC ""-//W3C//DTD HTML 4.01//EN"" ""http://www.w3.org/TR/html4/strict.dtd"">
<html>
<head>
...
</head>
<body>
...
</body>
</html>"B

When parsing an HTTP request message, we most likely want to return not just a list of strings but something more descriptive, such as an abstract syntax tree represented as a F# discriminated union. A perfect candidate for taking in our request message above and producing the desired result is Seq.fold.

1

val fold : ('State -> 'a -> 'State) -> 'State -> seq<'a> -> 'State

The left fold is a very useful function. It equivalent to the Enumerable.Aggregate extension method in LINQ. This function takes in a state transformation function, an initial seed value, and a sequence of data, and produces a final state value incrementally.

Looks like we’re done here. However, there are still problems with stopping here:

Composition

You can certainly use function composition to generate an appropriate state incrementing function, but you would still have the problem of being able to pause to delay selecting the appropriate message body processor.

Early termination

Suppose you really only ever want just the message headers. How would you stop processing to free your resources as quickly as possible? Forget it. Fold is going to keep going until it runs out of chunks. Even if you have your fold function stop updating the state after the headers are complete, you won’t get a result until the entire data stream has been processed.

Iteratees

The iteratee itself is a data consumer. It consumes data in chunks until it has either consumed enough data to produce a result or receives an EOF.

An iteratee is based on the fold operator with two differences:

1. The iteratee may complete its processing early by returning a Done state. The iteratee may return the unused portion of any chunk it was passed in its Done state. This should not be confused with the rest of the stream not yet provided by the enumerator.

2. The iteratee does not pull data but receives chunks of data from an “enumerator”. It returns a continuation to receive and process the next chunk. This means that an iteratee may be paused at any time, either by neglecting to pass it any additional data or by passing an Empty chunk.

Inspiration

For those interested in reading more, there are any number of articles and examples. Here are a few I’ve referenced quite frequently:

• Iteratee IO
• enumerator package for Haskell
• Yesod Book - Enumerator package
• Scalaz Tutorial: Enumeration-Based I/O with Iteratees

The Scalaz version arguably comes closest to the F# implementation, but I have generally based my implementations on the Haskell source. I’ve got two implementations currently, a version based on the enumerator package, and a continuation-passing style version based on the iteratee package.

The iteratee is composed of one of two states, either a done state with a result and the remainder of the data stream, or a continue state containing a continuation to continue processing.

1
2
3

type Iteratee<'el, 'a> =
    | Done -> 'a * Stream<'el>
    | Continue -> (Stream<'el> -> Iteratee<'el, 'a>)

The Stream type used here is not System.IO.Stream but yet another discriminated union to represent different stream chunk states:

1
2
3
4

type Stream<'el> =
    | Chunk -> 'el  // A chunk of data
    | Empty         // An empty chunk represents a pause and can be used as a transition in compositions
    | EOF           // Denotes that no chunks remain

Some Iteratees

The following are some sample iteratees using the List<’a> type in F#. While not the most efficient, they are easy to express concisely and are thus better for illustrating the use of iteratees.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101

let length<'a> : Iteratee<'a list, int> =
    let rec step n = function
        | Empty | Chunk [] -> Continue (step n)
        | Chunk x -> Continue (step (n + x.Length))
        | EOF -> Done(n, EOF)
    in Continue (step 0)

let peek<'a> : Iteratee<'a list, 'a option> =
    let rec inner =
        let rec step = function
            | Empty | Chunk ([]:'a list) -> inner
            | Chunk(x::xs) as s -> Done(Some x, s)
            | EOF -> Done(None, EOF)
        Continue step
    in inner

let head<'a> : Iteratee<'a list, 'a option> =
    let rec inner =
        let rec step = function
            | Empty | Chunk ([]:'a list) -> inner
            | Chunk(x::xs) -> Done(Some x, Chunk xs)
            | EOF -> Done(None, EOF)
        Continue step
    in inner

let drop n =
    let rec step n = function
        | Empty | Chunk [] -> Continue <| step n
        | Chunk str ->
            if str.Length < n then
                Continue <| step (n - str.Length)
            else let extra = List.skip n str in Done((), Chunk extra)
        | EOF -> Done((), EOF)
    in if n <= 0 then empty<_> else Continue (step n)

let private dropWithPredicate pred listOp =
    let rec step = function
        | Empty | Chunk [] -> Continue step
        | Chunk x ->
            match listOp pred x with
            | [] -> Continue step
            | x' -> Done((), Chunk x')
        | EOF as s -> Done((), s)
    in Continue step

let dropWhile pred = dropWithPredicate pred List.skipWhile
let dropUntil pred = dropWithPredicate pred List.skipUntil

let take n =
    let rec step before n = function
        | Empty | Chunk [] -> Continue <| step before n
        | Chunk str ->
            if str.Length < n then
                Continue <| step (before @ str) (n - str.Length)
            else let str', extra = List.splitAt n str in Done(before @ str', Chunk extra)
        | EOF -> Done(before, EOF)
    in if n <= 0 then Done([], Empty) else Continue (step [] n)

let private takeWithPredicate (pred:'a -> bool) listOp =
    let rec step before = function
        | Empty | Chunk [] -> Continue (step before)
        | Chunk str ->
            match listOp pred str with
            | str', [] -> Continue (step (before @ str'))
            | str', extra -> Done(before @ str', Chunk extra)
        | EOF -> Done(before, EOF)
    in Continue (step [])

let takeWhile pred = takeWithPredicate pred List.span
let takeUntil pred = takeWithPredicate pred List.split

let heads str =
    let rec loop count str =
        match count, str with
        | (count, []) -> Done(count, EOF)
        | (count, str) -> Continue (step count str)
    and step count str s =
        match str, s with
        | str, Empty -> loop count str
        | str, (Chunk []) -> loop count str
        | c::t, (Chunk (c'::t')) ->
            if c = c' then step (count + 1) t (Chunk t') 
            else Done(count, Chunk (c'::t'))
        | _, s -> Done(count, s)
    loop 0 str

// readLines clearly shows the composition allowed by iteratees.
// Note the use of the heads and takeUntil combinators.
let readLines =
    let toString chars = String(Array.ofList chars)
    let newlines = ['\r';'\n']
    let newline = ['\n']
    let isNewline c = c = '\r' || c = '\n'
    let terminators = heads newlines >>= fun n -> if n = 0 then heads newline else Done(n, Empty)
    let rec lines acc = takeUntil isNewline >>= fun l -> terminators >>= check acc l
    and check acc l count =
        match l, count with
        | _, 0 -> Done(Choice1Of2 (List.rev acc |> List.map toString), Chunk l)
        | [], _ -> Done(Choice2Of2 (List.rev acc |> List.map toString), EOF)
        | l, _ -> lines (l::acc)
    lines []

Enumerators

Enumerators feed data into iteratees, advancing their state until they complete.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

// Feed the EOF chunk to the iteratee in order to force the result.
let rec enumEOF = function 
    | Done(x,_) -> Done(x, EOF)
    | Continue k ->
        match k EOF with
        | Continue _ -> failwith "enumEOF: divergent iteratee"
        | i -> enumEOF i

// Run the iteratee and either return the result or throw an exception for a divergent iteratee.
let run i =
    match enumEOF i with
    | Done(x,_) -> x
    | Continue _ -> failwith "run: divergent iteratee"

// Feed the data stream to the iteratee as a single chunk.
// This is useful for testing but not conserving memory.
// val enumeratePure1Chunk :: 'a list -> Enumerator<'a list,'b>
let enumeratePure1Chunk str i =
    match str, i with 
    | [], _ -> i
    | _, Done(_,_) -> i
    | _::_, Continue k -> k (Chunk str)

// Feed data to the iteratee in chunks of size n.
// val enumeratePureNChunk :: 'a list -> int -> Enumerator<'a list,'b>
let rec enumeratePureNChunk n str i =
    match str, i with
    | [], _ -> i
    | _, Done(_,_) -> i
    | _::_, Continue k ->
        let x, xs = List.splitAt n str in enumeratePureNChunk n xs (k (Chunk x))

// Feed data to the iteratee one value at a time.
//val enumerate :: 'a list -> Enumerator<'a list,'b>
let rec enumerate str i = 
    match str, i with
    | [], _ -> i
    | _, Done(_,_) -> i
    | x::xs, Continue k -> enumerate xs (k (Chunk [x]))

Reading an HTML Request by Lines

Back to our earlier example of reading HTTP requests, let’s see how we might read it by lines. I’ll add a few caveats. This is not encapsulating the Stream well as we are using a memory stream. I could wrap this up in an enumerateMemoryStream function that takes all the bytes to read (which will likely happen soon). The example below is using the Iteratee.Binary module, not the Iteratee.List module.

1
2
3
4
5
6
7
8
9
10

let runIteratee() =
    let sw = Stopwatch.StartNew()
    let result =
        [ for _ in 1..10000 do
            use stream = new MemoryStream(httpRequest)
            yield! match enumStream 128 stream readLines |> run with
                   | Choice1Of2 x -> x
                   | Choice2Of2 y -> y ]
    sw.Stop()
    printfn "Iteratee read %d lines in %d ms" result.Length sw.ElapsedMilliseconds

So what?

System.IO.Stream-based processing

The System.IO.Stream type should be familiar to anyone who has ever worked with I/O in .NET. Streams are the primary abstraction available for working with streams of data, whether over the file system (FileStream) or network protocols (NetworkStream). Streams also have a nice support structure in the form of TextReaders and TextWriters, among other, similar types.

Using the standard Stream processing apis, you might write something like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

let rec readConsecutiveLines (reader:System.IO.StreamReader) cont =
    if reader.EndOfStream then cont []
    else let line = reader.ReadLine()
         if System.String.IsNullOrEmpty(line) then cont []
         else readConsecutiveLines reader (fun tail -> cont (line::tail))

let readFromStream() =
    let sw = System.Diagnostics.Stopwatch.StartNew()
    let result =
        [ for _ in 1..10000 do 
            use stream = new System.IO.MemoryStream(httpRequest)
            use reader = new System.IO.StreamReader(stream)
            yield readConsecutiveLines reader id ]
    sw.Stop()
    printfn "Stream read %d in %d ms" result.Length sw.ElapsedMilliseconds

readFromStream()

What might be the problems with this approach?

Blocking the current thread

This is a synchronous use of reading from a Stream. In fact, StreamReader can only be used in a synchronous fashion. There are no methods that even offer the Asynchronous Programming Model (APM). For that, you’ll need to read from the Stream in chunks and find the line breaks yourself. We’ll get to more on this in a few minutes.

Poor composition

First off, the sample above is performing side effects throughout, and side-effects don’t compose. You might suggest using standard function composition to create a complete message parser, but what if you don’t know ahead of time what type of request body you’ll receive? Can you pause processing to select the appropriate processor? You could certainly address this with some conditional branching, but where exactly do you tie that into your current logic? Also, the current parser breaks on lines using the StreamReader’s built in logic. If you want to do parsing on individual lines, where would you tie in that logic?

Memory consumption

In this example, we are not taking any care about our memory use. To that end, you might argue that we should just use StreamReader.ReadToEnd() and then break up the string using a compositional parser like FParsec. If we don’t care about careful use of memory, that’s actually quite a good idea. However, in most network I/O situations – such as writing high-performance servers – you really want to control your memory use very carefully. StreamReader does allow you to specify the chunk size, so it’s not a complete case against using StreamReader, which is why this is a last-place argument. And of course, you can certainly go after Stream.Read() directly.

IEnumerable-based processing

How can we add better compositionality and refrain from blocking the thread? One way others have solved this problem is through the use of iterators. A common example can be found on Stack Overflow. Iterators allow you to publish a lazy stream of data, either one element at a time or in chunks, and through LINQ or F#’s Seq module, chain together processors for the data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

// Read the stream byte by byte
let readStreamByByte (stream: System.IO.Stream) =
    seq { while true do
            let x = stream.ReadByte()
            if (int x) < 0 then ()
            else yield x }

// Read the stream by chunks
let readStreamByChunk chunkSize (stream: System.IO.Stream) =
    let buffer = Array.zeroCreate<byte> chunkSize
    seq { while true do
            let bytesRead = stream.Read(buffer, 0, chunkSize)
            if bytesRead = 0 then ()
            else yield buffer }

// When you are sure you want text by lines
let readStreamByLines (reader: System.IO.StreamReader) =
    seq { while not reader.EndOfStream do
            yield reader.ReadLine() }

Three options are presented. In each, I’m using a Stream or StreamReader, but you could just as easily replace those with a byte[], ArraySegment<byte>, or SocketAsyncEventArgs. What could be wrong with these options?

Lazy, therefore resource contentious

Iterators are pull-based, so you’ll only retrieve a chunk when requested. This sounds pretty good for your processing code, but it’s not a very good situation for your sockets. If you have data coming in, you want to get it moved out as quickly as possible to free up your allocated threads and pinned memory buffers for more incoming or outgoing traffic.

Lazy, therefore non-deterministic

Each of these items is lazy; therefore, it’s impossible to know when you can free up resources. Who owns the Stream or StreamReader passed into each method? When is it safe to free the resource? If used immediately within a function, you may be fine, but you’d also be just as well off if you used a list or array comprehension and blocked until all bytes were read. (The one possible exception might be when using a co-routine style async pattern.)

Summary

Iteratees are able to resolve a number of issues related to general stream processing, as well as either lazy, e.g. seq<_>- or Async<_>-based, non-determinism. I’m sure you are wondering why I didn’t dive deeper in explaining the iteratees above. In the next part, we’ll explore a more idiomatic .NET solution for creating iteratees. We’ll then decompose the iteratees above and cover each in more detail. In the meantime, please checkout FSharpx for additional information.

1. Iteratees in F# and their use in creating a chunk-driven parser
2. HTTP as the command line interface (CLI) for the web

Stay-tuned