Sunday, December 30, 2007

Pipelines Using Fibers in Ruby 1.9

Users of the command line are familiar with the idea of building pipelines: a chain of simple commands strung together to the output of one becomes the input of the next. Using pipelines and a basic set of primitives, shell users can accomplish some sophisticated tasks. Here’s a basic Unix shell pipeline that reports the ten longest .tip files in the current directory, based on the number of lines in each file:


 wc -l *.tip | grep \.tip | sort -n | tail -10

Let’s see how to add something similar to Ruby. By the end of this set of two articles, we’ll be able to write things like


puts (even_numbers | tripler | incrementer | multiple_of_five ).resume

and a palindrome finder using blocks:


words            = Pump.new %w{Madam, the civic radar rotator is not level.}
is_palindrome = Filter.new {|word| word == word.reverse}

pipeline = words .| {|word| word.downcase.tr("^a-z", '') } .| is_palindrome

while word = pipeline.resume
puts word
end

Great code? Nope. But getting there is fun. And, who knows? The techniques might well be useful in your next project.


A Daily Dose of Fiber


Ruby 1.9 adds support for Fibers. At their most basic, let you create simple generators (much as you could do previously with blocks. Here’s a trivial example: a fiber that generates successive Fibonacci numbers:


      fib = Fiber.new do
f1 = f2 = 1
loop do
Fiber.yield f1
f1, f2 = f2, f1 + f2
end
end

10.times { puts fib.resume }

A fiber is somewhat like a thread, except you have control over when it gets scheduled. Initially, a fiber is suspended. When you resume it, it runs the block until the block finishes, or it hits a Fiber.yield. This is similar to a regular block yield: it suspends the fiber and passes control back to the resume. Any value passed to Fiber.yield becomes the value returned by resume.


By default, a fiber can only yield back to the code that resumed it. However, if you require the "fiber"library, Fibers get extended with a transfer method that allows one fiber to transfer control to another. Fibers then become fully fledged coroutines. However, we won’t be needing all that power today.


Instead, let’s get back to the idea of creating pipelines of functionality in code, much as you can create pipelines in the shell.


As a starting point, let’s write two fibers. One’s a generator—it creates a list of even numbers. The second is a consumer. All it does it accept values from the generator and print them. We’ll make the consumer stop after printing 10 numbers.


    evens = Fiber.new do
value = 0
loop do
Fiber.yield value
value += 2
end
end

consumer = Fiber.new do
10.times do
next_value = evens.resume
puts next_value
end
end

consumer.resume

Note how we had to use resume to kick off the consumer. Technically, the consumer doesn’t have to be a Fiber, but, as we’ll see in a minute, making it one gives us some flexibility.


As a next step, notice how we’ve created some coupling in this code. Our consumer fiber has the name of the evens generator coded into it. Let’s wrap both fibers in a method, and pass the name of the generator into the consumer method.


    def evens
Fiber.new do
value = 0
loop do
Fiber.yield value
value += 2
end
end
end

def consumer(source)
Fiber.new do
10.times do
next_value = source.resume
puts next_value
end
end
end

consumer(evens).resume

OK. Let’s add one more fiber to the weave. We’ll create a filter that only passes on numbers that are multiples of three. Again, we’ll wrap it in a method.


    def evens
Fiber.new do
value = 0
loop do
Fiber.yield value
value += 2
end
end
end

def multiples_of_three(source)
Fiber.new do
loop do
next_value = source.resume
Fiber.yield next_value if next_value % 3 == 0
end
end
end

def consumer(source)
Fiber.new do
10.times do
next_value = source.resume
puts next_value
end
end
end

consumer(multiples_of_three(evens)).resume

Running this, we get the output


0
6
12
18
. . .

This is getting cool. We write little chunks of code, and then combine them to get work done. Just like a pipeline. Except…


We can do better. First, the composition looks backwards. Because we’re passing methods to methods, we write


    consumer(multiples_of_three(evens))

Instead, we’d like to write


    evens | multiples_of_three | consumer

Also, there’s a fair amount of duplication in this code. Each of our little pipeline methods has the same overall structure, and each is coupled to the implementation of fibers. Let’s see if we can fix this.


Wrapping Fibers


As is usual when we’re refactoring towards a solution, we’re about to get really messy. Don’t worry, though. It will all wash off, and we’ll end up with something a lot neater.


First, let’s create a class that represents something that can appear in our pipeline. At it’s heart is theprocess method. This reads something from the input side of the pipe, then “handles” that value. The default handling is to write that value to the output side of the pipeline, passing it on to the next element in the chain.


    class PipelineElement

attr_accessor :source

def initialize
@fiber_delegate = Fiber.new do
process
end
end

def resume
@fiber_delegate.resume
end

def process
while value = input
handle_value(value)
end
end

def handle_value(value)
output(value)
end

def input
source.resume
end

def output(value)
Fiber.yield(value)
end
end

When I first wrote this, I was tempted to make PipelineElement a subclass of Fiber, but that leads to coupling. In the end, the pipeline elements delegate to a separate Fiber object.


The first element of the pipeline doesn’t receive any input from prior elements (because there are no prior elements), so we need to override its process method.


    class Evens < PipelineElement
def process
value = 0
loop do
output(value)
value += 2
end
end
end

evens = Evens.new

Just to make things more interesting, we’ll create a generic MultiplesOf filter, so we can filter based on any number, and not just 3:


    class MultiplesOf < PipelineElement
def initialize(factor)
@factor = factor
super()
end
def handle_value(value)
output(value) if value % @factor == 0
end
end

multiples_of_three = MultiplesOf.new(3)
multiples_of_seven = MultiplesOf.new(7)

Then we just knit it all together into a pipeline:


    multiples_of_three.source = evens
multiples_of_seven.source = multiples_of_three

10.times do
puts multiples_of_seven.resume
end

We get 0, 42, 84, 126, 168, and so on as output. (Any output stream that contains 42 must be correct, so no need for any unit tests here.)


But we’re still a little way from our ideal of being able to pipe these puppies together. It’s a good thing that Ruby let’s us override the “|” operator. Up in classPipelineElement, define a new method:


    def |(other)
other.source = self
other
end

This allows us to write:


    10.times do
puts (evens | multiples_of_three | multiples_of_seven).resume
end

or even:


    pipeline = evens | multiples_of_three | multiples_of_seven

10.times do
puts pipeline.resume
end

Cool, or what?


In The Next Thrilling Installment


The next post will take these basic ideas and tart them up a bit, allowing us to use blocks directly in pipelines. We’ll also reveal why our PipelineElement class I just wrote is somewhat more complicated than might seem necessary. In the meantime, here’s the full source of the code so far.


    class PipelineElement

attr_accessor :source

def initialize
@fiber_delegate = Fiber.new do
process
end
end

def |(other)
other.source = self
other
end

def resume
@fiber_delegate.resume
end

def process
while value = input
handle_value(value)
end
end

def handle_value(value)
output(value)
end

def input
source.resume
end

def output(value)
Fiber.yield(value)
end
end

##
# The classes below are the elements in our pipeline
#
class Evens < PipelineElement
def process
value = 0
loop do
output(value)
value += 2
end
end
end

class MultiplesOf < PipelineElement
def initialize(factor)
@factor = factor
super()
end
def handle_value(value)
output(value) if value % @factor == 0
end
end

evens = Evens.new
multiples_of_three = MultiplesOf.new(3)
multiples_of_seven = MultiplesOf.new(7)

pipeline = evens | multiples_of_three | multiples_of_seven

10.times do
puts pipeline.resume
end

Wednesday, December 12, 2007





Ruby 1.9 is just around the corner, so it looks like a good time to create a new edition of Programming Ruby. So, I’m pleased to announce that the Third Edition of the PickAxe has just entered beta.


The book’s home page is at http://pragprog.com/titles/ruby3.


Although 1.9 is largely compatible with 1.8, there are definite differences. And it’s been an interesting ride getting the examples in the book to compile and run with the current 1.9 interpreter. The book pushes the envelope in many different areas, and includes example code designed to illustrate edge cases. When I find these, I’m flagging them in the text and (if they look like bugs) adding them to the tracking system. But, so far, 1.9 is looking like a big win for Ruby.


Nice job, everyone.

Thursday, November 8, 2007

Michael Swaine on Computer Book Publishing

I few weeks back, Michael Swaine emailed asking for our impressions of the computer book publishing situation. He also checked in with some other publishers. His conclusions are posted in a new online Dr Dobbs article.


I know that traditional publishing will never die—people just like holding paper books too much. But the reality of the new world will continue to drive publishers to find new ways of getting information out there. That’s why our business is successful. We started by applying software development project techniques to book creation (version control, DRY, testing, etc). That gave us the lowest cost model of any publisher I know, which in turn let us accept proposals because we wanted to see the book published, and not because some marketing model told us it would sell. We tried other innovations, such as applying agile principles to publishing, gaining early and continuous feedback with our beta book program. And we’ll continue to innovate in the production of books—I still get too much of a thrill from holding the “fresh-from-the-printer smelling first copy of one of our paper books not to.


But, underneath it all, we’re not really book publishers. What drives us is helping the community we know: software developers. The initial starter kit series was simply a way of spreading the information that we gave to teams during our consulting gigs to a wider audience. The Ruby and the Rails books were just a way of sharing a passion. The Erlang book, and books such as Release It! and Design Accessible Web Sites highlight technologies and techniques that we think are cool, and that we think will help other developers. We choose to get that information out in book form because we feel that the process of creating a book helps distill and refine the information. It lets us add value as editors.


I think the problems of the publishing industry will turn out to be a good thing. People get complacent during good times. When when times get harder, the folks who survive are the ones who focus in on the core of what they’re doing. All the publishers still in the field are experimenting and innovating as they refine their focus.


Our focus is a passion for software development and a desire to help software developers have more fun while doing one of the most challenging jobs there is. I’m having a blast.

Tuesday, October 16, 2007

Art in Programming

Andy and I gave a talk a JAOO a few years back called The Art n Programming. Since then, I’ve been giving variants around the world, talking about the ways in which developers can learn from artists (and, along the way, how we should avoid the false dichotomy of art vs. engineering).




David Troy came up to me after and said that he’d heard that the Museum of Modern Art in New York might be featuring two of his Rails applications in an exhibition.


Now it’s officialTwittervision and Flickrvision will be on display next year.


In his blog announcement, David says:


Today, creativity and imagination (what some folks are calling the right brain) are becoming the key drivers of software and design. With imagination, we can see around the corners of today’s most pressing challenges. While technical skill is certainly valuable, if it’s applied to the wrong problems, it’s wasted effort.

Creativity, imagination, and artistry help us identify the areas where we should put our efforts. They help us see things in new ways.




Hear hear! I like to think of it as bringing balance back to a process that for too long has been seen as some poor bastard child of mathematics and engineering.


Software development is neither. Nor is it art. It’s just software development. People who look for the “software is like xxx" analogies are missing the point. Software develpment is like software development. Let’s decide what works for us, and have fun while doing it.



Wednesday, August 29, 2007

Superators: neat Ruby hack

imageImage stolen from jicksta.com

So, I’m not sure if Jay Phillips (of Adhearsion fame) is trying to annoy the purists with a remarkably Perlish hack in Ruby. I just know that his superators gem is a clever bit of pure Ruby code that allows you to apparently define new infix operators in Ruby programs. Imagine the joy of being able to write


a /~ b

and


rocket <=- fuel

You can with superators:



require 'rubygems'
require 'superators'

class String
superator "<=-" do |arg|
self << arg.upcase
end
end

Is this a good idea? I don’t know. I can see that it would have been useful in some DSLs that I’ve written. But I do think that it’s a wonderfully clever implementation (it doesn’t do anything nasty to the interpreter–it simply intercepts built-in operators (such as <<= and -) in my example above. And that’s what makes it great. Every now and then, just when you think you know Ruby. someone comes up with something—such as symbol.to_proc or superators—that makes you go “oh!.”


Every long-lasting relationship needs the occasional surprise to keep it interesting and fun. This kind of hack is one of the reasons I love Ruby.

Sunday, August 12, 2007

Possibly my favorite Rake target

rake gerbils:deploy


Cry “Havoc!” and let slip the rodents of PDF generation.

Saturday, June 23, 2007





Riding the elevator up to my room in a Tokyo hotel, I glanced at a poster on the side wall and had to do a double take. Is it just me, or does this bear an uncanny resemblance to someone we know…?

Monday, May 21, 2007

Paying Back

$33,000


That’s how much was raised in charitable donations during the course of this weekend’s RailsConf.


It started with the Rails Guidebook. This was a pre-conference all-day tutorial intended to bring people up to speed in Rails, and to point them towards interesting talks. We waived the fee, and asked for a minimum donation of $75 from each attendee. The conference organizers graciously donated the room and AV support. That event raised $12,000.


Then Chad Fowler took it to the next level. During his conference introduction the next day, he threw down the gauntlet. As a result, attendees continued to give.


The Challenge


So the Rails Community has shown what’s possible. Let’s not let it stop there. Everyone in our industry is privileged; we’re all way better off than the folks being helped by these charities. So let’s take this up to the next level. Let’s see if we can make all industry conferences into fund raising events. It’s not hard:



  • If you run a conference, set up a charitable giving site and advertise it in the program and during the event. (The Pragmatic Studio used JustGive). Sell off extra T-Shirts (the way O’Reilly did during RailsConf) and donate the proceeds.


    Consider running a pre-conference Guidebook session. Attendees find them really useful, they have a small marginal cost, and they raise a lot of money. Your speakers will be keen to help out.




  • If you are speaking at a conference, consider donating your fee to charity. Invite attendees to contribute too. If the conference isn’t planning a charitable tie-in, push them to experiment with one. The overhead is low, and the payback is great.




  • If you’re attending a conference that isn’t offering a charitable tie-in, ask them why not. Point them to this blog post and to Mike Clark’s description of RailsConf. Tell the organizers that they can contact me if they’d like more details.



Just imagine the difference we could make if, as an industry, we turned each of these conferences into a chance to raise much needed money for worthy charities. Imagine if, rather than getting yet one more burlap bag with a sponsor’s name on it, you instead got a slip of paper saying that the price of that bag was being used to buy vaccine for 5 kids, or a book for a literacy project. Imagine what could happen if a conference with 5,000 attendees raised just $20 per attendee. Then imagine $50, or $100. It starts to get serious.


Let’s start paying back.

Thursday, April 19, 2007

Test-First Word Wrap in Erlang

image



I’m continuing to play with Erlang. This week, I needed to write some code that extracts information from a bunch of source files. Part of the output was supposed to be a list of strings, nicely word wrapped to fit on a terminal screen.


I decided to experiment with using a test-first approach to developing the word wrap function. I wrote unit tests that describe the functionality I want, and then wrote some code to make the tests run successfully.


To do that, I used the new EUnit-2.0 unit testing framework. It’s still under development, but it seems to work nicely (although I’d love for it to have an assert_equal method).


Installing EUnit


First, we’ll install EUnit from their Subversion repository:


  [dave/Erlang/common] svn co http://svn.process-one.net/contribs/trunk/eunit
A eunit/Emakefile
A eunit/sys.config
A eunit/include
: : :
[dave/Erlang/common] cd eunit
[Erlang/common/eunit] make
: : :

Now we need to add the EUnit library into your default Erlang path. Edit the file .erlang in your home directory (create it if it’s not there already) and add the line


  code:add_pathz("/Users/dave/Erlang/common/eunit/ebin").


(You’ll need to change the path to reflect the location where you downloaded EUnit. Remember to add the /ebin part at the end. And, yes, the “z” is really path of that line.)


EUnit In 60 Seconds


EUnit basically lets you run assertions against Erlang code. You can use it in a number of different modes. In our case we’ll include the tests in the same file that contains the code we’re testing. This is very convenient, but longer term it has a downside—anyone using your code will also need to have EUnit installed, even if they don’t want to run the tests. So, later we can split the tests into their own file.


The key to using EUnit is including the library in the source file containing the tests. That’s as simple as adding the line:


  -include_lib("eunit/include/eunit.hrl").

Any module that includes this line will automatically have a new function called test/0 defined. Thistest function will automatically run all the tests in the module.


So, how do you define tests? As with any xUnit framework, you can write tests by defining regular methods that follow a naming convention. With EUnit, the convention is that the function name should end _test. The EUnit convention is that any test function that returns is considered to have completed successfully; any function that throws an exception is a failure. It’s common to use pattern matching as a kind of assertion:


  length_test()  ->  3 = length("cat").
reverse_test() -> [3, 2, 1] = lists:reverse([1,2,3]).

However, for simple tests, it’s easier to write a test generator. This is a function whose name ends_test_ (with a final underscore). Test generators return a representation of the tests to be run. In our case, we’ll return a list of tests created using the EUnit _assert macro. We can rewrite our previous two test functions using a test generator called demo_test_:


  demo_test_() -> [
?_assert(3 == length("cat")),
?_assert([3, 2, 1] == lists:reverse([1,2,3]))
].

Here the generator returns an array containing two tests. The tests are written using the _assertmacro. This takes a boolean expression (rather than a pattern) which is why we’re now using two equals signs.


Word Wrap


My application needs a library function which takes an array of strings and joins them together to form an array of lines where no line is longer than a given length. We’ll write a function wrap/1 in a module called text. Let’s start with a basic test. If you wrap no words, you get a single empty line.


  -module(text).
-export([wrap/1]).

-include_lib("eunit/include/eunit.hrl").


wrap_test_() -> [
?_assert(wrap([]) == [""])
].

This won’t compile: the wrap/1 function hasn’t been defined. Getting it to pass this single test isn’t hard:


  wrap([]) ->
[ "" ].

We can run this in the Erlang shell:


  1> c(text).
{ok,text}
2> text:test().
Test successful.
ok

Let’s add the next test. Wrapping a single short word should result in a single line containing that word.


  wrap_test_() -> [
?_assert(wrap([]) == [""]),
?_assert(wrap(["cat"]) == ["cat"])
].

The tests now fail when we run them: our wrap function doesn’t yet know how to wrap strings:


  1> c(text).
{ok,text}
2> text:test().
text:9:wrap_test_...*failed*
::{error,function_clause,
[{text,wrap,[["cat"]]},{text,'-wrap_test_/0-fun-2-',0}]}

=======================================================
Failed: 1. Aborted: 0. Skipped: 0. Succeeded: 1. error

Let’s see if we can fix this. We’ll add a second version of the wrap/1 function that knows how to wrap a single string. Do do this, we’ll use a key feature of Erlang.


Pattern Matching and Function Definitions


In the same way that Erlang uses pattern matching when evaluating the = operator, it also uses pattern matching when invoking functions. This is often illustrated with an inefficient implementation of a function to calculate the nth term in the Fibonacci sequence (the sequence that goes 0, 1, 1, 2, 3, 5, …, where each term is the sum of the preceding two terms).


  -module(fib).
-export([fib/1]).

fib(0) -> 1;
fib(1) -> 1;
fib(N) -> fib(N-1) + fib(N-2).

Here we have three definitions of the same function (note how they’re separated by semicolons, and terminated with a period). The first two match when the function is called with a parameter or zero or one. The third is called with any other parameter.


We can use this to define two versions of our wrap/1 function, one for the case where it is called with no words in the list, the other when we have a single string.


  wrap([]) ->
[ "" ];

wrap([String]) ->
[ String ].

The tests now pass.


  9> text:test().
All 2 tests successful.

Add a test which wraps two strings, though, and our tests again fail:


  wrap_test_() -> [
?_assert(wrap([]) == [""]),
?_assert(wrap(["cat"]) == ["cat"]),
?_assert(wrap(["cat", "dog"]) == ["cat dog"])
].

  text:10:wrap_test_...*failed*
::{error,function_clause,
[{text,wrap,[["cat","dog"]]},{text,'-wrap_test_/0-fun-4-',0}]}

We’re going to have to do some work. For a start, we’re going to have to rewrite our wrap methods to give them somewhere to store the result. Then they’re going to have to recurse over the input parameter list, adding each word to the output until the input is exhausted.


Building an Output List


We’re going to do two things here. The externally visible function, wrap/1, is simply going to invoke an internal function, wrap/2, passing it the word list and an addition parameter. This second parameter is the list that holds the results. Remember that in Erlang the number of arguments is part of a function’s signature, so wrap/1 and wrap/2 are separate functions.


Then we’re going to use our parameter matching trick to define two variants of wrap/2:


  % This is the exported function: it passes the initial
% result set to the internal versions
wrap(Words) ->
wrap(Words, [""]).

% When we run out of words, we're done
wrap([], Result) ->
Result;

% Otherwise append the next word to the result
wrap([NextWord|Rest], Result) ->
wrap(Rest, [ Result ++ NextWord]).

The second version of wrap/2 uses a neat feature of Erlang list notation. The pattern [NextWord|Rest]matches a list whose head is matched with NextWord and whose tail is matched with Rest. If we invoked this function with:


  wrap([ "cat", "dog", "elk" ], Result).

then NextWord would be set to "cat" and Rest would be set to ["dog", "elk"].


Unfortunately, this fails our test:


  1> c(text).    
{ok,text}
2> text:test().
text:10:wrap_test_...*failed*
::{error,{assertion_failed,[{module,text},
{line,10},
{expression,
"wrap ( [ \"cat\" , \"dog\" ] ) == [ \"cat dog\" ]"},
{expected,true},
{value,false}]},
[{text,'-wrap_test_/0-fun-4-',0}]}

=======================================================
Failed: 1. Aborted: 0. Skipped: 0. Succeeded: 2.

This is where I wish EUnit had an assert_equals function that could report the actual and expected values. However, I can still invoke my wrap method from the shell to see what it’s returning. (Stop press: EUnit does have the equivalent, and I didn’t notice it. It’s called assertMatch(Expected, Actual). Sorry.)


  3> text:wrap(["cat", "dog"]).
["catdog"]

Oops. We forgot the space between words. We need to add this when we add a word to an existing line, but only if that line is not empty. See how we use pattern matching to distinguish a list containing an empty line from one containing a non-empty one.


  % This is the exported function: it passes the initial
% result set to the internal versions
wrap(Words) ->
wrap(Words, [""]).

% When we run out of words, we're done
wrap([], Result) ->
Result;

% Adding a word to an empty line
wrap([NextWord|Rest], [ "" ]) ->
wrap(Rest, [ NextWord]);

% or to a line that's already partially full
wrap([NextWord|Rest], [ CurrentLine ]) ->
wrap(Rest, [ CurrentLine ++ " " ++ NextWord]).

Now our tests pass again:


  1> c(text).
{ok,text}
2> text:test().
All 3 tests successful.
ok

When Clauses


For testing purposes, let’s assume that we wrap lines longer than 10 characters. That means that if we give our method the strings “cat”, “dog”, and “elk”, we’ll expect to see two lines in the output (because “cat<space>dog<space>elk” is 11 characters long). Time for another test.


  wrap_test_() -> [
?_assert(wrap([]) == [""]),
?_assert(wrap(["cat"]) == ["cat"]),
?_assert(wrap(["cat", "dog"]) == ["cat dog"]),
?_assert(wrap(["cat", "dog", "elk"]) == ["cat dog", "elk"])
].

We don’t even have to run this to know it will fail: our wrap function knows nothing about line lengths. Time for some code. We’re now going to have to treat out result as a list or strings, rather than a single string. We’re also going to have to deal with the case where there’s not enough room in the current output line for the next word. In that case we have to add a new empty line to the function’s result and put the word into it.


  wrap(Words) ->
wrap(Words, [""]).

% When we run out of words, we're done
wrap([], Result) ->
Result;

% Adding a word to an empty line
wrap([NextWord|Rest], [ "" | PreviousLines ]) ->
wrap(Rest, [ NextWord | PreviousLines ]);

% or to a line that's already partially full
% there are two cases:
% 1. The word fits
wrap([NextWord|Rest], [ CurrentLine | PreviousLines ])
when length(NextWord) + length(CurrentLine) < 10 ->
wrap(Rest, [ CurrentLine ++ " " ++ NextWord | PreviousLines ]);

% 2. The word doesn't fit, so we create a new line
wrap([NextWord|Rest], [ CurrentLine | PreviousLines ]) ->
wrap(Rest, [ NextWord, CurrentLine | PreviousLines ]).


This introduces a new Erlang feature—you can further qualify the pattern
matching used to determine which function is selected using a when
clause. In our case, the parameter patterns for the last two definitions of the
wrap/2 method are the same. However, the first of them has a
whenclause

. This means that this function will only be selected if the length of the current output line plus the length of the next word is less that our maximum line length (hard coded to 10 in this case).


Unfortunately, our tests fail:


1> text:test().              
text:11:wrap_test_...*failed*
::{error,{assertion_failed,[{module,text},
{line,11},
{expression,
"wrap ( [ \"cat\" , \"dog\" , \"elk\" ] ) == [ \"cat dog\" , \"elk\" ]"},
{expected,true},
{value,false}]},
[{text,'-wrap_test_/0-fun-6-',0}]}

=======================================================
Failed: 1. Aborted: 0. Skipped: 0. Succeeded: 3.
error

Running text:wrap manually shows why:


2> text:wrap(["cat", "dog", "elk"]).
["elk","cat dog"]

We’re building the result list backwards, adding each successive line to the front of it, rather than the back. We could fix that by changing the way we add lines to the list, but, like Lisp, Erlang favors list manipulations that work on the head of the list, not the last element. It turns out to be both idiomatic and more efficient to build the result the way we’re doing it, and then to result the resulting list before returning it to our caller. That’s a simple change to our exported wrap/1 function.


  wrap(Words) ->
lists:reverse(wrap(Words, [""])).

Now all out tests pass.


So, let’s review our tests. We have covered an empty input set, a set that fits on one line, and a set that (just) forces the result to take more than one line. Are there any other cases to consider? It turns out that there’s (at least) one. What happens if a word is too long to fit on a line on its own? Let’s see:


  wrap_test_() -> [
?_assert(wrap([]) == [""]),
?_assert(wrap(["cat"]) == ["cat"]),
?_assert(wrap(["cat", "dog"]) == ["cat dog"]),
?_assert(wrap(["cat", "dog", "elk"]) == ["cat dog", "elk"]),
?_assert(wrap(["cat", "dog", "hummingbird", "ibix"]) == ["cat dog", "hummingbird", "ibix"])
].

Run the tests:


1> c(text).                         
{ok,text}
2> text:test().
All 5 tests successful.
ok

Wrapping Up


Using simple tests like this are a great way of designing some code. They’re also a goo way to teach yourself the language. As I was writing this code, I used the tests to test my understanding of Erlang semantics.


Update: Kevin Smith has produced a wonderful screencast on EUnit as episode 5 of his Erlang by Example series.


The Final Program


-module(text).
-export([wrap/1]).

-include_lib("eunit/include/eunit.hrl").

wrap_test_() -> [
?_assert(wrap([]) == [""]),
?_assert(wrap(["cat"]) == ["cat"]),
?_assert(wrap(["cat", "dog"]) == ["cat dog"]),
?_assert(wrap(["cat", "dog", "elk"]) == ["cat dog", "elk"]),
?_assert(wrap(["cat", "dog", "hummingbird", "ibix"]) == ["cat dog", "hummingbird", "ibix"])
].

% This is the exported function: it passes the initial
% result set to the internal versions
wrap(Words) ->
lists:reverse(wrap(Words, [""])).

% When we run out of words, we're done
wrap([], Result) ->
Result;

% Adding a word to an empty line
wrap([ NextWord | Rest ], [ "" | PreviousLines ]) ->
wrap(Rest, [ NextWord | PreviousLines ]);

% Or to a line that's already partially full. There are two cases:
% 1. The word fits
wrap([ NextWord | Rest ], [ CurrentLine | PreviousLines ])
when length(NextWord) + length(CurrentLine) < 10 ->
wrap(Rest, [ CurrentLine ++ " " ++ NextWord | PreviousLines ]);

% 2. The word doesn't fit, so we create a new line
wrap([ NextWord | Rest], [ CurrentLine | PreviousLines ]) ->
wrap(Rest, [ NextWord, CurrentLine | PreviousLines ]).

Monday, April 16, 2007

Adding Concurrency to Our Erlang Program

image



In our last thrilling installment, we used Erlang to fetch a book’s title and sales rank from Amazon. Now let’s extend this to fetch the data for multiple books, first one-at-a-time, and then in parallel.


A word from our lawyers: read your Amazon Terms of Service before trying this code. You may have limitations on the number of requests you can send per second or somesuch. This code is just to illustrate some Erlang constructs.

Let’s start with the function that fetches sales ranks in series. We’ll pass it a list of ISBNs, and it will return a corresponding list of {title, rank} tuples. For convenience, let’s define a function that returns a list of ISBNS to check. (Later, we can change this function to read from a database or a file). We’re editing our file ranks.erl.


  isbns() ->
[ "020161622X", "0974514004", "0974514012", "0974514020", "0974514039",
"0974514055", "0974514063", "0974514071", "097669400X", "0974514047",
"0976694018", "0976694026", "0976694042", "0976694085", "097451408X",
"0976694077", "0977616606", "0976694093", "0977616665", "0976694069",
"0976694050", "0977616649", "0977616657"
].

Users of our code call the fetch_in_series function. It uses the built-in lists:map function to convert our ISBN list into the result list.


  fetch_in_series() ->
lists:map(fun fetch_title_and_rank/1, isbns()).

The first parameter to lists:map is the function to be applied to each element in the ISBN list. Here we’re telling Erlang to call the fetch_title_and_rank function that we defined in the first article. The/1 says to use the version of the function that takes a single argument.


We need to export this function from the module before we can call it.


  -export([fetch_title_and_rank/1, fetch_in_series/0]).

Let’s fire up the Erlang shell and try it.


  1> c(ranks).
{ok,ranks}
2> ranks:fetch_in_series().
[{"The Pragmatic Programmer: From Journeyman to Master","4864"},
{"Pragmatic Version Control Using CVS","288118"},
{"Pragmatic Unit Testing in Java with JUnit","116011"},
. . .

“But wait!” I hear you cry. “Isn’t Erlang supposed to be good at parallel programming. What’s with all this fetch-the-results-in-series” business?”


OK, let’s create a parallel version. We have lots of options here, but for now let’s do it the hard way. We’ll spawn a separate (Erlang) process to handle each request, and we’ll write all our own process management code to coordinate harvesting the results as these processes complete.


Erlang processes are lightweight abstractions; they are not the same as the processes your operating system provides. Unlike your operating system, Erlang is happy dealing with thousands of concurrent processes. What’s more, you can distribute these processes across servers, processors, and cores within a processor. All this is achieved with a handful of basic operations. To send a tuple to a process whose process ID is in the variable Fred, you can use:


  Fred  ! {status, ok}

To receive a message from another process, use a receive stanza:


  receive 
{ status, StatusCode } -> StatusCode
end

There’s something cool about the receive code. See the line


    { status, StatusCode } -> StatusCode

The stuff on the left hand side of the arrow is a pattern match. It means that this receive code will only accept messages which are a two element tuple where the first element is the atom status. This particular receive stanza then strips out just the actual code part of this tuple and returns it. In general a receive stanza can contain multiple patterns, each with its own specialized processing. This is remarkably powerful.


There’s one last primitive we need. The function spawn takes a function and a set of parameters and invokes that function in a new Erlang process. It returns the process ID for that subprocess.


Let’s write a simple, and stupid, example. This module exports test/1. This function spawns a new process that simply doubles a value. Here’s the code—we’ll dig into it in a second.


  -module(parallel).
-export([test/1]).

test(Number) ->
MyPID = self(),
spawn(fun() -> double(MyPID, Number) end),
receive
{ answer, Val } -> { "Child process said", Val }
end.

double(Parent, Number) ->
Result = Number + Number,
Parent ! { answer, Result }.

Because we’re handling all the details ourselves, we have to tell the process running the double function where to send its result. To do that, we pass it two parameters: the first is the process ID of the process to receive the result, and the second is the value to double. The second line of the double function uses the ! operator to send the result back to the original process.


The original process has to pass its own process ID to the double method. It uses the built-in self()function to get this PID. Then, on the second line of the function, it invokes spawn, passing it an anonymous function which in turn invokes double. There’s a wee catch here: it’s tempting to write:


  test(Number) ->
spawn(fun() -> double(self(), Number) end),
...

This won’t work: because the anonymous function only gets executed once the new process is started, the value returned by self() will be that process’s ID, and not that of the parent.


Anyway, we can invoke this code using the Erlang shell:


  1> c(parallel).
{ok,parallel}
2> parallel:test(22).
{"Child process said",44}

Back to Amazon. We want to fetch all the sales ranks in parallel. We’ll start with the top-level function. It starts all the fetcher processes running in parallel, then gathers up their results.


  fetch_in_parallel() ->
inets:start(),
set_queries_running(isbns()),
gather_results(isbns()).

The first line of this method is a slight optimization. The HTTP client code that we use to fetch the results actually runs behind the scenes in its own process. By calling the inets:start method, we precreate this server process.


Here’s the code that creates the processes to fetch the sales data:


  set_queries_running(ISBNS) ->
lists:foreach(fun background_fetch/1, ISBNS).

background_fetch(ISBN) ->
ParentPID = self(),
spawn(fun() ->
ParentPID ! { ok, fetch_title_and_rank(ISBN) }
end).

The lists:foreach function invokes its first argument on each element of its second argument. In this case, it invokes the background_fetch function that in term calls spawn to start the subprocess. Perhaps surprising is the fact that the anonymous function acts as a closure: the values of ParentID andISBN in its calling scope are available inside the fun’s body, even though it is running is a separate process. Cool stuff, eh?


There’s an implicit decision in the design of this code: I decided that I don’t care what order the results are listed in the returned list—I simply want a list that contains one title/rank tuple for each ISBN. I can make use of this fact in the function that gathers the results. It again uses lists:map, but bends its meaning somewhat. Normally the map function maps an value onto some other corresponding value. Here we’re simply using it to populate a list with the same number of entries as the original ISBN list. Each entry contains a result from Amazon, but it won’t be the result that corresponds to the ISBN that is in the corresponding position in the ISBN list. For my purposes, this is fine.


  gather_results(ISBNS) ->      
lists:map(fun gather_a_result/1, ISBNS).

gather_a_result(_) ->
receive
{ ok, Anything } -> Anything
end.

Let’s run this:


  1> c(ranks).                              
{ok,ranks}
2> ranks:fetch_in_parallel().
[{"The Pragmatic Programmer: From Journeyman to Master","6359"},
{"Pragmatic Version Control Using CVS","299260"},
{"Pragmatic Unit Testing in Java with JUnit","131616"},
. . .

What kind of speedup does this give us? We can use the built-in timer:tc function to call our two methods.


 timer:tc(ranks, fetch_in_parallel, []).
{1163694, . . .
timer:tc(ranks, fetch_in_series, []).
{3070261, . . .

The first element of the result in the wallclock time taken to run the function (in microseconds). The parallel version is roughly 3 times faster than the serial function.


So, do you have to go to all this trouble to make your Erlang code run in parallel? Not really. I just wanted to show some of the nuts and bolts. In reality, you’d probably use a library method, such as thepmap function that Joe wrote for the Programming Erlang book. Armed with this, you could turn the serial fetch program to a parallel fetch program by changing


  lists:map(fun fetch_title_and_rank/1, ?ISBNS).

to


  lib_misc:pmap(fun fetch_title_and_rank/1, ?ISBNS).


Now that’s an abstraction!


Anyway, here’s the current version of the ranks.erl program, containing both the serial and parallel fetch functions.


  -module(ranks).
-export([fetch_title_and_rank/1, fetch_in_series/0, fetch_in_parallel/0]).
-include_lib("xmerl/include/xmerl.hrl").

-define(BASE_URL,
"http://webservices.amazon.com/onca/xml?Service=AWSECommerceService"
"&SubscriptionId=<your ID goes here>"
"&Operation=ItemLookup"
"&ResponseGroup=SalesRank,Small"
"&ItemId=").

isbns() ->
[ "020161622X", "0974514004", "0974514012", "0974514020", "0974514039",
"0974514055", "0974514063", "0974514071", "097669400X", "0974514047",
"0976694018", "0976694026", "0976694042", "0976694085", "097451408X",
"0976694077", "0977616606", "0976694093", "0977616665", "0976694069",
"0976694050", "0977616649", "0977616657"
].

fetch_title_and_rank(ISBN) ->
URL = amazon_url_for(ISBN),
{ ok, {_Status, _Headers, Body }} = http:request(URL),
{ Xml, _Rest } = xmerl_scan:string(Body),
[ #xmlText{value=Rank} ] = xmerl_xpath:string("//SalesRank/text()", Xml),
[ #xmlText{value=Title} ] = xmerl_xpath:string("//Title/text()", Xml),
{ Title, Rank }.

amazon_url_for(ISBN) ->
?BASE_URL ++ ISBN.


fetch_in_series() ->
lists:map(fun fetch_title_and_rank/1, isbns()).


fetch_in_parallel() ->
inets:start(),
set_queries_running(isbns()),
gather_results(isbns()).

set_queries_running(ISBNS) ->
lists:foreach(fun background_fetch/1, ISBNS).

background_fetch(ISBN) ->
ParentPID = self(),
spawn(fun() ->
ParentPID ! { ok, fetch_title_and_rank(ISBN) }
end).

gather_results(ISBNS) ->
lists:map(fun(_) ->
receive
{ ok, Anything } -> Anything
end
end, ISBNS).

Sunday, April 15, 2007

A First Erlang Program

image


One of the joys of playing at being a publisher is that I get to mess around with the technology in books as those books are getting written. Lately I’ve been having a blast with Joe Armstrong's new Erlang Book. At some point I’ll blog about the really neat way the Erlang database unifies the set-based SQL query language and list comprehensions (it’s obvious when you think about it, but it blew me away when I first read it). But I just wanted to start off with some simple stuff.


One of my standard Ruby examples is a program that fetches our book sales ranks from Amazon using their REST interface. I thought I’d try the exercise again in Erlang. In this first part we’ll write a simple Erlang function that uses the Amazon Web Services API to fetch the data on a book (identified by its ISBN). This data is returned in XML, so we’ll then use some simple XPath to extract the title and the sales rank.


My Erlang module is called ranks, and I store it in the fileranks.erl. Because I’m just playing for now, the interface to this module is simple: I’ll define a function calledfetch_title_and_rank. This returns a {title, rank} tuple. So, I started off with something like this:


  -module(ranks).
-export([fetch_title_and_rank/1]).

fetch_title_and_rank(ISBN) ->
{ "Agile Web Development with Rails", 1234 }.

The -module directive simply names the module. The next line tells Erlang that this module exports a function called fetch_title_and_rank. This method takes one parameter. (This idea of specifying the number of parameters seems strange until you realize that in Erlang the function signature is the function name and the parameter count. It is valid to export two functions with the same name if they take a different number of arguments—as far as Erlang is concerned they are different functions.)


Next we define the fetch_title_and_rank function. It takes a single parameter, the book’s ISBN. In Erlang, variable names (including parameters) start with uppercase letters. This takes a little getting used to. Just to make sure thinks are wired up correctly, let’s try running this code. In a command shell, I navigate to the directory containing ranks.erl and started the interactive Erlang shell. Once inside it, I compiled my module (using c(ranks).) and then invoke the fetch_title_and_rank method.


  dave[~/tmp 11:51:09] erl
Erlang (BEAM) emulator version 5.5.4 [source] ...

Eshell V5.5.4 (abort with ^G)
1> c(ranks).
./ranks.erl:11: Warning: variable 'ISBN' is unused
{ok,ranks}
2> ranks:fetch_title_and_rank("0977616630").
{"Agile Web Development with Rails",1234}

Let’s start wiring this up to Amazon.


To fetch information from Amazon using REST, you need to issue an ItemLookup operation, specifying what you want Amazon to return. In our case, we want the sales rank information, plus something called the small response group. The latter includes the book’s title. The URL you use to get this looks like this (except all on one line, and with no spaces):


http://webservices.amazon.com/onca/xml?
Service=AWSECommerceService
&SubscriptionId=<your ID goes here>
&Operation=ItemLookup
&ResponseGroup=SalesRank,Small
&ItemId=0977616630

The ItemID parameter is the ISBN of our book. Let’s write an Erlang function that returns the URL to fetch the information for an ISBN.


  -define(BASE_URL,
"http://webservices.amazon.com/onca/xml?" ++
"Service=AWSECommerceService" ++
"&SubscriptionId=<" ++
"&Operation=ItemLookup" ++
"&ResponseGroup=SalesRank,Small" ++
"&ItemId=").

amazon_url_for(ISBN) ->
?BASE_URL ++ ISBN.

This code defines an Erlang macro called BASE_URL which holds the static part of the URL. The functionamazon_url_for builds the full URL by appending the ISBN to this. Note that both the macro and the function use ++, the operator that concatenates strings.


We now need to find a way of sending this request to Amazon and fetching back the XML response. Here we bump into one of Erlang’s current problems—it can be hard to browse the library documentation. My solution is to download the documentation onto my local box and search it there. I tend to use both the HTML and the man pages. Figuring I wanted something HTTP related, I tried the following:


  dave[Downloads/lib 12:04:07] ls **/*http*
inets-4.7.11/doc/html/http.html
inets-4.7.11/doc/html/http_base_64.html
inets-4.7.11/doc/html/http_client.html
inets-4.7.11/doc/html/http_server.html
inets-4.7.11/doc/html/httpd.html
inets-4.7.11/doc/html/httpd_conf.html
inets-4.7.11/doc/html/httpd_socket.html
inets-4.7.11/doc/html/httpd_util.html

Opening up http.html revealed the http:request method.



request(Url) -> {ok, Result} | {error, Reason}


This says that we give the request method a URL string. It returns a tuple. If the first element of the tuple is the atom ok, then the second element is the result for the request. If the first element is error, the second element is the reason it failed. This technique of returning both status and data as a tuple is idiomatic in Erlang, and the language makes it easy to handle the different cases.


Let’s look a little deeper into the successful case. The Result element is actually itself a tuple containing a status, the response headers, and the response body.


So let’s take all this and develop our fetch_title_and_rank method a little more.


  fetch_title_and_rank(ISBN) ->
URL = amazon_url_for(ISBN),
{ ok, {Status, Headers, Body }} = http:request(URL),
Body.

We call our amazon_url_for method to create the URL to fetch the data, then invoke the http:requestlibrary method to fetch the result. Let’s look at that second line in more detail.


The equals sign in Erlang looks like an assignment statement, but looks are deceptive. In Erlang, the equals sign is actually a pattern matching operation—an assertion that two things are equal. For example, it is perfectly valid in Erlang to say


  6> 1 = 1.
1
7> 9 * 7 = 60 + 3.
63

So what happens when variables get involved? This is where it gets interesting. When I say


  A = 1.

I’m not assigning 1 to the variable A. Instead, I’m matching A against 1 (just like 9*7 = 60+3 asserts that the results of the two expressions are the same). So, does A match 1? That depends. If this is the first appearance of A on the left hand side of an equals sign, then A is said to be unbound. In this condition, Erlang says to itself “I can make this statement true if I give A the value 1,” which is does. From this point forward, A has the value 1 (in the current scope). If we say A = 1. again in the same scope in our Erlang program, A is no longer unbound, but this pattern match succeeds, because A = 1.is a simple assertion of truth. But what happens if we say A=2.?


  10> A = 1.
1
11> A = 1.
1
12> A = 2.

=ERROR REPORT==== 16-Apr-2007::12:32:36 ===
Error in process <0.55.0> with exit value: {{badmatch,2},[{erl_eval,expr,3}]}

** exited: {{badmatch,2},[{erl_eval,expr,3}]} **

We get an error: Erlang was unable to match the values on the left and right hand sides of the equals, so it raised a badmatch error. (Erlang error reporting is, at best, obscure.)


So, back to our HTTP request. I wrote


  { ok, {Status, Headers, Body }} = http:request(URL)

The left hand side matches a tuple. The first element of the tuple is the atom ok. The second element is another tuple. Thus the entire expression only succeeds if http:request returns a two element tuple with ok as the first element and a three element tuple as the second element. If the match succeeds, then the variables StatusHeaders, and Body are set to the three elements of that second tuple.


(An aside for Ruby folk: you can do something similar using an obscure Ruby syntax. The statement


  rc, (status, headers, body) = [ 200, [ "status", "headers", "body" ] ]

will leave headers containing the string "headers".)


In our Erlang example, what happens if http:request returns an error? In this case, the first element of the returned tuple will be the atom error. This won’t match the left hand side of our expression (because the atom error is not the same as the atom ok) and we’ll get a badmatch error. We won’t bother to handle this here—it’s someone else’s problem.


Let’s compile our program.


  1> c(ranks).
./ranks.erl:13: Warning: variable 'Headers' is unused
./ranks.erl:13: Warning: variable 'Status' is unused
{ok,ranks}

Erlang warns us that we have some unused variables. Do we care? To a point. I personally like my compilations to be clean, so let’s fix this. If you prefix a variable’s name with an underscore, it tells the Erlang compiler that you don’t intend to use that variable. As a degenerate case, you can use just a lone underscore, which means “some anonymous variable.” So, we can fix our warnings by writing either


  { ok, {_Status, _Headers, Body }} = http:request(URL),

or


  { ok, {_, _, Body }} = http:request(URL),

I mildly prefer the first form, as it helps me remember what those elements in the tuple are when I reread the program later.


Now we can recompile and run our code:


  1> c(ranks).
{ok,ranks}
2> ranks:fetch_title_and_rank("0977616630").
"<?xml version=\"1.0\" encoding=\"UTF-8\"?><ItemLookupResponse
xmlns=\"http://webservices.amazon.com/AWSECommerceService/2005-10-05\">
<OperationRequest><HTTPHeaders><Header ...
... <SalesRank>1098</SalesRank>
... <Title>Agile Web Development with Rails (Pragmatic
Programmers)</Title>...
...

Cool! We get a boatload of XML back. (If you’re running this at home, you’ll need to substitute your own AWS subscription ID into the request string to make this work.) Now we need to extract out the title and the sales rank. We could to this using string operations, but wouldn’t it be more fun to use XPath? Another quick filesystem scan reveals the xmerl library, which both generates and parses XML. It also supports XPath queries. To use this, we first scan the Amazon response. This constructs an Erlang data structure representing the original XML. We then call the xmerl_xpath library to find the elements in this data structure matching our query.


  -include_lib("xmerl/include/xmerl.hrl").

fetch_title_and_rank(ISBN) ->
URL = amazon_url_for(ISBN),
{ ok, {_Status, _Headers, Body }} = http:request(URL),
{ Xml, _Rest } = xmerl_scan:string(Body),
[ #xmlText{value=Rank} ] = xmerl_xpath:string("//SalesRank/text()", Xml),
[ #xmlText{value=Title} ] = xmerl_xpath:string("//Title/text()", Xml),
{ Title, Rank }.

The -include line loads up the set of Erlang record definitions that xmerl uses to represent elements in the parsed XML. This isn’t strictly necessary, but doing so makes the XPath stuff a little cleaner.


The line


    { Xml, _Rest } = xmerl_scan:string(Body),

parses the XML response, storing the internal Erlang representation in the variable Xml.


Then we have the line


    [ #xmlText{value=Rank} ]  = xmerl_xpath:string("//SalesRank/text()", Xml),

The notation #xmlText identifies an Erlang record. A record is basically a tuple where each element is given a name. In this case the tuple (defined in that .hrl file we included) is called xmlText. It represents a text node in the XML. The xmlText record has a field called value. We use pattern matching to assign whatever is in that field to the variable Rank. But there’s one more twist. Did you notice that we wrote square brackets around the left hand side of the pattern match? Thats because XPath queries return an array of results. By writing the left hand side the way we did, we force Erlang to check that an array containing just one element was returned, and then to check that this element was a record of type xmlText. That’s pretty cool stuff—pattern matching extends down deep into the bowels of Erlang.


The last line of the method simply returns a tuple containing the title and the sales rank.


We can call it from the command line:


  1> c(ranks).
{ok,ranks}
2> ranks:fetch_title_and_rank("0977616630").
{"Agile Web Development with Rails (Pragmatic Programmers)","1098"}

That’s it for now. Next we’ll look at fetching the data for multiple books at once. In the meantime, here’s the full listing of our ranks.erl file.


  -module(ranks).
-export([fetch_title_and_rank/1]).
-include_lib("xmerl/include/xmerl.hrl").

-define(BASE_URL,
"http://webservices.amazon.com/onca/xml?" ++
"Service=AWSECommerceService" ++
"&SubscriptionId=<your id>" ++
"&Operation=ItemLookup" ++
"&ResponseGroup=SalesRank,Small" ++
"&ItemId=").


fetch_title_and_rank(ISBN) ->
URL = amazon_url_for(ISBN),
{ ok, {_Status, _Headers, Body }} = http:request(URL),
{ Xml, _Rest } = xmerl_scan:string(Body),
[ #xmlText{value=Rank} ] = xmerl_xpath:string("//SalesRank/text()", Xml),
[ #xmlText{value=Title} ] = xmerl_xpath:string("//Title/text()", Xml),
{ Title, Rank }.

amazon_url_for(ISBN) ->
?BASE_URL ++ ISBN.

Wednesday, March 28, 2007
















The coolest app you never knew was on your hard drive


If you use OS X, then you own an application that’ll waste hours of your time just sitting on your hard drive.


Say hello to Applications > Utilities > Grapher.


We needed a simple Fourier square wave for a book. Just type in the equations, and out it pops.


But the time wasting comes from playing with its example graphs.


Did I mention that they animate, and it can export Quicktime?

The RADAR Architecture: RESTful Application, Dumb-Ass Recipient

It Starts With Verbs and Nouns


Roy Fielding’s REST work showed us that the design of the interaction between a client and a server could make a major difference to the reliability and scalability of that application. If you follow a set of conventions embodied in REST when designing your application, is means that (for example) the network will be able to cache responses to certain requests, offloading work from both upstream network components and from the application.


But REST was about more than performance. It offered a way of separating the actions performed by your applications (the verbs) from the things acted upon (the nouns, or in REST terms, the resources). What’s more, REST said that the verbs that you use should apply consistently across all your resources.


When using HTTP, the HTTP verbs (such as GET, PUT, POST, and DELETE) are also the verbs that should be used with REST applications. This makes a lot of sense, because things like routers and proxies also understand these verbs.


One of the successes built using these principles is the Atom Publishing Protocol. Atom allows clients to fetch, edit, and publish web resources. Atom is probably most commonly applied to allow people to use local editors to create and maintain blog articles—client software on the user’s computer uses Atom to fetch existing articles and to save back edited or new articles.


Let’s look at how that happens with Atom.


I fire up my Atom-based local client. It needs to fetch the updated list of all the articles currently on the server, so it issues a GET request to a URL that represents the collection of articles: the verb is GET, and the resource is the collection.


   GET   /articles

What comes back is a list of resources. Included with each resource are one or two URIs. One of these URIs can be used to access that resource for editing. Let’s say I want to edit a particular article. First, my client software will fetch it, using the URI given in the collection:


   GET   /article/123

As a user of the editing software, I then see a nicely formatted version of the article. If I click the [EDIT]button, that view changes: I now see text boxes, drop down lists, text areas, and so on. I can now change the article’s contents. When I press [SAVE], the local application then packages up the changed article and sends it back to the server using a PUT request. The use of PUT says “update an existing resource with this new stuff.”


Browsers are Dumb


If you use a browser to interact with a server, you’re using a half-duplex, forms-based device. The host sends down a form, you fill in the form, and send it back when you’re ready. This turns out to be really convenient for people running networks and servers. Because HTTP is stateless, and because applications talk to browsers using a fire-and-forget model, your servers and networks can handle very large numbers of devices. This is exactly the architecture that IBM championed back in the 70s. The half-duplex, poll-select nature of 3270-based applications allowed mainframes with less processing power than a modern toaster to handle tens of thousands of simultaneous users. The browser is really not much better than a 3270 (except the resolution is better when displaying porn). (Recently, folks have been trying to circumvent this simplicity by making browser-based applications more interactive using technologies such as Ajax. To my mind, this is just a stop-gap until we throw the browser away altogether—Ajax is just lipstick on a pig.)


How well do browsers play in a RESTful world?


Well, for a start, they really only use two of the HTTP verbs: GET and POST. Is this a problem? Yes and no.


We can always hack around the lack of verbs by embedding the verb that you wanted to use somewhere in the request that you send to the server. It can be embedded in the URL that you use to access a resource or it can be embedded in the body of the request (for example as an element of the POST data). Both work, assuming the server understands the conventions. But both techniques also mean that you’re not using REST and the HTTP verb set; you’re simply using HTTP as a transport with your own protocol layered on top. And that means that the network is less able to do clever things to optimize your traffic.


But a browser talking HTML over HTTP is also lacking in another important area. It isn’t running any application functionality locally. Remember our Atom-based blogging client? It had smarts, and so it knew how to display an article for editing. Browsers don’t have that (without a whole lot of support in terms of bolted on Javascript or Flash). So, as a result, people using RESTful ideas to talk to browsers have to put the smarts back on the server. They invent new URLs which (for example) return a resource, but return it all wrapped up in the HTML needed to display it as a form for browser-based editing. This is how Rails does it. Using the Rails conventions, if I want to fetch an article for viewing on a browser, I can use


  GET   /article/1

If, instead, I want to allow the user to edit the resource, I issue


  GET  /article/1;edit

The application then sends down a form pre-populated with the data from that article. When I hit submit, the form contents are sent back to the server, along with a hidden field that tells the server to pretend that the HTTP POST request is actually a RESTful PUT request—the result is that the resource is updated on the server.


The ;xxx notation looks like it was tacked on. I think that in this case, looks aren’t deceiving—these modifiers really were something of an afterthought, added when it became apparent that a pure Atom-like protocol actually wouldn’t do everything needed in a real-world browser-based web application. That’s because the client of an Atom server is assumed to be more that just a display device. An Atom client would not say “fetch me this resource and (oh, by the way) send it down as a form so that I can edit it.” The client would just say GET, do what it had to do, then issue a PUT.


So, in pure REST, the client is a peer (or maybe even in charge) of the interchange. But when talking to a dumb browser, the client is no better than an intermediary between the human and the application. That’s why it was necessary to add things such as ;edit.


Client-independent Applications


One of the other benefits of designing your applications against the limited verbs offered by HTTP REST is that you can (in theory) support many different clients with the same application code. You could write blogging server that (for example) looked at the HTTP Accepts header when deciding what to send back to a client. The response to a GET request to /articles could be a nicely formatted HTML page if the client wants an HTML response, an XML payload if the client asks for XML, or an Atom collection if the client asks for application/atomcoll+xml. In Rails terms, this is handled by therespond_to stanza that is duplicated in each controller action of a RESTful server.


But the benefits run deeper than simply being able to serve up different styles of response.


The HTTP verb set maps (with a little URL tweaking) pretty nicely onto the database verb set. POST, GET, PUT, and DELETE get mapped to the database CRUD verbs: Create, Read, Update, and Delete.


So, in theory at least, you should be able to write your application code as a fairly thin veneer over a set of resources. The application exports the verbs to manipulate the resources and the clients have fun accessing them. This is similar to the vision of Naked Objects.


However, I personally think it was optimistic to try to treat the two styles of interaction (smart and dumb clients) using the same protocol. The ugliness of the ;xxx appendages was a hint.


I think there’s a lot of merit in following a CRUD-based model for interacting with your application’s resources. I’m not convinced all the hassle of bending dumb browser interactions into a REST-based set of verbs, and then extending those verbs to make it work, is worth the extra hassle. To put it another way, I believe the discipline and conventions imposed by thinking of your application interface as a RESTful one are worthwhile. They lead to cleaner and easier-to-understand designs. The hoops you have to jump through to implement it with a browser as a client seem to suggest that a slavish adherence to the protocol might be trying to push it too far.


An Alternative Model


Does that mean I’m down on the RESTful, CRUD based approach to application development? Not at all. For some categories of application, I think it’s a great way of structuring your code. But REST isn’t designed for talking to people. So let’s accept that fact when creating applications. And, while we’re at it, let’s take advantage of the fact that HTTP is such a flexible transport. Rather than trying to design one monolithic application than has both the CRUD functionality and the smarts to be able to talk HTML to end users, why not split it into two smaller and simpler applications?


image


Put the main application logic into a RESTful server. This is where all the CRUD-style access to resources takes place.


Then, write a second proxy server. This is an HTTP filter, sitting between dumb browsers and your core resources. When browser-based users need to interact with your resources, they actually connect to this proxy. It then talks REST to your main server, and interprets the RESTful responses back into a form that’s useful on the browser. And this filter doesn’t have to be a dumb presentation layer—there’s nothing to say that it can’t handle additional functionality as well. Things like menus, user preferences, and so on could all sit here.


Where does this filter sit? That depends. Sometimes it makes sense to have it running locally on the user’s own computer. In this case you’re back in a situation like the one we looked at with the blogging software. Sometimes it makes sense to have the proxy running on the network. In this case you gain some interesting architectural flexibility—there’s no rule that says you must colocate your REST and presentation servers. Potentially this can lead to better response times, lower network loads, and increased security.


With this approach, we can lose all the hacks we need to make to our URLs and POST data to make them act RESTfully. And we can lose all the ugly respond_to stanzas in our Rails server code. We might even be able to get better reuse, sharing both resource servers and presentation servers between applications.


Let’s call it RADAR. We have a RESTful Application talking to Dumb-Ass Recipients.


Does it work? I don’t know. You tell me.

Wednesday, March 21, 2007

SYWTWAB 7: Reviewers, And How Not to Kill Them

This is the seventh of a series of personal notes to people who may be thinking of writing (or who have embarked upon writing) a book. You’ll be able to find them all (eventually) by selecting the tag Writing A Book.




Writing a book is an intensely personal thing. You spend hour after hour nurturing it, turning raw words into something you can be proud of. You’ve done your best to convey your message, and to make a book that’s both accurate and entertaining. It’s gorgeous.


It’s really quite a shame that you have to show it to other people.


But you do.


image



And those other people don’t realize how much work you’re put in, and how subtly you’ve plotted it, and how many ideas you researched. They just want to tell you what’s wrong. That’s their job—they’re reviewers.


Your publisher will have their own policies when it comes to reviewing. Some have a book reviewed chapter by chapter (which seems strange, as good books are not written chapter by chapter). Some review content in chunks. Some wait until the end. And some use a beta-book process, where the general public can subscribe to the book as it is being developed.


Whatever the process, you’ll get reviews. And these can be tough to read. Let’s see how to handle them.


Start by remembering that every review comment is there because someone took the time to write it. The writer hoped to help make your book better. It may be painful, but take every review as something positive.


Each reviewer will have his or her own style, interests, level of experience, and so on. This is just what you want—if your publisher has done its job correctly, your reviewers will be representative of your target audience (with a couple of world experts thrown in to keep your content honest).


Some reviewers may seem to be terribly naive (or even just plain stupid). You’ll find yourself thinking “this guy just doesn’t get it.” And that’s the point. If a reviewer doesn’t get some point, or doesn’t understand something that you feel should be blindingly obvious, that’s an indication that you may have a problem with the book. After all, it’s your job as the author to make that technical detail obvious. So look hard at the reviewer’s comment and see where they’re coming from. Maybe you assumed that the world would interpret some phase or explanation one way, but the reviewer took it another way. Maybe changing one or two words would clear it up; maybe an extra paragraph of introduction; maybe a sidebar. Or maybe the section could be lost altogether or moved later in the book; it’s easy to introduce advanced material too early.


Other reviewers can came across as being incredibly arrogant. They seem to take almost vicious pleasure in finding fault. Their tone suggests that only an idiot could have made this kind of mistake. These reviewers suffer from what experts call penicranial inversion.


This type of reviewer is hard to deal with. You’ll feel defensive—after all, this person is apparently attacking you. Your natural reaction will be to dismiss what they say. But if you look past the aggressive wording, you may well find that the reviewer has a point. You shouldn’t have to take abuse from reviewers, but if you can rise above it, you may find gems buried in the crap.


Other reviewers like to copy edit. They find every little typo, complain about every breach of the punctuation rules, and note bad line breaks and other layout issues. Be careful when reacting to these reviewers. Obviously you can fix spelling issues, but you might want to chat with your publisher about punctuation—every publisher has a house style. And don’t sweat the layout stuff: your publisher should handle it for you (but you might want to double check the final proofs to double check that layout issues reported by reviewers have been fixed).


However, all these strange characters are extremes. You’ll probably find the majority of reviewers are supportive and helpful. Cherish these folks. If you have the opportunity to interact with them, do so. And be sure to thank particularly helpful reviewers somewhere in your book’s front matter.


Handling reviews can be tough. Try to remember as you grind your teeth that it is better to find these issues now, rather than after the ink has dried on the paper.



Sunday, March 18, 2007

Agile Mastery

I’m in London, having just finished presenting at QCon.




One of the tracks was called Reflecting on our Agile Journey - How do we reach Mastery? The more I think about that, the more the title worries me, because it seems to contain an implicit assumption—that mastery is some state that can be reached.


I think there are methodologies which can be mastered, where you can say “now I know it all.” I don’t believe agility is in that camp. For me, agility is all about the journey. Along the way, we’ll always be faced with forks in the road. The agile principles help us decide which to take. And we just carry on, enjoying the trip and doing our best along the way.



Sunday, March 11, 2007

SYWTWAB 5: Finding Your Voice

image





This is the fifth of a series of personal notes to people who may be thinking of writing (or who have embarked upon writing) a book. You’ll be able to find them all (eventually) by selecting the tag Writing A Book.


So far we’ve discussed the idea that a book is a narrative, and that the reader is the hero, undertaking a journey to learn new things (to boldly go…). I suggested that you might even want to start your book with an explicit story, something to draw the reader in. So just how do you write in such a way that the reader wants to join you? I don’t pretend to know, but there are some tricks that pragmatic authors use.


First and foremost, always remember that the book is about the reader, not the writer. Your job as author is to help the reader become proficient. You are a mentor, a guide. So always invite the reader along as you explore and explain.


The consistent use of pronouns is a great start. As the author, don’t try to hide behind a passive or stuffy voice. Don’t say “It is widely believed that…”. Don’t say “One could argue that…”. Just come out and say it: “I think that…”. Engage with your reader.


Your reader is a person too, so address them directly. Say things such as “You’ll need to install Xyz now” and “If you run the code now, you’ll see the following output.”.


But remember, you’re sharing a journey. So, whenever possible, try to use “we” and “our” to refer to the reader/author team. Don’t start a chapter with “In this chapter I’ll show you…”. Don’t start with “In this chapter you’ll find out…”. Instead, join the reader as they learn: “In this chapter we’ll see how…”.


Remember what I said about the Dreyfus model in SYWTBAW 3? Novices want to be told what to do. So don’t be afraid to use the imperative voice, particular early on the the book.


Here are a couple of paragraphs from the Rails book.



Back in xxxxx we sketched out the basic content of the products table. Now let’s turn that into reality. We need to create a database table and a Rails model that lets our application use that table.


The migration has a sequence number prefix (001), a name (create_products) and the file extension (.rb, because it’s a Ruby program). Let’s add the code to this file that creates the table in the database. Go to the db/migrate directory and open the file 001_create_products.rb. You’ll see two Ruby methods.



See how the reader and author share the journey: “We need to create”, “Let’s add the code”, and so on. At the same time, the reader is told what to do: “Go to the …” and “open the file”.


Layered on top of all these tricks with pronouns, the book has to be readable and engaging. You need to find a way of injecting energy and passion into your writing. You have to make people want to turn the page, to learn more about your subject. Clearly, technical knowledge helps. A good turn of phrase can make the content more interesting. Light, casual humor (particularly if it is off-beat or subtle) can keep the reader on his or her toes. Adding the occasional piece of trivia or background can add a little spice.


But all of this is predicated on one truth: the voice of a book has to be your voice. It has to be genuine; it has to be you. This is probably the most difficult challenge an author faces on a new book project.


So—how can you find this voice?


My best advice is to look at your book’s content and choose a chapter where you’re really comfortable with the technical material. Then just sit down and start writing. Write without reviewing what you’ve written. Don’t look back and correct stuff. Just write. Keep going for as long as you can. When you stop, save the file without rereading it. The next day, keep on writing. And continue the day after that. At some point, something will click in your head. You’ll find your writing style suddenly gels. Your prose will start to flow and the words will start to hang together well. You’ll know the feeling when it happens.


And when it does, here’s what you do. Save the file you’ve been working on, and promise yourself you won’t look at it again for a long, long time. Instead, choose another chapter and start writing. This time, you’re writing for real. Still keep the flow going, but now it’s OK (actually, it’s vital) to go back and read what you wrote. Ask the readers on your shoulder what they think. Because now you’re writing a book for real.


At some point you’ll have to revisit that original chapter. My advice? Just throw away the one you wrote initially and do it again. It’ll come out infinitely better than it would if you tried to make incremental improvements to your original prose.


When I write a new book, I follow this approach. I plan to throw away something like the first 30 or so pages. And, because I know I’m going to do it, it doesn’t worry me. I no longer have writer’s block. I no longer agonize over the voice in the first paragraph. I just churn it out until I feel comfortable, delete everything I wrote, and then start in for real.


In The Mythical Man Month, Fred Brooks advises project teams to expect to “throw the first one away.” That’s still good advice after 25 years, and it applies just as much to the writing process as it does to software development.


Give yourself time. Give your book a voice.



Friday, March 9, 2007

SYWTWAB 6: Wrandom Writing Wrules

image


This is the sixth of a series of personal notes to people who may be thinking of writing (or who have embarked upon writing) a book. You’ll be able to find them all (eventually) by selecting the tag Writing A Book. In no particular order:


Work Where it Works


Writing is difficult. Part of the trick of writing is to be able to focus on the writing itself—there’s no point making things harder than they already are. So choose your writing environment carefully, and defend it when necessary.


So just what is a good writing environment? You tell me—everyones’ is different. Some folks I know go down the local coffee shop and blitz out chapters. Other people go to the park. Some write in bed while waiting to go to sleep.


I personally look for an environment where I know I won’t be disturbed. It isn’t good enough that I don’t get disturbed—I have to know that the interruptions aren’t going to happen. Without that, part of my brain is constantly scanning the background, waiting for the e-mail ping or the IM window to pop open. So, when I sit down for a writing session, I kill off all my network applications. I put the phone on DND, and I close the door. The risk of being disturbed drops and my productivity rises.


For this same reason I find I’m really productive is on airplanes: I stick the earphones in and I can churn out page after page.


It really doesn’t matter what environment works for you. Experiment until you find one, and then use it.


Stay Out of Lay Out


Many authors claim that they want to see the final format of the book as they write it. I don’t agree. Remember the underlying axiom: writing is difficult. Why make it doubly so by simultaneously worrying about form and content?


Assuming you’re using a decent tool for creating your book, you’ll be using logical markup. You won’t be saying “set this word in Courier.” Instead, you’ll use markup to say “this word represents a variable name.” If that’s the case layout is simply a matter of associating sectioning and character markup with the appropriate layout elements. You book can change appearance many times without changing anything in the content. With this kind of system, your job is to use the correct markup as you enter content. This lets a designer work on the layout in parallel. It’s just another case of separating concerns to make life easier.


A Time to Write


In some ways, writing a book is like going to the gym. It’s hard work and it feels better when it’s over.


To help us go to the gym, most people have some kind of schedule: they go before work, Mondays, Wednesdays, and Fridays; or they go whenever American Idol is on (the latter folks are fit.) Having a schedule means that you feel an obligation to go. It also means you tend not to schedule other things at the same time, which gives you one less excuse to skip a session.


Do the same with your writing. Pick a schedule for writing and stick to it. Some folks like to set their alarm clocks a couple of hours early and write before the rest of their house gets up. Some folks write after everyone else goes to bed. Or you can write on Sunday afternoons and Wednesday mornings. The key things are to (a) agree a schedule with those around you, (b) make sure it is practical to use your ideal writing environment during those times, and (c) to use those times to get some writing down. An entry in a diary doesn’t get you fitter; time spent exercising does.


End at the Beginning


One of the biggest mistakes new writers make is to start their book by entering “Chapter 1: Introduction” into their editor. No one really knows what their book will contain when they start out, so writing an introduction to it is really pretty futile. Make life easy for yourself and write the introduction last.


So, where should you start? Assuming you’ve already written the throw-away prose while finding your voice, I think you’re best off choosing a chapter where you feel at home with the material. Belt it out, and then move on to something else fairly straightforward. Make sure you have the voice business nailed cold. Show what you’re written to some friends. Get into the swing.


Then change gears. You’ve mastered the writing part. Now look at the content. Is there some area where you don’t feel so confident, or where you need to do some research? Now’s the time to get that out of the way. Tackle the difficult stuff while you’re still relatively fresh. You’ll find it easier to do now than later. And, more importantly, you won’t have it hanging over you while you write. Getting the difficult stuff down also gives others longer to review it.


Cut and Paste is Your Friend


No one gets it totally right when they plan a large project. You always discover things that you thought went here, but that work better over there. So don’t get overly protective of your outline.


As you gain experience with your book, take every opportunity to look at your content and ask whether it is flowing the way you’d like. Cut out text that doesn’t fit. If you know where it belongs, paste it into the new chapter. But don’t try to put it in exactly the right place. Instead, paste it at the very top of the chapter and flag it somehow. Then get on with writing what you were writing originally—don’t break your flow. Then, at some point in the future, you can go back and knit the orphaned piece of text into its new home.


I keep a special dummy chapter available for snippets that don’t seem to have a home. As the book developer, I periodically look to see if any of these snippets might add value to any existing content, moving them into place if they do. I’m not particularly sad if I finish a book and find the bucket still has stuff in it: books are ruined by authors trying to add every single idea they have to the flow.


Interact


Never, ever, write in a vacuum. Show your work to people: friends, colleagues, reviewers. Check with your publisher to see how they want to handle this, but get feedback early, and get it often. If you work with a publisher who offers a beta book program (where books are released as works in progress), see if you can take advantage of it: releasing your book to the general readership early will generate a tremendous amount of feedback.


Save and Archive Often


It’s almost embarrassing to have to say this, but please make sure you back up your book. When the Pragmatic Bookshelf signs a new book, we create a new project for the book’s files in our Subversion version control repository. As the author creates and edits content for the book, we strongly encourage them to commit these changes to the repository, multiple times a day. That way, if they have a hard drive failure they won’t lose their work: it’ll tucked away securely on our servers.


Other publishers may not offer a Subversion repository for your work. If that’s the case, I strongly recommend you set up something similar for yourself. At the very least, get into the habit of archiving your writing onto some external machine or external media on a regular basis. Ideally, keep the content in a version control system.


Use Cross References as Placeholders


Although I’ve written about this before, I’d like to mention it again. Use cross references as placeholders for content you haven’t written yet.


Outline the Plot, not the Paragraphs


It’s just natural: people like lists. And all the conventional wisdom tells writers to produce outlines for their books. So many writers take this to heart, producing long, deeply nested notes that tell them what each paragraph in a chapter should contain.


If this works for you, do it. But don’t feel that strict outlining is necessary. When I write, I start each chapter with a simple list of the things I want the reader to learn by the time they get to the end. I then sort them into an order which seems to make sense as a narrative. Then I replace each item with a section or subsection of prose. I find that this scheme gives me the skeleton I need for a chapter, but it also gives me the flexibilitiy I need when fleshing it out.


This list just scratches the surface—there are whole books on writing tricks. As yo’re starting out, ask your editor for their ideas. Search out more experienced authors and see what they do.


Writing a book is difficult. Don’t make it harder than it has to be.