Chad Fowlers Speaks!

SysCon interviewed Chad Fowler about My Job Went to India.

Lots of cool music talk too.

If this makes sense to you right now, then stop thinking about it! You might screw it up! ;) Otherwise…don’t sweat it too much. Move on; let your subconscious do the work later.

Chris Pine’s Learn to Program book, in the section on class objects. It struck me as good advice regardless of the context:

Magical Thinking

I’m reading Augusten Burroughs’ wonderful “Magical Thinking” on the plane to Boston. It opens with a definition:

Magical Thinking

A schizotypal personality disorder attributing to ones own actions something that has nothing to do with him or her and thus assuming that one has greater influence than is actually the case

I couldn’t help but mutate it slightly:

Project Planning

A schizotypal personality disorder attributing to ones own actions something that has nothing to do with him or her and thus assuming that one has greater influence than is actually the case

Weighty Matters

A link on digg gets us to a page about Germany’s new Water Bridge, an impressive aquaduct.

So, today’s Thanksgiving dinner question: what happens to the load on the bridge’s supporting columns as traffic enters an empty bridge? Do they feel additional weight? Does the load on them stay the same? (Please don’t e-mail me: just a question to get the family thinking.)

It was an intense three days, but it’s over! We just finished the very first Rails Studio in sunny (if chilly) Reston, VA. The site couldn’t have been better. FGM, Inc hosted us, and they did us proud. The meeting room was great (it even had windows :), and they laid on a network and plenty of power (we were initially surprised at how many folks came with laptops, but FGM came to the rescue with additional power strips and we just squeaked by).

The course itself seemed to go really well. We learned a lot along the way (Dave now knows not to start a new topic and delay lunch). We also adapted the presentation style as we proceeded based on the prior day’s feedback. This lead to probably the biggest overall lesson learned. We originally had people coding along with us as we developed our Rails application, but at some point or other just about everyone went down a rabbit hole and had a hard time catching back up. Next time we’ll have checkpoint versions of the application available so everyone can sync back up before we move on to the next topic.

I think everyone came away having picked up a lot about Rails coding (not to mention the “I’d Rather Be on Rails” bumper sticker and the exclusive Golden Spike T-Shirt). On the final day, everyone coded their own Rails application—some of these had components and Ajax support!

We’ll be announcing the next studio shortly: we’re probably looking at the west coast next, with a possible return to the east coast shortly thereafter. In the meantime, there’s a bunch of pictures up on flickr which give a pretty good idea of the fun atmosphere.

Thank you, golden spike members, for coming along and making this a success…

Symbol#to_proc

The Ruby Extensions Project contains an absolutely wonderful hack. Say you want to convert an array of strings to uppercase. You could write

  result = names.map {|name| name.upcase}

Fairly concise, right? Return a new array where each element is the corresponding element in the original, converted to uppercase. But if you include the Symbol extension from the Ruby Extensions Project, you could instead write

  result = names.map(&:upcase)

Now that’s concise: apply the upcase method to each element of names. So, how does it work?

It relies on Ruby doing some dynamic type conversion. Let’s start at the top.

When you say names.map(&xxx), you’re telling Ruby to pass the Proc object in xxx to map as a block. Ifxxx isn’t already a Proc object, Ruby tries to coerce it into one by sending it a to_proc message.

Now :upcase isn’t a Proc object—it’s a symbol. So when Ruby sees names.map(&:upcase), the first thing it does is try to convert the Symbol :upcase into a Proc by calling to_proc. And, by an incredible coincidence, the extension project has defined a to_proc method for class Symbol. It looks like this:

    def to_proc
      proc { |obj, *args| obj.send(self, *args) }
    end

It creates a Proc which, when called on an object, sends that object the symbol itself. So, whennames.map(&:upcase) starts to iterate over the strings in names, it’ll call the block, passing in the first name and invoking its upcase method.

It’s an incredibly elegant use of coercion and of closures.

Rick Wayne sent me a screenshot that blew me away, and I just couldn’t resist sharing :) (sorry about the quality: I cranked it down to save download time)

Is Ruby Better Than…?

As I speak about Ruby with folks around the world, a question comes up with depressing regularity: “is Ruby better than Xyz?”, where Xyz is the questioner’s current language du jour.

The simple answer is “no.”

But then let’s ask the same question differently. Is Xyz better that Ruby?

The simple answer is “no.”

The longer answer is that the question falls into the same category as “have you stopped beating your wife?”—it makes an implicit and (hopefully) incorrect assumption. In this case, it assumes that it’s possible to compare two languages and come up with some kind of binary A trumps B evaluation.

Is your chisel better than my hammer? If I’m forming dovetail joints, yes. If I’m nailing two-by-fours, no.

Is my Ruby better than your Java (or C#, or …)? It depends.

So, over the next few weeks I’ll try to post some answers. I’ll look at areas of Ruby which seem to be hot buttons for folks when making language comparisons.

Let’s start with a big one:

Performance

Ruby is probably slower than Java or C# in most real-world situations. Does that worry me? Not normally.

Think about your average web application. An incoming request appears on an Ethernet interface. The packets make their way up through the IP and TCP stacks, then get queued up as a data buffer. The web server receives the data, decodes some of the content and passes it off to your application server. There the information is massaged into some kind of request object, and your application is invoked. Your code hits the database a couple of times, probably using an OR mapping layer such as Hibernate to sanitize the access. It then constructs response data before calling some kind of templating library to format a response. The response heads back through the application server, and possibly through the web server, before being deconstructed into packets and sent over the network.

Whew! There’s a bunch of processing going on there. Millions and millions of machine instructions being executed. There’s latency while packets are queued, processes are scheduled, threads are dispatched, and disk heads seek to the right track.

How much of this processing is done in code that you write? How much of the total time spend handling the request is spend executing the actual instructions corresponding to your code?

Zip.

I’m guessing maybe 5% on a really bad day, with a complex application. Let’s face it: you average commercial application isn’t burning CPU cycles solving NP-complete problems. We typically write code that moves chunks of data about and adds up a couple of numbers.

In these scenarios, is it worth worrying about the relative performance of the language used to do the moving and adding? Not in my book.

Instead, I’d rather write in a language that let’s me focus on the application, and which lets me express myself clearly and effectively. And, if I can do those two things, I believe that sometimes I’ll be able to write code which is cleverer than something I’d write in a lower-level language. A better algorithm will easily gain back any marginal performance hit I take for using a slower language.

Justin Ghetland experienced this recently on a Rails project. Having coded the same application twice, once in Java and once using Ruby on Rails, he was surprised to discover that the Rails application outperformed the Java one. Why? Justin believes it’s because Rails does smarter caching. The Rails framework contains some very high-level abstractions, and that allows the folks writing the framework to be smart about what they do. They accepted the linear hit of writing in a language that executes more slowly because they got a non-linear increase in speed from being able to write better code.

Clearly there’ll be times where you need to squeeze the most out of your CPU, where your application itself is the bottleneck and it’s CPU bound. In these cases, Ruby might be a bad choice. You probably want to look at a high-performance language such as OCAML. But even then, the choice isn’t always clear. Ruby with a good parallizing matrix library would probably beat Java or C# code which inlines the same operations.

So, is Ruby fast enough for your application?

I don’t know. But I do know that I wouldn’t assume that I can’t use Ruby for performance reasons. There are plenty of sites out there pumping high transaction volumes through a Ruby application. The real answer, as always, is it depends. If you’re concerned about performance as you start to develop a new application, it doesn’t matter if you’re writing Ruby, Java, C#, or assembler. It’s prudent to spend a small amount of time doing some rough tests on your proposed architecture before you start laying down the code.

Is Rails Ready for Prime Time?

I give talks on Rails and Ruby to Java developers on average once every two weeks. I’ve probably spoken to thousands of folks over the last year. And I don’t think I’ve ever had a talk that didn’t end with someone asking “that’s all very nice, but is Rails ready to be used in my company? Can I stop writing Java and start writing Ruby?”

As with most good questions, there’s no one simple answer. Here’s what I tell people.

First, Ask the Right Question

One of the main selling points for Java in the late 90’s was the idea of having a language, runtime, and library set that could be used everywhere. Use Java to code your cell phone, and use Java to code your largest enterprise application. To some extent that thinking has insinuated itself into the mind sets of many developers—they are looking for one solution, one framework, and one language to solve all their problems.

That’s silly. Some situations need one tool, other situations another tool. Using the full might of a J2EE stack to write a small stand-alone application is using a sledgehammer to crack a nut. But I keep hearing the sound of nuts being pulverized as developers seem to think that using anything other than J2EE is somehow unprofessional.

At the same time, there are situations that call for the shock and awe that is J2EE. Some applications need to make use of facilities rolled into the stack, and it would be crazy to reinvent the wheel.

So when people ask “should I be using Rails instead of Java?”, the answer has to be “not exclusively: you’re likely to want to use Rails as well as Java.” Why? Because I’m a firm believer in having a bag of tools at your disposal, tools that you know how and—most importantly—when to use. When Rails is appropriate, you’re going to be hard pressed to find a more productive environment.

And, even if Rails is the clear winner for a particular application, developers face a second challenge. They have to convince their companies to let them use it.

So the real questions are “when is Rails appropriate?”, and “how can I introduce it where I work?”

Let’s start by looking at some of the hurdles Rails has to clear to make it an appropriate choice where you work.

Next, Make Sure It’s Possible

When shouldn’t you use Rails?

• 
  

  If you need to handle transactions across multiple databases. Rails/Ruby doesn’t have two phase commit (yet).

  
  


• 
  

  If you have really strange database schemas. ActiveRecord beats Hibernate when the schemas are fairly well structured. Hibernate wins when you have to map the schema-from-hell before using it inside your application.

  
  


• 
  

  If you’re adding small chunks of functionality to an existing, large application. OK, this is contentious. If I was reading this, I’d probably fire up my mail client and complain that this is exactly when you should use Rails. Integrate to the monster using web services, and add new functionality in something more malleable, such as Rails.

  
  

  Except… most monstrously large applications I’ve come across got there over time. They’re messy, hard to work with, and tightly coupled. If you wanted to expose parts of them via web services, you’d first have to restructure them to make those parts callable from the outside, and that probably isn’t worth the effort (if it’s possible at all).

  
  


• 
  

  The developers don’t want to. If the rest of the developers don’t want to use Rails, then forcing them to help you create a pilot project is a simple recipe for embarrassment. Pull, don’t push, when it comes to change.

  
  

Then, Make it Realistic

Companies are all different, and there’s no one road to Rails adoption.

In some environments, there may be no resistance to using Rails. I’ve even heard of some places where the managers are working to persuade the development staff to give Rails a try. Typically, it’s the small and medium size businesses that have this kind of flexibility. Companies such as Odeo and VitalSourceembraced Rails as a key component of their business plan.

Other companies, including most large ones, have a more conservative approach when adopting new technologies. So how can a developer who’s convinced that Rails would benefit their company get it into production?

In general, I suggest an incremental approach. Start with small, developer-focused applications. For example, a new table may have been added to the database. Developers need to be able to maintain test data in it, so they can continue to write parts of the larger (Java) application. Whip up a quick Rails application in 20 minutes and leave it running on a Webrick server somewhere. Folks will notice how easy it seemed to be.

Maybe at some point the development group might need a web application for their own use (change request trqcking, for example). Suggest writing it in Rails. Mention that not only will it take less time (and hence cost less in terms of overhead dollars), but it will also allow the group to get some experience with the technology. This is where you get to show off. First deliver a solid solution, then add stuff such as Ajax support to show them how things might be (if only they used Rails in all their applications…).

Then the time might come to roll out an inward-facing corporate application. If the development team are now comfortable with the technology, propose a Rails application. Now you’ll get experience running Rails in a production environment.

Finally, once everyone is comfortable, think about rolling out an externally-facing Rails application.

Easing the Transition

Here are three suggestions for making the transition to Rails applications less extreme:

• 
  

  Start using Rails Conventions in Your Java Applications. Maybe you aren’t in a position to make the switch now. Even so, think about using some of the Rails conventions when you do things such as add new tables to your Java applications. Using the Rails naming schemes won’t make any difference to Hibernate, and you’ll be in a stronger position to write Rails applications against that schema. Even though the external application is still J2EE, you can write internal, ancillary applications in Rails. And if the schema is Rails friendly, the effort will be less.

  
  


• 
  

  Do parallel developments. The jury is still out on the productivity increases that come from using Rails, but I see some very credible people talking about numbers in the 5-15x range. So if Rails really does let you write applications in 10-20% of the time you’d otherwise take, why not do a Rails development in parallel with the real one? If the main project is slated to take 5 months using J2EE, and after a month you have a working Rails version, your company gets to make a decision.

  
  
  • 
    

    They can use the Rails version, having invested 2 months of effort (one month for Rails, one month for the parallel Java team). This is a net saving of 3 months over the original 5 month estimate.

    
    
  
  
  • 
    

    They can choose not to use the Rails version. In this case they’ve spent an extra month, and learned something valuable. Is that extra month significant? Probably not—it seems to me that any project that comes in within 20% of its estimate is a raging (and unusual) success, so this extra time just gets swallowed by the general budget slop that surrounds all software development.

    
    
  
  


• 
  

  Make sure you know what you’re doing before leaping into an ambitious project. As the cliché says, you only get one chance to make a first impression. If you want to show that Rails is viable in your company, then you need to make sure that all the folks doing the work on your first visible Rails project know what they’re doing. Think hard about the deployment and maintenance issues before committing. Choose something simple, and implement it well.

  
  

What do you think? Is all this realistic? Have you already introduced Rails to your company? How’d it go?Let me know.

Mining Build Statistics

Mike Clark’s been posting some interesting views of CruiseControl build statistics (here and here). These are cool pictures, and I suspect there’s a bunch of information to be gleaned from them:

• what’s the blob density (how frequently do folks check in)?


• are they mostly red or mostly blue (red is broken builds)?


• after a red blob, how many more red blobs do you see before you get a blue one?

That last statistic is one I think might be interesting. If you were to plot a histogram of the number of sequences of red blobs of length one, length two, length three, and so on, what would it look like? If it is heavily weighted to the low end, then you’re looking at a team that fixes broken windows.

I’m sure that there’s a lot more cool stuff in these pictures, but I’m impressed at how quickly things jump out at you, and just how much tacit information they convey. Maybe project tea-leaf reading will become a sought-after discipline…

Spam Volume

I’m sure I’m just in the minor leagues here, but for the last month or so I’ve been averaging 50Mb of incoming spam a day. All but a handful of messages die before reaching my inbox, but it makes me wonder. I’m lucky enough to have fiber to the house, with some pretty decent bandwidth both up and down. But two months ago I was on DSL, and only a few years ago I was on ISDN, with dial-up before that. As the connection speed increased, so did the volume of junk that found its way down the pipe.

Maybe we’re seeing a kind of Parkinson’s Law in action here: trash expands to fit the bandwidth available to it.

ongoing by Tim Bray · Shiny Red Rock

Link: ongoing by Tim Bray · Shiny Red Rock

Thanks to a post on LtU, I got to read Tim Bray’s piece on Ruby. His conclusion? “It sure looks like more than a fad to me.”

We Reached the Station

Agile Web Development with Rails has been finalized. We worked hard with our printers, Malloy, to come up with what is possibly the most agile book publishing schedule ever. As a result, the final PDF went in for printing today (with 47 pages changed since I first sent them the book a week or so ago), and the books will be bound and boxed on July 28. They then enter the distribution phase, so it’ll still be a week or so before they start shipping, but it looks as if we’ll have some for OSCON.

Some stats:

• The PML source for the book is about 145k words, and the files weigh in at 1.3Mb.


• There are 1041 Ruby and rhtml source code files used as examples in the book (about 560kb).


• The book went through roughly 2000 Subversion commits since inception.


• Reviewers generated well in excess of 6Mb of comments.


• We’ve had four released versions of the beta book, and now one final version.


• Beta readers made over 780 suggestions, comments, and typo reports, most of which made their way into the printed book.


• To date the book has been bought by people in sixty three countries (that blows me away).

Thank you again to everyone who trusted the process and participated in the Beta program. You can update to the final version of the PDF at the regular URL (I’ll be sending an e-mail around to you all sometime today with the link).

And for everyone else—the folks waiting for the dust to settle before exploring the book—now’s a great time to come on board. Welcome!

Knee Testing

Link: Knee Testing

Knowing the User and Their Unique Environment As I was working on an article for Better Software magazine, I found consistent patterns in cases where I have found a repeatable case to a so-called &…

I still occasionally bump into folks who claim that all testing can (and should) be automated. For those optimistic souls, I offer this tale, by way of Brian Marick’s blog.

Automated tests are necessary, and automated unit tests in particular are a wonderful design aid. But ultimately they only test the things you know to test. It’s the other things that get your knees wobbling (you’ll have to read the story).

Shared Find Clipboard on OSX

Thanks to Allan Odgaard for pointing out the shared find clipboard in OS X. Highlight text in one application and hit Command-E. Go to another application, hit Command-G, and it searches for that text. And all without disturbing the regular clipboard. That tip has saved me a boatload of time over the last couple of days as I’ve been reconciling data between various systems.

Putting Code in Python Documentation

Link: Putting Code in Python Documentation

To continue the thread, Rod Morehead gave me this link to Python doctest, a utility that performs what are almost unit tests on the code fragments in your Python docstrings.

Anyone in the Ruby community feel inspired?

Putting Code in Books

Maybe there’s something to synchronicity. In the last month, I’ve had 4 or 5 people all ask me about (or e-mail with quotes about) putting source code in books. So, just because it isn’t written down anywhere else, I thought I’d jot down some notes.

When we wrote Pragmatic Programmer, we knew we were going to be including a fair amount of code. We wanted it to look decent, and we also wanted it to be correct. For us, that meant that we needed to be able to run most of the code shown in the book.

We used TeX to do the book typesetting (the book was printed from a high-resolution Postscript file), so that gave us a lot of flexibility (the power of plain text…). In particular, it let me implement a preprocessor to handle all the code in the book.

For the short code snippets, you could say:

   \begin{code}
     a = 1;
     b = 2;
     System.out.println(a+b);
   \end{code}

Now we’ve started the Pragmatic Bookshelf, and we have external authors. Rather than expect them to wrestle with TeX, we switched to an XML-based markup. Using it, you’d instead write:

   <code>
     a = 1;
     b = 2;
     System.out.println(a+b);
   </code>

The preprocessor finds all these code blocks and did the necessary magic to format them nicely. If you wanted syntax highlighting, you had to tell it the language:

   <code language="c">
     for (int i = 0; i < 10; i++) {
       printf("i = %d\n", i);
     }
   </code>

There are many other options: font size, line numbering, indentation, and so on.

We keep the bigger code samples in source files that can be compiled and tested. These are included in the books by specifying a file name (which also triggers the correct syntax highlighting):

   <code file="code/fib.c" />

Sometimes we have running examples, or the need to show just a small part of a larger program. To handle this, the preprocessor allows you to specify tags. Inside the source file, you delimit chunks of text with START:tag and END:tag in the code’s comments. Only those chunks are then copied to the final book.

   <code file="code/fib.c" part="setup"/>

For Ruby code, we get a bit fancier. One major difference is that we want to show the results of executing Ruby code.

  <code language="ruby">
    <run saying="This outputs">
      Dave
      says
    </run>
    a = gets
    b = gets
    puts a + " " + b
  </code>

We wanted to book to show:

        a = gets
        b = gets
        puts a + " " + b

   This outputs

        Dave says

The preprocessor can also extract and cross reference the code from books. The online summary of the Ruby book’s code (including things like “show hidden code”) is cross referenced to page numbers and generated automatically by the preprocessor, TeX, and a wee bit of post-processing glue.

The nice thing about all this (particularly with Ruby books) is that we know that the results shown in the book are correct as of the time of writing: every single one of those results is produced afresh each and every time the book is formatted. And when Ruby changes and things break, the book just plain stops formatting. It’s also nice to be able to pull a sequence of extracts from a single program source file: it’s a great way of ensuring they are consistent.

It’s true that XML and other plain-text based documentation systems can be a bear to use at times, but that’s a price we’re happy to pay in exchange for the flexibility of being able to pre- and post process the book as it is formatted.

The Pragmatic Studio

For a while now, Andy and I have been giving our Pragmatic Workshop to companies around the world.Mike Clark, who wrote our Automation Book has been doing the same, running workshops on testing and automation.

So when three of us got talking, it seemed natural to see if we could bring the threads together and produce a combined workshop covering the Starter Kit essentials. The more we talked, the more fun it sounded—we wanted to put together something a bit more lively and interactive that the typical “sit and listen to someone talk” session. And we wanted to make it stick—these practices are vital to project success, and the more we could help teams come away with the will and knowledge to use them, the better.

Mike did all the work, and he’s done an incredible job. The result is the Pragmatic Studio, a series of two-day events held in cities around the country. The first is in Denver on June 20th and 21st, followed by dates in Seattle and Reston. This is just the start—if you want us to come to your town, drop Mike a note and let us know.

I’m excited about this. In the past, we needed to find companies who’d pay the full cost for us to come in and work with their teams. Now Mike’s put together events where companies can share the costs, sending as many team members as they want.

Maybe we’ll see you there…

The Wiki Way of Writing

Leaving dangling cross references when writing a book is like leaving dangling hyperlinks in a Wiki—they’re both a promise and a question.

I’m deep down in the bowels of writing the new Rails book. As usual, I’m adopting a fairly agile development style, moving stuff around between chapters, adding and deleting whole chapters, and generally working stuff out as I go along.

One of the tricks that I discovered myself using is the dangling cross reference. If I’m writing some introductory material, for instance, I might do something such as

  Of course, you'll need to tidy those session files. We'll talk
  about this later in <pref linkend="sec.tidy.session.files"/>.

(Our books use a simple XML markup, and <pref…> creates a cross reference [including a page number] to a named tag.)

As I haven’t yet written about tidying session files, when I format the book, the cross reference shows up as

    about this later in ? on page ?.

But even better, the formatting process itself summarizes the list of all missing cross references.

This all means that as I write, I can drop in these references, not caring if the section exists. Every now and then I’ll look through the list of unknown tag names. If I’ve since written about that material, I’ll marry the text and the tag. If I haven’t I’ll know that at some point I’ll have to add more text on the subject.

In a WikiWiki web, a CamelCase word referencing an unwritten page acts as a promise of future content, and as an invitation for someone to produce that content. The same seems to apply to writing books. It’s fascinating to see the list of unfulfilled tags ebb and flow as the Rails book progresses. It’s a kind of dialectic writing process.

Failure and Surgeons

Glenn Vanderburg pointed me at a wonderful article, The Physical Genius by Malcolm Gladwell, originally published in the New Yorker in 1999.

In the middle, Gladwell is reporting on Charles Bosk, a sociologist who interviewed doctors in an attempt to determine the factors that differentiate successful from unsuccessful surgeons. Bosk says:

"In my interviewing, I began to develop what I thought was an indicator of whether someone was going to be a good surgeon or not. It was a couple of simple questions: Have you ever made a mistake? And, if so, what was your worst mistake? The people who said, ‘Gee, I haven’t really had one,’ or, ‘I’ve had a couple of bad outcomes but they were due to things outside my control’—invariably those were the worst candidates. And the residents who said, ‘I make mistakes all the time. There was this horrible thing that happened just yesterday and here’s what it was.’ They were the best. They had the ability to rethink everything that they’d done and imagine how they might have done it differently."

Thinking about how we might have done things differently is vital if we’re to improve (or even survive, given the way the industry is trending). If you live in a rut, the next cart that comes along is likely to flatten you.

Thanks to all my Fans

There’s a whole lot of work being done on monitoring the health and status of systems using differing patterns of background sound. Perhaps your server farm sounds like a rain forest, and a cricket chirps every time someone portscans you.

It turns out I have one of these set up for free. I run continuous builds for our books in production on my office G5, and the G5 fans are very responsive to processor load. So whenever the build system decides to rebuild a book (because an author has checked in changes), the fans kick in, and I can wander over and see what’s cooking. It’s surprisingly useful feedback.