Some Agile History

I spent this afternoon reorganizing (ok, just organizing) my files. In the process, I came across some of the notes I made in Snowbird back in 2001.

That was when 17 mawgs^[1] who cared about projects and software got together to discuss what were then called Lightweight Methodologies. XP had just taken off, and SCRUM was making waves, and we all wanted to see if there was enough common ground for us to isolate some underlying principles. In the end, I think we did something far more important—we isolated a set of four common values, and wrote them up as the Agile Manifesto.

As the conversation began to gel, I wrote down a short bullet list of things we were saying:

Soon after, we broke for lunch. Martin Fowler and I were standing at the whiteboard, trying to see if we could come up with a summary which captured the mood of the room. I noticed that as we were talking we often seemed to focus on what we didn’t like as much as on the values we believed in, so we experimented with writing down our conversation that way: we prefer X over Y. Gradually folks drifted over from the lunch table and the conversation flowed. (The picture that Ward took that forms the background on the Agile Manifesto site shows the group clustered around the board). I think we sat down after lunch with five bullets on the board. By the end of the afternoon, it was down to four, plus an introduction. I copied the whiteboard into my notes too:

You can argue the implementation (I see some teams that use the word “agile” when they really mean “chaotic”) , but the values still ring true to me today. I think the Agile Manifesto has helped teams around the world rethink their priorities, and in the process has helped re-humanize software development.

And that’s as it should be.

---

^[1] Middle-aged white guys

Agile Retail

I’m wondering: could the same organizing principles that work for community web sites work for retail stores?

If you go into a bookstore, you’ll notice that not all books are displayed the same. Some are displayed on tables at the store entrance, or at the entrance to particular sections. Some are displayed on endcaps—that narrow space at the end of a shelf. And some are displayed on shelves with their covers facing the public. And then there’s the vast majority of books, coyly showing you just their spines.

And then there’s the shelving itself. Some books have clear homes: a book on the Java libraries likely belongs under Java programming. But where does a book on Ruby belong? Programming languages? Web development? Next to Perl and Python? The jewelry section?

Ever wonder what determines where a book goes and how it is displayed.

The basic answer is that most large bookstores ignore the publishers shelving suggestions and decide at head office where each book goes, often based on title alone. This is why you sometimes find a book a long way from where you’d logically expect it to be.

And when it comes to displaying a book more prominently than the standard shelving, there’s another simple answer. Publishers pay for promotions, and stores display the books appropriately.

Are either of these practices in the interest of the customer? Perhaps indirectly—bookstores make a lot of money from the fees they charge for promoting books, and it could be argued that those fees keep the stores in business. But in terms of serving their primary purpose, which I’d define as making it easy for customers to find and buy interesting books, I’d say that both practices are suboptimal.

Having a central department decide shelving relies on one person being able to decide the likely audience for each book under their care, and then to decide where the average reader would expect to find that book. Both can at best cater to the typical case, and are bound to lead to frustration.

The reader is also poorly served when it comes to promotional placement. Sometimes it’s useful: having a table by the front door piled with the latest Harry Potter is helpful, given that you know that a large number of people will flock in to pick one up. But for the less popular books the practice again moves decisions from the consumer to some central marketing department. Publisher Xyz may want the world to know about a new book, but the world might not be interested.

Online we address these kinds of problems using various self-organizing folksonomy systems: tagging, search, top-ten lists, “readers who bought X also bought Y” and so on.

So could we do the same in the retail world? I think it might be possible.

Say you had a store where you encouraged people to reshelve things. You’d have broad sections (Computers, Photography, and so one), and some cardboard labels you could put on the shelves themselves. People could write their own labels, create their own sections, and move books as they saw fit. If someone really likes a book, they could turn it face out to show others they approve. They could even move the books onto display tables. Not sure if a Photoshop title belongs in Photography or Computers? Put a copy in both sections.

Would such a store work, or would it just be chaos? I frankly don’t know. But I suspect that people felt that user-organized sites such as wikis wouldn’t work, and Wikipedia and friends have proven that wrong. Flickr and del.icio.us have shown that tagging is a great tool for organizing content. And all these sites have an additional, emergent, property that would be wonderful to see in a bricks and mortar store—they allow serendipitous discoveries. You come across things that delight you that you wouldn’t have thought to look for.

Wouldn’t that make it worth going to a real store again?

Cheese on Toast

I love good cheese on toast. The trick is to get a nice, thin, lightly burned top while keeping the bulk of the cheese good and gooey. The skin gives it taste, the goop texture and follow through.

Tonight I saw a trailer for a cooking show where the chef caramelized the top of a creme brulée with a blowtorch. I couldn’t resist.

Lightly toast some bread. Add a really, really thin spread of marmalade (optional, but try it…). Top with grated cheese and apply the MAPP torch. You’d be surprised just how close you have to hold it to get the cheese to brown—I started waving it vaguely in the direction of the bread, but ended up playing the flame directly over it.

The result was gorgeous.

Two Hands Bad: The Frustrations of Dreyfus Level One

A while back an interviewer asked what I would do if I had three months of free time. Without hesitation I said “I’d take piano lessons.” I’ve been hacking away on pianos since I was a kid, but I never really learned how to play anything real.

So my wife got wind of the interview (thanks, Google), and for my birthday I got piano lessons. In fact, I got perfect piano lessons—my piano teacher is totally flexible, into theory, and can play just about anything, in any style. He sets homework to do one thing, I get sidetracked and bring back something different, and he just rolls with it and I learn from what I did. It’s invigorating, and great fun.

But I’m learning first hand about the Dreyfus model.

If you’ve heard Andy or I speak, you’ll know that we like the way the Dreyfus model explains the ways that people become experienced. I blogged a little about it in the context of keeping your job, and againin the context of CodeKata.

Now, when it comes to the piano, it turns out that I’m at multiple Dreyfus levels. For some reason, I’m OK(ish) at the theory side. I see the patterns, and I get it when my teacher rips into my attempts at composition. Some of the coolest times are when he takes over the piano and starts riffing on something I wrote, talking out loud as he finds harmonies and progressions. I might be a Dreyfus 2 or 3 at theory.

However, when it comes to the piano, it turns out that I’m a solid Dreyfus 1 when it comes to playing (I’d be a zero, but the Dreyfus brothers never heard me play, so never realized they’d need a level below 1). I can play a simple melody in a single hand and get by. But I’m aware of the fact that I’m playing—I’m consciously saying to myself “next it’s a B, now remember to tuck your thumb, C coming up” and so on. I’m clearly controlling my playing by thinking through each step. And that becomes painfully clear once I play with two hands. A simple line that I could play fluidly on its own suddenly comes to a juddering stop as my brain performs a process switch to concentrate on the notes to be played by the other hand.

When we talk about Dreyfus, we describe the difference between having to think about each step and the feeling you get once you’ve gained enough experience to be able to do something intuitively, below the conscious level. Once you’ve internalized something, you leave your conscious brain free to work on the nuances.

So this frustration I’m feeling while sitting at the keyboard is exactly the same frustration that someone new to programming feels when faced with Rails, or someone new to Java feels when all of J2EE is plunked down in front of them.

And knowing that is going to make me more tolerant when dealing with folks who’ve just starting out at something. It’s been a while since I felt like such a rank beginner at something. It’s a humbling (and worthwhile) experience.

The Canary Benefit

I haven’t done production work on a Windows machine for a long, long time. My desktops have been Linux since 0.99pl11 (was that ‘93?). My laptops were Windows until Linux started working on them, and then I switched.

It was good. I put up with the hassles: the upgrades, the incompatibilities, the laptops that would talk to some video projectors but not others. I got behind in my patching, and had a server root-kitted once (well before we had an online store, in case you’re concerned).

As the business grew, I found myself spending more and more time admining boxes. So, somewhat late, I made the switch maybe 3 or 4 years ago, first with an Apple laptop, then with a Mac Pro. As I grew more and more confident in the decision, I switched more and more of what I did to OSX. A bunch of our externally facing code runs on Linux and BSD, but it is all administered by third parties—folks whose job is to keep up with all the stuff that needs doing. Everything else runs on Macs.

And I’ve never really regretted the switch. I still don’t.

But, like miners keeping an eye on the canary, I monitor the one real thing that makes my switching viable. The key benefit of switching for me is the lack of hassle. I spend a bit extra for stuff that just works. .Mac syncing lets me move from desktop to laptop without a thought, but now I have a syncing loop where each machine tries to override information that is identical in the other. Things just aren’t as smooth as they were.

Am I regretting the switch? No. OSX works well for what I do, and it gives me access to tools I need (such as InDesign for covers, Sibelius for scoring, and so on). But I’m also not such a acolyte that I’ll never move off the Mac if I start seeing the kind of hassle I used to experience with Linux reentering my life. Modern Linux distros have come a long way since I last used one on a laptop, and I know that I could probably switch if I needed to with little regret.

So, what do I want from Apple, both tomorrow at MacWorld and then over the coming months? Easy. I want to see fewer cool features—features which seem to add problems—and a refocusing on what made Apple the machine of choice for a certain kind of developers. I want my Mac to be as hassle free, secure, and reliable as it was when I first started using OSX.

Right now, the hassle-free canary seems to be somewhat distressed. I’m monitoring its health closely.

Rails and the Legacy World

I gave what turned out to be a slightly controversial keynote at RailsConf. In it, I pointed out that people (like me) who can use Rails on green-field projects are incredibly privileged. We get to code using cool technologies in an incredibly agile, reactive environment, producing applications that just work.

But, at the same time, there are a whole lot of people who don’t get to enjoy the world of Rails.

I see these people regularly: I talk at No Fluff symposia around the country, a great set of shows aimed at Java developers. I’m the outsider, normally giving a full day’s worth of talks on Ruby and Rails. And, in a way, it makes me sad to do it, because I often see folks get really excited about the possibilities present in Rails only then to realize that they’ll be going back to work on Monday with no real hope of using what they’d just seen. These folks come to me during the breaks and talk about how bad their world currently is, and how much better it would be if only they could use Rails. If only…

Sometimes, the things that block these people are not technical issues: companies have policies dictating that all work is done in Java (or .NET, or whatever). But many times the road blocks are technical. In their particular environment, with legacy code and legacy databases, they need to be able to do things that Rails just doesn’t support out of the box—more complex schemas, staged deployment, and so on.

So, in my keynote, I asked a simple question. Can we, as a community, do anything to make the lives of these developers easier. Can we find a way of bringing them in out of the cold?

Clearly, I believe the answer is yes. As I said in Chicago, the answer is not to change Rails. Instead, the answer is to use the fact that Rails is based on Ruby, and Ruby gives developers incredible power to extend existing code in directions not anticipated by the original developers. I asked the community to consider creating extensions to Rails to allow currently disenfranchised developers to join the party.

This seems to have touched people’s nerves. Some have reacted violently against the idea, others have welcomed it. It’s an interesting, although ultimately sterile, debate. If people need changes to Rails, Ruby will let them do it.

An interesting example is Greg Houston’s recent blog post, Let ActiveRecord support Enterprise Databases. (Uh oh! He used the enterprise word). He describes a problem he faces trying to use Active Record migrations to maintain a legacy schema. The database-neutral migrations didn’t give him the kind of control he needed, so his post describes how he extended Active Record to do what he wants (including a very simple but nicely powerful trick to make migrations spit out SQL, rather than execute it). And, importantly, he did all this as a regular user of Rails: he didn’t need any changes to the framework or any endorsement of his work by the core team. (He does, however, make a good point at the end of his post. If Active Record had an extension API it would make it easier for these kinds of plugins to survive changes to the core code base.)

Greg’s post is a basic experience report: he took what he needed from Rails, extended it where necessary, and solved a business problem. By blogging, he’s shown other developers just how to solve this category of problem. A .NET developer needing to manage a legacy SqlServer schema could read his post and realize that something that seemed impossible in Rails is actually fairly easy. One more person can join the party.

You could argue that the stuff in Greg’s post is just a simple matter of coding. There’s nothing stopping any .NET developer downloading Rails and making the simple changes he’s suggesting.

But the reality is that you have to have a good idea that this kind of thing is possible before you invest time in the exercise. If you’re a Java or .NET developer looking for salvation, and you spend some time looking at Rails, you’ll currently come across a lot of information explicitly telling you that Rails is never going to address the issues you face—Rails is explicitly not ”enterprise.” And so you’d move on. And you’d be missing an incredible opportunity.

And this is one of the challenges I’d like us, the Rails and Ruby communities, to address. While keeping the Rails core focused on new ways of writing web applications, I’d like others of us to find ways of educating corporate software developers, showing them that they can also bring our technologies into their daily lives—using Ruby as an enterprise glue language and Rails as a viable enterprise web development platform.

Should Rails be everything to everyone?

No.

Should Rails be “marketed” as a replacement for J2EE or .NET?

God forbid.

But there are a whole group of people out there who could be using Rails now but aren’t, simply because they don’t know that it is up to their challenges. And I’d like to try to help these folks.

Let’s not be the only people having fun.

Migrations Outside Rails

I’m about 3 weeks into the rewrite of the Active Record chapters for the new Rails book. In the book, I try to demonstrate Active Record with real, live code. At the same time, I don’t want to run every single piece of code in the context of a Web application. So, I use Active Record stand-alone, without having the rest of Rails loaded. All my demonstration files start:

    require "rubygems"
    require_gem "activerecord"

and then include a call to establish_connection to connect to the database.

At this point, I’m up and running, and I can play with all the Active Record functionality. But… I still wanted to create tables in the underlying database. In the first edition, I used DDL to do this, but in the second I wanted to use migrations.

My first hack was to use the fact that the various schema definition methods are defined both for migrations and in every database connection object. That let me use the following in my code:

    ActiveRecord::Base.connection.instance_eval do
      create_table children, :force => true do |t|
        t.column :parent_id, :integer
        t.column :name,      :string
        t.column :position,  :integer
      end
    end

I was pretty chuffed with this until Jamis Buck (who else) pointed out a more elegant way:

    ActiveRecord::Schema.define do
      create_table children, :force => true do |t|
        t.column :parent_id, :integer
        t.column :name,      :string
        t.column :position,  :integer
      end
    end

As I see more and more people start to use Ruby (and Active Record) as enterprise glue, being able to bring these kinds of Rails goodies to non-Rails applications is a win all around.\