Rails and the Legacy World

I gave what turned out to be a slightly controversial keynote at RailsConf. In it, I pointed out that people (like me) who can use Rails on green-field projects are incredibly privileged. We get to code using cool technologies in an incredibly agile, reactive environment, producing applications that just work.

But, at the same time, there are a whole lot of people who don’t get to enjoy the world of Rails.

I see these people regularly: I talk at No Fluff symposia around the country, a great set of shows aimed at Java developers. I’m the outsider, normally giving a full day’s worth of talks on Ruby and Rails. And, in a way, it makes me sad to do it, because I often see folks get really excited about the possibilities present in Rails only then to realize that they’ll be going back to work on Monday with no real hope of using what they’d just seen. These folks come to me during the breaks and talk about how bad their world currently is, and how much better it would be if only they could use Rails. If only…

Sometimes, the things that block these people are not technical issues: companies have policies dictating that all work is done in Java (or .NET, or whatever). But many times the road blocks are technical. In their particular environment, with legacy code and legacy databases, they need to be able to do things that Rails just doesn’t support out of the box—more complex schemas, staged deployment, and so on.

So, in my keynote, I asked a simple question. Can we, as a community, do anything to make the lives of these developers easier. Can we find a way of bringing them in out of the cold?

Clearly, I believe the answer is yes. As I said in Chicago, the answer is not to change Rails. Instead, the answer is to use the fact that Rails is based on Ruby, and Ruby gives developers incredible power to extend existing code in directions not anticipated by the original developers. I asked the community to consider creating extensions to Rails to allow currently disenfranchised developers to join the party.

This seems to have touched people’s nerves. Some have reacted violently against the idea, others have welcomed it. It’s an interesting, although ultimately sterile, debate. If people need changes to Rails, Ruby will let them do it.

An interesting example is Greg Houston’s recent blog post, Let ActiveRecord support Enterprise Databases. (Uh oh! He used the enterprise word). He describes a problem he faces trying to use Active Record migrations to maintain a legacy schema. The database-neutral migrations didn’t give him the kind of control he needed, so his post describes how he extended Active Record to do what he wants (including a very simple but nicely powerful trick to make migrations spit out SQL, rather than execute it). And, importantly, he did all this as a regular user of Rails: he didn’t need any changes to the framework or any endorsement of his work by the core team. (He does, however, make a good point at the end of his post. If Active Record had an extension API it would make it easier for these kinds of plugins to survive changes to the core code base.)

Greg’s post is a basic experience report: he took what he needed from Rails, extended it where necessary, and solved a business problem. By blogging, he’s shown other developers just how to solve this category of problem. A .NET developer needing to manage a legacy SqlServer schema could read his post and realize that something that seemed impossible in Rails is actually fairly easy. One more person can join the party.

You could argue that the stuff in Greg’s post is just a simple matter of coding. There’s nothing stopping any .NET developer downloading Rails and making the simple changes he’s suggesting.

But the reality is that you have to have a good idea that this kind of thing is possible before you invest time in the exercise. If you’re a Java or .NET developer looking for salvation, and you spend some time looking at Rails, you’ll currently come across a lot of information explicitly telling you that Rails is never going to address the issues you face—Rails is explicitly not ”enterprise.” And so you’d move on. And you’d be missing an incredible opportunity.

And this is one of the challenges I’d like us, the Rails and Ruby communities, to address. While keeping the Rails core focused on new ways of writing web applications, I’d like others of us to find ways of educating corporate software developers, showing them that they can also bring our technologies into their daily lives—using Ruby as an enterprise glue language and Rails as a viable enterprise web development platform.

Should Rails be everything to everyone?

No.

Should Rails be “marketed” as a replacement for J2EE or .NET?

God forbid.

But there are a whole group of people out there who could be using Rails now but aren’t, simply because they don’t know that it is up to their challenges. And I’d like to try to help these folks.

Let’s not be the only people having fun.

Migrations Outside Rails

I’m about 3 weeks into the rewrite of the Active Record chapters for the new Rails book. In the book, I try to demonstrate Active Record with real, live code. At the same time, I don’t want to run every single piece of code in the context of a Web application. So, I use Active Record stand-alone, without having the rest of Rails loaded. All my demonstration files start:

    require "rubygems"
    require_gem "activerecord"

and then include a call to establish_connection to connect to the database.

At this point, I’m up and running, and I can play with all the Active Record functionality. But… I still wanted to create tables in the underlying database. In the first edition, I used DDL to do this, but in the second I wanted to use migrations.

My first hack was to use the fact that the various schema definition methods are defined both for migrations and in every database connection object. That let me use the following in my code:

    ActiveRecord::Base.connection.instance_eval do
      create_table children, :force => true do |t|
        t.column :parent_id, :integer
        t.column :name,      :string
        t.column :position,  :integer
      end
    end

I was pretty chuffed with this until Jamis Buck (who else) pointed out a more elegant way:

    ActiveRecord::Schema.define do
      create_table children, :force => true do |t|
        t.column :parent_id, :integer
        t.column :name,      :string
        t.column :position,  :integer
      end
    end

As I see more and more people start to use Ruby (and Active Record) as enterprise glue, being able to bring these kinds of Rails goodies to non-Rails applications is a win all around.\

Rails Guidebook

RailsConf 2006 is in its final day, and I just realized that I hadn’t blogged on Rails Guidebook

The Guidebook was a preconference event. For a day before the conference proper started, attendees got an introduction to all the stuff that’d be hearing about in the following days. It was a combination of a tutorial and a glossary. We didn’t charge. Instead, the cost of admission was a contribution to charity.

The Rails community raised over $8,000

This is wonderful stuff, and Mike and I thank everyone who made it possible: the Ruby Central folks who agreed to add a day to the agenda, Jay Zimmerman for organizing the facilities, the core team members and other experts who turned up to help, Mike and Nicole for doing all the Guidebook logistics, and, more importantly, the folks who turned up and contributed.

A Challenge

If you’re already organizing a conference, adding a guidebook day ahead of it isn’t a great incremental cost. So, here’s a challenge to conference organizers. Let’s start a tradition of these types of charitable events. Let’s make a point of sharing some of our success with others. Not only does it do some good—it’s fun.

Glue That Doesn't Set

Back in the old days of computing—those days where I could still see my knees when standing up—folks used to talk about languages such as Perl as glue languages. What did they mean?

Well, Perl is a great general purpose programming language. But it is exceptional at doing something that no previous language could do well: Perl made it easy to glue together other stuff. Data, programs, external systems—all these things could be integrated using Perl. Thanks to its integration with the underlying operating system, and its amazing support for text processing, Perl made a name for itself as an integrator par excellence. Want to take the last 20 lines of an error log, convert them to HTML, and upload the result to your web server? Perl could do that in a handful of lines of code. Need to screen-scrape a book’s sales rank from Amazon and writing it into a local MySQL database. Perl to the rescue.

Perl let us glue stuff together. It was magic.

But there was a problem. Perl is a great language. It was my scripting language of choice before I discovered Ruby. But even on my most charitable days I could never really claim that these Perl programs were exemplars of readability. And, in turn, this lack of readability made it hard for me to pick one of them up six months after I’d written it and make some change—even small alterations could involve long periods of head-scratching as I tried to work out exactly what the entries in my hash of arrays of hash references actually contained.

As a result, I found that the longer it was since I last looked at a program, the harder it was to maintain and extend it. The programs effectively became less malleable and more viscous. It became stickier; I got mired in a gooey mess of details. Leave a program long enough, and I’d often decide to rewrite it rather than attempt some piece of surgery: the program had become totally inflexible.

The glue had set.

Now Ruby is also a glue language. Just like Perl, Ruby works and plays incredibly well with external data, resources, and programs. I can do everything with Ruby that I used to do with Perl—I can use it to integrate all kinds of stuff just as easily as I used to be able to with Perl. But, there’s a major difference. I’ve discovered that, over time, my Ruby programs stay malleable and easy to change. Ruby is the glue that doesn’t set.

Glue and the Web

That’s all very well, but what does writing little utility scripts have to do with real-world web programming? A whole lot, as it turns out.

As we move into a world where the network is the computer, applications change in character. Rather than writing free-standing islands of code, developers are increasingly writing programs that knit together other, existing web resources. Want to annotate a map with pictures of famous landmarks? Maybe you’ll want to use the Google Maps and Flikr APIs to carry most of the load for you. Want to let your readers know when their product has shipped? Why not generate the information as RSS and let their aggregator client do all the gruntwork of displaying the result? As we move forward, more and more of our applications will be less and less like one-man bands and more like orchestra conductors.

Our applications are increasingly becoming glue code. And, if that’s the case, I’d rather write them using a glue that doesn’t set over time.

It was standing-room only at Mike Clark’s Capistrano talk: one of the first talks at this year’s RailsConf. Latecomers stole chairs from other rooms and fought their way to some free floor space to hear Mike talk about the way we deploy Rails applications. I’d just finished giving the first keynote.

One of my topics was making deployment in Rails easier and more amenable to corporate use. I suspect from the interest in my talk, followed by the overwhelming interest in Mike’s talk, that deployment will be one of the hot Rails topics of the coming year.

Direct Shipping Record

I really like that fact that Andy and I still ship direct orders in-house: we’re not currently using a fulfillment service. It means we can take that little bit of personal care with the packing, and we can correct minor order problems on-the-fly. We also get to see first-hand just what is going on. We see a surprising number of orders for multiple books, and it’s fun to track which books tend to sell with others. We also get the occasional surprise, like the order in the picture for 12 titles. Even more surprising is the fact that it was from Australia.

Agile Toolkit Podcast : RailsEdge 2007 - Dave Thomas - Monoculture, Music and Erlang ... Oh My

Link: Agile Toolkit Podcast : RailsEdge 2007 - Dave Thomas - Monoculture, Music and Erlang ... Oh My

Bob Payne recently interviewed me at the Reston No Fluff Just Stuff show. We chatted about agility, the Dreyfus model, Java, Ruby, Rails, and functional languages, all against the gentle backdrop of the hotel’s Muzak (so at least you can tell there are no edits…).

Rails is More...

One of the emerging benefits of attending a Rails Studio is membership in the Studio mailing list. The posts in that list are consistently on target.

A few weeks back, Eric Knapp posted a long message which I feel nicely sums up the difference Rails makes. It isn’t an issue of Rails being substantially different to the alternatives. It’s more that the lowered barriers to entry enable a broader base of people to benefit.

Anyway, with Eric’s permission, here’s the body of his post. It was titled

Rails is more than I thought

This past Friday I had a meeting with a client who is the director of educational research and IT at a medium sized public school district. I’m sure that you are all aware that school districts are under intense pressure to cut their budgets and expenses. My client can no longer replace any IT employee when someone leaves. They are down to one network administrator (!) and two programmers. Their programmers are just maintaining an old legacy system and really don’t do any new development anymore. New development is done by student interns from my Java program at my college. My students really struggle with all kinds of limits that are set by the district’s policies, procedures, and equipment.

My client has also been struggling with how to move to the future. He is under lots of pressure to cut even more deeply. There are those within the district who want to end all development of new things and only use off-the-shelf applications, as-is, with no customization.

My client is not an IT person and he knows nothing about programming. But, he is a very smart man and is very perceptive. My meeting started with an overview of his current development efforts with my students. Then I started sharing rails with him. I like rails and I tend to be enthusiastic. About ten minutes into my presentation he interrupted me and said, “So, this could save my IT department, couldn’t it?”

I was really taken off guard by this! I simply answered, “Yes.” He took over our meeting and started talking about all the things that he and his people would like to do, but just can’t afford. He talked about all the ideas that are shared with him by his staff and teachers that also are just too expensive. My interns have started to help him and we have worked very hard at trying to bring Java to his department. But, they are just too slow, everything takes too long to be viable anymore. I believe that we are now going to give up on Java.

We now have scheduled a kick-off meeting for maybe bringing Ruby on Rails to this school district. Later he told me that this has given him a little bit of hope. I might be able to have good geek fun while helping children at the same time.

This is exciting stuff! Rails appears to have reached some sort of critical mass of productivity where ideas that live in the hearts and minds of most people have a chance of getting out and onto the screen. I’m starting to ask students and friends what those ideas are. I am getting really interesting answers.

I hope I’m not making too much of this, there is still lots of work to be done and we will have to get grants to do it. But, there are possibilities now where there weren’t any. There is a little bit of hope. Sounds good to me. Many thanks to the rails community and to this group for helping me get to this point.

E Pluribus Anum

Went to a school production of The Wizard of Oz last night. Good fun. However, the Wizard had a slight hiccup. The line is supposed to go

Therefore, by virtue of the authority vested in me by the Universitartus Committiartum E Pluribus Unum, I hereby confer upon you the honorary degree of ThD.

Unfortunately, it came out E Pluribus Anum.

Which may well be the official motto for the wisdom of crowds…

R-Rails Gets R-Recognized

No, I’m not stuttering. This evening, at the Software Development Jolt Awards at SD West in Santa Clara, Agile Web Development with Rails received the Jolt Award for best technical book, and Rails 1.0 won the Jolt for best web development tools.

I have to say, when I saw that both the book and the framework were up for awards, I was nervous, thinking that the judges might not want to vote for Rails in two separate categories. The fact that they did says a lot for the way Rails is now a mainstream player in the enterprise software world. It’s clear the tipping point is behind us now, and we’re a player.

Congratulations to DHH on his Jolt. And congratulations to the Pragmatic Bookshelf: this is our second Jolt Award in three years.

And, mostly, thanks to the Ruby community for providing the environment in which all this can happen. I’m really proud of us all.

Annotate Models Plugin

Rails model files contain no information on the tables they represent. This is a good thing in general, because it reduces duplication—add a column to a table, and there’s no configuration to update in the model.

However, when you’re writing code, it’s sometimes nice to be able to see just what attributes a model has.

Enter annotate models, a really trivial Rails plugin I hacked up in the plane back from the first No Fluffof the year. The plugin adds a comment block to the top of each model file, documenting the schema. If you update the schema, run it again and it updates the comment.

  # Schema as of Mon Feb 27 00:55:58 CST 2006 (schema version 7)
  #
  #  id                  :integer(11)   not null
  #  name                :string(255)
  #  description         :text
  #  image_location      :string(255)
  #  price               :float         default(0.0)
  #  available_at        :datetime
  #

  class Product < ActiveRecord::Base

    validates_presence_of :name, :description
    . . .

Install using:

  > script/plugin install http://repo.pragprog.com/svn/Public/plugins/annotate_models

and run with:

  > rake annotate_models

It only handles models directly under app/models. And, as it’s new, you’d be advised to back up your model files before running it.

Rails Guidebook is Full

I’m blown away!

The Rails Guidebook, our day of pre-conference training before RailsConf, is now officially full. Chad Fowler and Jay Zimmerman, who are handling logistics, twice negotiated to get bigger rooms, but we’ve had to draw the line at 150.

Yes, 150!

One hundred and fifty people are coming to learn about Rails, and have agreed to donate at least $40 to charity. Many folks have already donated a lot more. (And, remember, the three biggest donations get some VIP treatment.) I’m so, so proud of our community for making this happen. Ruby folks have always been nice to be around. To see them contributing back to the outside world like this just confirms it.

Thank you all.

(If you are coming, remember to get your donation in, and bring your receipt. See you there.)

RailsConf and the Rails Guidebook

RailsConf in Chicago (June 22-25) promises to be an exciting affair. And it looks as if there’ll be a number of people coming who are fairly new to Rails.

So Mike and Nicole Clark and I were sitting down after a day of teaching a Rails Studio, and we came up with an idea. We’ll be running a one-day course before the conference. It is designed to bring newcomers to Rails up to speed, both with the framework and with Ruby.

We’re calling it the Rails Guidebook. Think of it as the difference between between turning up somewhere exotic knowing nothing about the language, customs, and places to visit, and turning up having spent a small amount of time reading a guide book. Hopefully, a day spent with us will make the rest of the conference more enjoyable (and informative).

We’re not charging for the Guidebook—it’s our way of saying “thank you” to the community. Instead, we’re asking folks to donate to charity before they come. Space is limited, so you might want to book early. (And, the top 3 contributors will get some perks…)

Link:

Andy and I have had to cut back on conferences this year, but it looks like we’re making an exception for this one.

Sharing External ActiveRecord Connections

I had a problem with a deployed Rails application today that Mike Clark helped me track down.

It started after Mike and I migrated a Rails app to a new, zippier server. Everything looked great (and the app ran a lot faster). However, after a while, I started getting problems with a database. To explain what went wrong, I need to chat a little about the application.

This piece of code is an attempt to migrate functionality away from a particularly nasty set of old CGIs, web applications, bailing wire, chewing gum, and other historical crud. There’s a lot of existing functionality, and I decided not to try to port it all across to Rails on day one. Instead, I gave the Rails application its own database to play with, and gave it access to the old legacy database for when it needed to look up stuff in the existing system. To do that, I created a set of ActiveRecord classes for the legacy tables, and explicitly connected them to the legacy database, rather than the default Rails application database. A typical legacy model class looked smething like this:

  class LegacyOrder < ActiveRecord::Base
    set_table_name  "orders"
    set_primary_key "o_id"

    establish_connection "legacy_#{RAILS_ENV}"
    ...
  end

(The ”legacy_#{RAILS_ENV}” stuff lets me have development, test, and production versions of the legacy database in my database.yml file.)

So far, so good. I had four different legacy tables I mapped this way, and everything worked fine.

However, after a while, other applications accessing the legacy database started failing, claiming they couldn’t connect. Sure enough, it turned out that there were hundreds of active connections to that database: the database wasn’t going to accept any more.

The problem was that I was creating way too many connections to the legacy database. Each spawned Rails fcgi process established a separate connection to the legacy database for each legacy ActiveRecord class. That factor of four killed us.

But fixing it also raised an interesting question: in my original code, I established a separate connection for each legacy table in each running Rails process. This was clearly dumb. But what’s the best way of sharing a single connection between all of the legacy model objects in a Rails instance?

Some chatting on #rails-core came up wih three suggestions:

1. Create the connection once (perhaps in one of the environment files), and then assign it to each of the legacy AR classes:
   

     legacy_connection = ActiveRecord::Base.mysql_connection(...)
   
     LegacyOrder.connection = legacy_connection
     LegacyLineItem.connection = legacy_connection
     ...
   

   
   


2. Establish the connection in one model, then use it in the others
   

    class LegacyOrder < ActiveRecord::Base
      establish_connection ...
    end
   
    class LegacyLineItem < ActiveRecord::Base
      connection = LegacyOrder.connection
    end
   

   
   


3. Somehow make all the legacy models share a common parent, and have that parent own the connection.

Options (1) and (2) seemed messy: they added a bunch of coupling to the code where it wasn’t really justified. Option (3) seemed like the way to go. The only problem was that no one on #rails-core was sure it would work…

It turns out it does.

  class LegacyBase < ActiveRecord::Base
    establish_connection ...
  end

  class LegacyOrder < LegacyBase
     ...
  end

  class LegacyLineItem < LegacyBase
    ...
  end

The parent class creates a single connection to the legacy database, and all the child classes share it. The reason it works is some smarts in Rails.

It turns out that Rails does just about everything lazily. That includes connecting to databases and reflecting on tables to extract the schema (needed to build the internals of the models). This improves performance, but it also makes this hack possible. In general, you’d expect the LegacyBase class to map to a database table called legacy_base. It would, if we ever tried to use it to access data. But because we don’t, and because Rails only reflects on the table the first time a data access occurs, we can safely create an ActiveRecord class with no underlying database table.

This scheme lets me specify the legacy connection once, and share that connection between all my legacy models. It’s tidy, expressive, and saves resources.

The Water Fountain Methodology

From a Rails irc channel (htonl is Scott Barron)…

htonl:  yeah we’ve canned agility here
htonl:  we’re doing the water fountain technique 
htonl:  we mostly just spew code up in an arch and it falls down the drain