[Provocation] YAGNI, 37S, Rails, and AOL

rubyonrails-talk
1

I just had a thought that's so obvious, in retrospect, I can't believe it took me three years to think of it.

I don't have time to properly defend the premise right now, but I'd like people to attack it anyway, so when I do write about it later, I can do a better job. This should really go on a blog but I'm midway through setting up a new blog server, so, oops, bad timing. It'll end up there, though.

Basically: 37Signals apps, and therefore Rails, are heavily opinionated in the YAGNI direction. Not just "simple is better", with an Apple design aesthetic, but "As little infrastructure as possible to get the job done".

They pride themselves on how few features they have, because they've got the minimal set of features that make, say, to-do lists *possible*, instead of a complex set of features that make to-do lists maximally flexible.

I just realized I've seen this movie before, and it was called AOL.

We used different words, but they meant a lot of the same things. We weren't talking about "YAGNI", but we were talking about "basic users" vs "power users". We didn't target power users. In fact, if you ever made the mistake of pitching an idea with the word "powerful", it would be shot down instinctively with "We don't target power users".

We didn't do feature creep, because we could see that most users don't use most features. We didn't do options, because we could see that 96% of options remain set at their default values. Etc. Not only ain't you gonna need it, but we have metrics that prove that you ain't gonna use it.

So if it needed user configuration or more than a few clicks to be useful, we just didn't implement it.

The problem is this: Everybody becomes a power user, for some value of "power". Everyone outgrows super-simple - and if they don't, their kids do, because they start at a higher baseline.

Fifteen years ago, a "power user" was someone who knew enough about computers to hook up a modem, open the network config window and get connected to an ISP, then configure their Internet mail server. Now we call that a "person". Sometimes they pay for "installation". Sometimes they ask their teenager. They never use AOL for it.

"So easy to use, no wonder it's #1!" A meaningless non-sequitur and a corporate philosophy.

It tanked once better alternatives came along.

Just something to think about. There are many holes in this, and I can already think of a few, but I'd love to hear from the peanut gallery.

2

Jay Levitt wrote:

Basically: 37Signals apps, and therefore Rails, are heavily opinionated in the YAGNI direction. Not just "simple is better", with an Apple design aesthetic, but "As little infrastructure as possible to get the job done".

YAGNI and opinionation make Rails easy to learn, and easy to get projects off the ground. But customization is not hard in Rails when you /do/ have a different opinion or /do/ need it.

So I view more complex and less opinionated frameworks as a kind of premature optimization. With Rails you can put effort into the hard and special stuff and let Rails takes care of the rest.

3

I just had a thought that's so obvious, in retrospect, I can't believe it took me three years to think of it.

Very interesting stuff, Jay. I don't necessarily accept the idea that Rails is like AOL, or that each's users are like the other's, but I'm going to give it some more thought.

Fifteen years ago, a "power user" was someone who knew enough about computers to hook up a modem, open the network config window and get connected to an ISP, then configure their Internet mail server. Now we call that a "person".

I think that doing the equivalent today would still take a "power user." My wife couldn't do it. My son probably could, and his generation will all be our "power users," but I think that by the time they take over, no one will need to do stuff like that. It will no more take a power user to connect to the Internet than it would take a power driver to merge onto the freeway.

///ark

4

The first time I heard about YAGNI was long before Rails. It comes from agile programming practices and, XP in particular.

It's not about power users vs. basic users, it's a mantra to quell the natural tendencies of software developers to add complexitity for the love of complexity.

And it's really a statement not so much about whether to do something than when.

The philosophy is to add features when you prove that you need them not just when you think you might.

Agile projects do things in small steps, discover the real requirements, and implement incrementally testing as they go. It's the right way for many teams, and it's particularly right for a large community framework project like rails.

In the case of Rails, even if the community as a whole hasn't discovered that they really do need your favorite feature, you're free to develop an extension or plugin for your own use, and if you care to submit it for others to use as they see fit. And if it's really valuable, it might just be made part of or adapted to become part of the core as several Rails plugins have.

5

As a resident of Massachusetts, which has the highest per-capita accident rate in the U.S., not to mention secret additives in the water supply that seek and destroy all neurons related to the concept of "merging", I'd like to say that that is a horrible, horrible example.... :slight_smile:

6

The first time I heard about YAGNI was long before Rails. It comes from agile programming practices and, XP in particular.

True enough. I'm talking more about "the philosophy as espoused by people in the Rails community" than either Rails or Rails apps in particular. I'm making the (intentionally overstated) claim that we often rationalize our way into not doing important things.

(And by "we", I mean you guys, because I haven't written a *damn* bit of code that matters. I'm too busy philosophizing on newsgroups. Priorities, you know.)

The philosophy is to add features when you prove that you need them not just when you think you might.

Ah, but:

- Where is the balance between "wait till you've proven you need it" and "knowing from experience that you will"? Building a database with indexes is, in a sense, premature optimization. Reductio ad absurdum, so's defining a text field to be longer than the longest string in your test data.

- How do you prove you need it when you don't have it? Some of the best features are ones nobody asked for. For that matter, you haven't proven anyone needs your web site in the first place. Chicken, egg, etc.

- What about emergent behavior? IMs were around for decades. They were kinda handy. Ditto address books. But combine them into buddy lists, and ZOMG you changed the game. Yet buddy lists wouldn't have happened without either IM or address books being developed first, even though neither was a particularly important feature.

7

- Where is the balance between "wait till you've proven you need it" and "knowing from experience that you will"?

You can never know that you -will- need something. One of the most important points Kent Beck made in the original XP book is that you could be out of business before that future need arises. The agile company stays in business by addressing all current needs before trying to predict future needs.

Building a database with indexes is, in a sense, premature optimization.

Right, if a table doesn't need indexes right now, then there's no need to add them. Doesn't hurt, though, because it's so easy.

- What about emergent behavior? IMs were around for decades. They were kinda handy. Ditto address books. But combine them into buddy lists, and ZOMG you changed the game. Yet buddy lists wouldn't have happened without either IM or address books being developed first, even though neither was a particularly important feature.

I strongly disagree. I'll bet storing addresses is one of the most important things most people's computer does! There was indeed a need for address books. That wasn't trying to predict the future. IM has changed the way many people communicate. I think it's huge, too.

///ark

8

- Where is the balance between "wait till you've proven you need it" and "knowing from experience that you will"?

You can never know that you -will- need something. One of the most important points Kent Beck made in the original XP book is that you could be out of business before that future need arises. The agile company stays in business by addressing all current needs before trying to predict future needs.

OK, so stepping back a few frames of reference and arguing from absurdity:

- How do you prove you need a text field to be longer than your longest piece of test data?

- How do you prove the world needs your web site in the first place?

- Isn't one of your current needs predicting future needs? If not, what good is an agile buggy-whip manufacturer?

(My own answer: There's no real objective line, and if we knew exactly where to draw it we'd all be rich; hence the duality of everything and the tension between experience and YAGNI. But since you're arguing from a purer agile/XP perspective, I'm curious to hear your take.)

- What about emergent behavior? IMs were around for decades. They were kinda handy. Ditto address books. But combine them into buddy lists, and ZOMG you changed the game. Yet buddy lists wouldn't have happened without either IM or address books being developed first, even though neither was a particularly important feature.

I strongly disagree. I'll bet storing addresses is one of the most important things most people's computer does!

Well, admittedly, I was thinking of the AOL address book, which was particularly useless at the time - it was a simple two-field database of real name and e-mail address, with no search, no integration, no auto-fill, just your basic CRUD.

There was indeed a need for address books. That wasn't trying to predict the future. IM has changed the way many people communicate. I think it's huge, too.

But remember - IM didn't really change the way people communicate *until* buddy lists. You were around back then; did you IM nearly as much pre-buddy-list? I wish I could go back and get the stats, but intuitively, I'd venture that buddy lists drastically increased the number of IMs sent.

At the time, I had the sense that IMs were a slight simplification on e-mail. If I had something short and more urgent, I'd IM it; if it was longer, less urgent, or the recipient wasn't online, I'd e-mail it. (Keep in mind this was intra-AOL mail, which was always delivered instantaneously and had recipient verification, so there wasn't any of the uncertainty of internet mail delivery.)

There was a difference in degree, but not a difference in kind. That came later.

9

and quoted Jay Levitt

>> - Where is the balance between "wait till you've proven you need it" and >> "knowing from experience that you will"?

It's a question of discipline, and knowing from experience that often what you think you need from "experience" or "conventional wisdom" quite often isn't what you ultimately need.

To make an analogy, in our legal system we expect prosecutors to honor the discipline that some one is "innocent until proven guilty" even though their experience tells them that "everyone" is guilty.

When that discipline is broken we get things like the Duke Lacrosse case.

> You can never know that you -will- need something. One of the most > important points Kent Beck made in the original XP book is that you > could be out of business before that future need arises. The agile > company stays in business by addressing all current needs before > trying to predict future needs.

OK, so stepping back a few frames of reference and arguing from absurdity:

- How do you prove you need a text field to be longer than your longest piece of test data?

This is a bit of an absurd question. What I try to do is to defer decisions as long as possible. This means using languages, and tools which allow late binding, and changes with minimal cost.

That's why we use database managers which can handle changing field lengths, and languages like Ruby which, unlike say C, or Cobol don't force arbitrary limits on things like string lengths.

I value tools which allow me to refactor an application as the requirements emerge and change.

I was a bit taken aback a week or so ago here during a discussion over whether has_and_belongs_to_many was 'obsolete' when some one said that they ALWAYS use has_many :through whether they had attributes on the association or not, basically 'just in case' they ever did, and damned habtm saying that "that leads to refactoring," I didn't comment at the time, but it struck me that refactoring is a good thing, and tools/languages which make it easy is a very good thing. That to me is the real value of dynamic languages like Ruby.

- How do you prove the world needs your web site in the first place?

You really can't which is the real point. That's the 37 Signals "Getting Real" message isn't it. You start out small, get some users to actually use it, and grow the function (and the investment in the app) over time.

The danger of doing "experienced based" chunks-o-function planning is that you over-invest before you've got the proof that you can successfuly predict the features which are needed to make the project successful.

- Isn't one of your current needs predicting future needs? If not, what good is an agile buggy-whip manufacturer?

(My own answer: There's no real objective line, and if we knew exactly where to draw it we'd all be rich; hence the duality of everything and the tension between experience and YAGNI. But since you're arguing from a purer agile/XP perspective, I'm curious to hear your take.)

If I could accurately predict things, I'd have made a fortune by now on sports-betting, investments, or something else. It's very hard to predict long term which is why the approach of taking small incremental steps tends to lead towards better return on investment by selecting more quality investments in effort, and using technologies which support agile moves like refactoring allow re-writing code to be viewed as a further investment or debt-repayment rather than writing off sunken costs.

10

- How do you prove you need a text field to be longer than your longest piece of test data?

All needs come from customer requirements. They would define what the longest text field needed to be - not the test data.

- How do you prove the world needs your web site in the first place?

Because a customer is willing to pay me money to do it. Again, the need is defined by the requirement.

- Isn't one of your current needs predicting future needs? If not, what good is an agile buggy-whip manufacturer?

Software change is way easier than physical plant change. That's part of what makes agile work, I think. We embrace change.

(My own answer: There's no real objective line, and if we knew exactly where to draw it we'd all be rich; hence the duality of everything and the tension between experience and YAGNI. But since you're arguing from a purer agile/XP perspective, I'm curious to hear your take.)

I know it sounds "pure," but it's borne of experience, really. If you don't need something now, why spend any time on it? Spend the time later ... IF necessary.

Yeah, I think we're talking across each other with the IM/buddy list thing. Actually, I myself have never used IM much. I guess the equivalent for me would be GEnie chat rooms, around 1985.

I'd venture that buddy lists drastically increased the number of IMs sent.

I think you're right.

But I don't think I understand your point. How does the IM/buddy list thing relate to agile?

[Actually, I just realized that this has nothing to do with Rails, so I'll just hit Send and bow out. :slight_smile: ]

///ark

11

- How do you prove you need a text field to be longer than your longest piece of test data?

All needs come from customer requirements. They would define what the longest text field needed to be - not the test data.

Ah, interesting - so you're looking at this from the perspective of a hired consultant, not a startup developing their own app. Yes, that's very different. Presumably, the customer knows what the hell they're doing, and if they don't, you still get paid, so it's all good.

Software change is way easier than physical plant change. That's part of what makes agile work, I think. We embrace change.

Excellent point, and Rick makes it too - Ruby, modern databases, and Agile in general make it much easier to change your mind that older technologies. There's a lot less "lock in". Need to add or resize a field? No problem.

(Does that still hold true for ginormous databases? We always used databases that could theoretically do this, but in practice, you couldn't keep a live, multi-terabyte, OLTP database up and running while you did it, and even if you did, there were internal inefficiencies to the way the new data was stored - page splitting and such. I get the impression that that's more of a solved problem now.)

I know it sounds "pure," but it's borne of experience, really. If you don't need something now, why spend any time on it? Spend the time later ... IF necessary.

So you use your experience to know whether or not you should use your experience. I like it. Very meta.

But I don't think I understand your point. How does the IM/buddy list thing relate to agile?

Just a useful metaphor. Pre-buddy-list IMs were kinda nice, but didn't do anything that e-mail couldn't do; in an agile world, they may not have been developed. And AOL address books were utterly useless, and didn't do anything a text file couldn't do, so again, why bother developing them? These were great candidates for YAGNI. Yet, without them, there was no way to get (mentally, at least) to buddy lists, which, it turns out, *everyone* wanted, and then needed, and that saved the company for a while.

Even if AOL had been written in an agile language and framework, where the cost of change was tiny, the very idea of buddy lists might not have occurred without those intermediate, YAGNI-style features. Sometimes you have to go do something you don't need to discover what you really do need.

It's vaguely related to the "irreducible complexity" fallacy of Intelligent Design, where people say "the eye must have been designed, because half an eye is useless."

Well, half an eye turns out to be ever-so-slightly useful - and gives you just enough evolutionary advantage to survive long enough to develop an eye. Still, if you were to tell me that you could, with a bunch of work, develop a feature that I hadn't asked for and would give me 0.05% advantage over my competitors, I'd say YAGNI.

So I guess I'm positing a Wikipedia WP:Ignore all rules analogy for Agile. As they put it, "If a rule prevents you from improving or maintaining Wikipedia, ignore it."

[Actually, I just realized that this has nothing to do with Rails, so I'll just hit Send and bow out. :slight_smile: ]

It does, but it's pretty tangential. It really does belong on a blog, and I really am going to put that up today, for some value of "today". (See also: The sun never sets on the British empire, because the British empire is in the east, and the sun sets in the west.)

It's more about the Rails community and dogmatism than Rails coding. I see too many people parroting "YAGNI" without realizing they just might.

12

Well said and worth repeating with little angle brackets in front.

13

> All needs come from customer requirements. They would define what the > longest text field needed to be - not the test data.

Ah, interesting - so you're looking at this from the perspective of a hired consultant, not a startup developing their own app.

The "customer" is not necessarily outside the company.

So you use your experience to know whether or not you should use your experience. I like it. Very meta.

I don't see how it has anything to do with experience, actually. It's not that difficult to differentiate between current and future needs.

It's vaguely related to the "irreducible complexity" fallacy of Intelligent Design, where people say "the eye must have been designed, because half an eye is useless."

I had actually written a few paragraphs making that exact same point, but not so well! :slight_smile:

So I guess I'm positing a Wikipedia WP:Ignore all rules analogy for Agile. As they put it, "If a rule prevents you from improving or maintaining Wikipedia, ignore it."

Wikipedia doesn't have a release date.

It's more about the Rails community and dogmatism than Rails coding. I see too many people parroting "YAGNI" without realizing they just might.

"Just might" isn't a customer requirement.

///ark

14

I think that's probably why Rails is so popular and attractive. It allows you to make bigger investments without bigger commitment. I can "invest" in something that I know from experience I'll need to do, spend less resources, and if it ends up being unnecessary I've lost less. If it ends up being the wrong way to go, then I can easily invest a little more and change my direction.

Actually, that's probably why better and newer tools emerge in the first place: to make our investments more efficient.

--Jeremy

15

I misunderstood what you were saying - sorry!

///ark

16

And yet it's an interesting distinction nonetheless, so I won't let you take it back :slight_smile:

17

I just had a thought that's so obvious, in retrospect, I can't believe it took me three years to think of it.

And I just got time to reply to a three week old thread....

They pride themselves on how few features they have, because they've got the minimal set of features that make, say, to-do lists *possible*, instead of a complex set of features that make to-do lists maximally flexible.

I think this is true to a point, and for something like the apps it makes sense. I'm not sure it makes sense for Rails, though.

There are two kinds of complexity: avoidable and necessary. The former arises because the problem is overthought, you have more capabilities than you need, etc. The latter arises because the problem is actually complicated. The question I'm staring at is how to deal with necessary complexity.

If you imagine a platform as a set of minimum functionality plus support for the ways people want to extend it, you wind up with a natural tension in how to handle the support for anything beyond the core. On the one hand, you can build the functionality into the product, and on the other hand you can provide extension mechanisms. Rails has taken the latter path.

The problem with building everything into the core is you risk an overengineered pile of features that many people don't need. Exhibit HTTP request. Rails avoids this, but...

The problem with relying on extensions is you get a complex and subtly incompatible pile of reinvented wheels. Exhibit B: CPAN, complete with 5 hours research necessary to understand all 19 implementations of Hello::World. Rails' authentication seems to be reaching this point, complete with multiple auth plugins that repeat the same security holes*.

The community's common response to complaints about the latter is "YAGNI," but what people often mean is "I didn't need it, so you shouldn't either." This is a bit of a perversion of "YAGNI" as practiced in Agile methods, which could more accurately be described as "The customer doesn't need it until they tell you they do, in which case, they probably do."

You can probably trace this disconnect to the fact that the Rails core team is, on some level, the only customer of the product: they're the ones paying for development and the ones who determine the needs for the product.

To be clear, I'm not arguing that one approach or the other is right in the general case. I do think that knee jerk arguments (in either direction) aren't helpful. There are benefits and costs to each approach and it's probably worth understanding them in your situation**.

* FWIW, I'm anticipating a "Rails isn't secure" blog flamewar because authentication currently requires too much configuration and relies on too much generated code.

** Also,<http://www.jwz.org/doc/worse-is-better.html&gt;

-faisal