rails build e3f5fd5 failed

The build failed.

CHANGES

... etc.

Am I overreacting or is anyone else a tiny bit annoyed by the size of
these emails? If you're mobile with a GSM card out of UTMS range , you
have around 4-8kB/s, and 70kB per mail is a noticeable wait.

The summary at the top is fine, but I propose a link to the build details.
If someone is interested in the gory details, they can open the link in
a browser.

Forget about me if this is too much of a hassle.

Cheers.

I’m sorry. Here’s some options:

  1. Filter these mails so you never see them (should be easy with Gmail)

  2. Make it so. CruiseControl.rb is open source, and may even have this feature already, and all the CI config is in the rails repo. However, I think some people may like seeing the error output inline.

  3. Help fix the build :slight_smile:

– Chad

1. Best option, since even if the build is broken, bandwidth won't
  let me do much, anyway.

  2. Maybe worth a look at least, but more importantly like you
  mentioned - some people may prefer having the errors inlined.

  3. As in 1, doing a git-pull with a sloppy connection isn't a good
  idea, and the problem will be fixed by the time I even finish the
  fetch. And I don't even consider myself competent enough to even do
  something about the broken build within a reasonable amount of time.
  Unless in the unlikely event that all the Rails developers go on a
  month-long holiday :slight_smile:

  Thanks for the reply and suggestions. Sorry for bringing this up.

If there are three commits—A, B and C, where A breaks the build and B,C are unrelated—I’m wondering why should CI send out emails for errors in B and C when it detected them in A.

In other words, CI should never re-send the same failures: only if they’ve changed. A trivial implementation of this would just be by comparing the standard output. Thoughts?

3. Help fix the build :slight_smile:

If there are three commits—A, B and C, where A breaks the build and B,C are
unrelated—I'm wondering why should CI send out emails for errors in B and C
when it detected them in A.
In other words, CI should never re-send the same failures: only if they've
changed. A trivial implementation of this would just be by comparing the
standard output. Thoughts?

"Don't forget the build is broken", thus, spam you with emails until
someone drags his ass over and fixes the build.

Also, how do you *really* detect it's unrelated? Commit B might use
code changed by A, which broke the build, so you don't really know if
it doesn't fail.

Cheers
-foca

The problem with the "spam emails" approach is that people will treat them as spam. I don't know what the answer is, but I wouldn't be surprised if a lot of the build mails to this group are going right to people's spam folders or being otherwise automatically disposed of.

Mike

I tend to print them out, beat them with a stick before I set them on
fire, unfortunately gmail was unable to automate this process for me.

Seriously though, I think you're right Mike. Actually, I can assure
you I know of at least a few people who have unsubscribed from the
list because of this. And as some find them annoying, plus it probably
doesn't do any good, why not just send them to a separate list where
people who actually _do_ want them can get them?

Eloy

I tend to print them out, beat them with a stick before I set them on
fire, unfortunately gmail was unable to automate this process for me.

Seriously though, I think you're right Mike. Actually, I can assure
you I know of at least a few people who have unsubscribed from the
list because of this. And as some find them annoying, plus it probably
doesn't do any good, why not just send them to a separate list where
people who actually _do_ want them can get them?

+1 to that

+1

I tend to print them out, beat them with a stick before I set them on
fire, unfortunately gmail was unable to automate this process for me.

Seriously though, I think you're right Mike. Actually, I can assure
you I know of at least a few people who have unsubscribed from the
list because of this. And as some find them annoying, plus it probably
doesn't do any good, why not just send them to a separate list where
people who actually _do_ want them can get them?

+1 to that

Eloy

"Don't forget the build is broken", thus, spam you with emails until
someone drags his ass over and fixes the build

The problem with the "spam emails" approach is that people will treat
them as spam. I don't know what the answer is, but I wouldn't be
surprised if a lot of the build mails to this group are going right to
people's spam folders or being otherwise automatically disposed of.

Isn't that why planecrashes happen? :slight_smile:

I tend to print them out, beat them with a stick before I set them on
fire, unfortunately gmail was unable to automate this process for me.

Seriously though, I think you're right Mike. Actually, I can assure
you I know of at least a few people who have unsubscribed from the
list because of this. And as some find them annoying, plus it
probably
doesn't do any good, why not just send them to a separate list where
people who actually _do_ want them can get them?

Actually, this *is* the right list. The motivation stirred up by the
series of CI emails either motivates you to help fix the problem or to
unsubscribe. That sounds about right. If seeing the Rails build broken
is too much of a burden, then perhaps being on the rails-core list
isn't the right place for you.

-Rob

Dude,

This is a list for discussions, and I'm hear to see if people might
have problems with code that I wrote for Rails, not for fun.
Please take your patronizing comments to the playground, you have
added 0.0 to this discussion.

Eloy

Argh, obviously I meant "here".

I agree with Rob that this list is the right place for it. It's easy
for individuals to "opt out" by setting up a mail filter. Takes 2
seconds.

  - Ken

Hey Ken,

Sure it's 2 seconds of work to setup a mail filter, but so is signing
up for a specialized mailing list. So that doesn't make it a good
reason.

I wouldn't be bothered by these CI emails if we'd see one every now
and then. But nowadays, it seems like every other build is broken.
That's the real problem here.

Are people no longer running the test suite before committing? I can
see how you could get the occasional failure for db drivers other than
the one you use, in fact that's how Rails edge used to be. But it
hasn't been like that for a long time now.

Ask yourself, for whom is this info interesting? Not for me, I see
those failures when I run the tests of Rails. Especially before I
commit. So I surely hope that it's not being meant as a tool for devs
to no longer run the full suite yourself, because that's not what it's
there for plus it generates noise. If you want that noise, go ahead, I
just kindly ask the maintainers to take it to a specific list, as many
other projects do.

Cheers,
Eloy

We use hudson, and we're very happy with it.
It can be configured to only email to the person who broke the build.

Aside from that, the annoying part is indeed that the build is broken
every hour.

Thanks for the feedback, everyone. FWIW, this is not new, build
failures used to go to this list [1]

However, not fixing the build promptly is a problem, and one of the
reasons I wanted to reinstate notifications to this list.

On a large/distributed project like Rails - especially one where many
people run the master branch live in their projects - I believe there
are exactly three appropriate responses to a broken build, in order of
decreasing desirability:

1. Fix the failing tests, ASAP
2. Roll back the change which broke the tests, ASAP
3. Comment/disable the failing tests, ASAP

Any other response to a broken build is wrong and lazy, and leads to
broken windows [2]. See my longer rant here: [3]

Now, it is FINE for the build to go red once in a while, especially
for an open source project like Rails. It is not reasonable to expect
the core team to run the full suite before every checkin, on every
supported database and Ruby interpreter. That would be a waste of
their valuable time and effort, and is why we set up CI to do it
automatically.

However, LEAVING the build red is NOT fine. Spamming the list and
annoying people is just a symptom of the real problem, which is not
promptly resolving a broken build. Fixing the symptom by hiding the
problem (sending failure emails to a separate list) is
counterproductive, and only exacerbates the broken-window effect.

Anyway, I've disabled the notifications for now, and the core team has
the ability to disable them anytime they want as well [4]. I didn't
disable them because people complained, but because we just got a new
suite of servers donated to set up an awesome, multi-branch,
multi-interpreter Rails CI environment (thanks EngineYard!), and I
have to complete that migration/setup.

[1] http://groups.google.com/group/rubyonrails-core/browse_thread/thread/2b610e77506812c0#
[2] http://www.pragprog.com/the-pragmatic-programmer/extracts/software-entropy
[3] http://pivots.pivotallabs.com/users/chad/blog/articles/484-how-you-can-learn-to-stop-worrying-and-love-continous-integration
[4] http://github.com/rails/rails/blob/master/ci/cruise_config.rb#L4

The problem is that, unless I'm mistaken, the majority of the people on this list don't have commit rights, so we're unable to do any of these things ASAP. The value of having the notifications sent out to non committers is quite limited, which is why it would be nice to be able to opt out.

Thanks for the feedback, everyone. FWIW, this is not new, build
failures used to go to this list [1]

How about a two stage commit. If CI repo works, then commit to main
repo. If not, mail the commiter that the commit failed, while the
whole world still has a working build. This is to prevent "broken
windows" and the commiter usually has the most information about what
could be broken and how to fix, anyway. Or get people pull always via
a tag LAST_WORKING_BUILD, but that will be less obvious for many.

However, not fixing the build promptly is a problem, and one of the
reasons I wanted to reinstate notifications to this list.

Originally, I was only noting the size. I prefer to read what people
write than what machines spit out.

However, LEAVING the build red is NOT fine. Spamming the list and
annoying people is just a symptom of the real problem, which is not
promptly resolving a broken build. Fixing the symptom by hiding the
problem (sending failure emails to a separate list) is
counterproductive, and only exacerbates the broken-window effect.

I agree. In this case there are actually two broken widows - the
commit and what people pull when they want the latest version. For me,
the large-volume machine-generated mail became a third broken window
in the way I follow what is going on in rubyonrails-core and how I
personally use my mail reader.

Anyway, I've disabled the notifications for now, and the core team has
the ability to disable them anytime they want as well [4]. I didn't
disable them because people complained, but because we just got a new
suite of servers donated to set up an awesome, multi-branch,
multi-interpreter Rails CI environment (thanks EngineYard!), and I
have to complete that migration/setup.

The best thing, as pointed out already, would be to prevent these
things from ever happening.

That aside, I was toying of an idea of a tool like CI, but one that
would run continuous integration tests to find the latest set of
working gems using or being used by rails (plugins, libraries, etc).
Something that could show, for example, if the latest commit to rails
or rspec would break the other. An early warning for the commiter
about which plugins will be broken and who to mail about what was
changed.

Any ideas about this? Something already like this exists?

Cheers.

Perhaps a timeout within which the build should be fixed? I mean it
doesn't take hours for CI to detect the problem. And then mail if fix
doesn't come in time.

I don't believe committers will want to create and subscribe to a
separate list. And this is *their* mailing list. Then again, the rest
of us may want to know if we should "pull" or not.