Hey guys and gals,
I suggested that perhaps we should use Integrity
(GitHub - foca/integrity: Don't mind this, go to http://github.com/integrity/integrity) for the CI for Rails rather than
CC.rb. DHH said he was all for it, but I should bring it before you
all before he made a decision.
I'm open minded. As Koz said, I've done most of the work myself, with
some help from Mike Gunderloy. We (my company) use ccrb mostly by
default, because it used to be the only mature, fully-featured CI
server in Ruby. Apparently that isn't the case anymore, which is a
Good Thing!
If using Integrity means that more people will be on board with
supporting CI (which primarily means making sure Rails Core applies
patches to keep it green), then I'm in favor of it. There are a few
things that ccrb does have going for it that I wouldn't want to lose,
though - I'll mention those later.
I also wrote Cinabox
(GitHub - thewoolleyman/cinabox: Continuous Integration. In a Box!) which makes it
almost a complete no-brainer to set up ccrb on a clean Ubuntu distro
(just two scripts and everything is running, including a working init
script)
So, here's why I like Integrity:
1. It needs less hacking to do interesting things. For example,
multiruby support would be dead simple to setup, which is something we
really need.
Is this really any easier than ccrb? Once you have the multiple
interpreters installed (with multiruby or something like cinabox's
bootstrap_ruby script, that's another discussion), most of the work is
kicking off the build with the proper environment. As Integrity's
homepage says: "as long as your build process can be run from an
unix-y environment and it returns a zero status code for success and
non-zero for failure, then integrity works for you".
For the rails build, this is just ci_build.rb
(http://github.com/rails/rails/blob/9bcf01b23c25e640da7908ac8b1b49fbf7d2e51a/ci/ci_build.rb)
- it doesn't matter what CI tool you use to invoke it, as long as you
can pass some env or param to indicate the interpreter to use. I've
hacked ccrb build scripts to dynamically take this from the ccrb
project name via env var, which is a slick way to do it. That (plus
GemInstaller) is how I handle dynamically building against multiple
rails gem versions without modifying any source code or re-freezing.
2. Adding different sorts of notifiers to it is really easy. Being
able to dump CI reports into IRC, Campfire, etc. without a lot of
hassle would be awesome.
CCRB is also fairly extensible, and it's all Ruby, so it is a wash in
terms of feasability. However, if Integrity already has all these
working and mostly bug-free, then that is a plus.
3. It's really lightweight.
This is actually pretty compelling. CCRB is in rails, and it loads
the rails env for each builder, which is a significant memory overhead
on low-memory servers running several builds.
There are a number of other reasons I prefer it, but these hit the
main technical ones. The others are mostly opinion. 
What do you all think?
Here's the things that I'd like to confirm before I commit to it:
* A working debian-based init script that is shipped with the project.
By working, I mean it works seamlessly on reboots, on killed servers
(no stale pids - e.g. mongrel cluster not mongrel), and standard
start-stop-restart. Cinabox + ccrb handles all this automatically -
e.g. just run setup_ci.rb and all the init script is working. Full
disclosure, the stale pid problem still isn't fixed in ccrb's init
script if the mongrel is killed.
* Updated docs in rails
(http://github.com/rails/rails/blob/9bcf01b23c25e640da7908ac8b1b49fbf7d2e51a/ci/ci_setup_notes.txt)
that explain exactly how to set up integrity. Preferably, this will
be by adding an "integrity" flavor to cinabox - currently the
ruby/ccrb server setup in the previous doc is one line: "Use cinabox
to perform rest of ruby/ccrb setup:", which consists of naught but
downloading cinabox and running two command line scripts.
* Git support: Is git support rock solid? How about for multiple
non-master branches? Because it took a lot of hacking from various
people to get all the known bugs out of ccrb's git support.
* How about git submodules/piston/braid/giternal? Specifically, can
it automatically trigger a build based on changes in a
submodule/pistonized/braidified/giternal? This is completely
unrelated to the Rails build, but something that would make me really
want to switch to Integrity for my other CI projects
* I want to be disciplined about having EVERYTHING repeatable on a
clean Ubuntu server via /ci/ci_setup_notes.txt. In other words, I
DON'T want to take the approach of hacking and manually installing
crap on the CI server and nobody knows what is running or how. I want
to be able to completely blow away the server and have it back up and
running by following the ci_setup_notes.txt exactly. This lets other
people reproduce the rails CI environment on their own, which is
critical to being able to reproduce problems that "only" occur on CI
(which really means all the edge case tests, gem versions, databases,
and libraries that aren't even running on your dev box but are on CI).
It also (by leveraging GemInstaller and having a solid init script)
mostly eliminates the need for people to manually log on to the server
to debug problems - everything should be reproducible in an ubuntu VM.
* all the other mature CI features that ccrb has - reporting of scm
failures, changesets, build failures, control over custom build
artifacts, history of build output and artifacts, etc...
So, I'm totally open to try it given the above caveats. If you want
to do it, get everything set up on a fork of rails, and we'll set them
up to run concurrently for a while on ci.rubyonrails.org (following
the diff of your forked /ci/ci_setup_notes.txt exactly) at a different
nginx virtual host. If Integrity runs smoothly and does everything
ccrb does and more, then why not?
-- Chad