Integrity for CI instead of CC.rb?

Hey guys and gals,
I suggested that perhaps we should use Integrity
(http://github.com/foca/integrity) for the CI for Rails rather than
CC.rb. DHH said he was all for it, but I should bring it before you
all before he made a decision.

So, here's why I like Integrity:
  1. It needs less hacking to do interesting things. For example,
multiruby support would be dead simple to setup, which is something we
really need.
  2. Adding different sorts of notifiers to it is really easy. Being
able to dump CI reports into IRC, Campfire, etc. without a lot of
hassle would be awesome.
  3. It's really lightweight.

There are a number of other reasons I prefer it, but these hit the
main technical ones. The others are mostly opinion. :slight_smile:

What do you all think?

--Jeremy

Hey guys and gals,
I suggested that perhaps we should use Integrity
(http://github.com/foca/integrity) for the CI for Rails rather than
CC.rb. DHH said he was all for it, but I should bring it before you
all before he made a decision.

Almost all the work on the CI server to dahas been done by chad
woolley, so from my perspective the decision is basically completely
up to him :).

So, here's why I like Integrity:
1. It needs less hacking to do interesting things. For example,
multiruby support would be dead simple to setup, which is something we
really need.

I believe the multiruby config stuff is currently underway for our CI
setup. Perhaps chad or mike could ump in here and compare the
configuration difficulty when switching?

2. Adding different sorts of notifiers to it is really easy. Being
able to dump CI reports into IRC, Campfire, etc. without a lot of
hassle would be awesome.

Indeed, this sounds great.

3. It's really lightweight.

Does it scale? j/k

There are a number of other reasons I prefer it, but these hit the
main technical ones. The others are mostly opinion. :slight_smile:

What do you all think?

Basically the guys doing the work should have the most say in what
happens. From my perspective the CI box is something that needs to
email the core-list when things stop working, and the configuration
and build scripts need to live in git alongside the rest of the code.
Everything else beyond that is gravy!

Hey guys and gals,
I suggested that perhaps we should use Integrity
(http://github.com/foca/integrity) for the CI for Rails rather than
CC.rb. DHH said he was all for it, but I should bring it before you
all before he made a decision.

I'm open minded. As Koz said, I've done most of the work myself, with
some help from Mike Gunderloy. We (my company) use ccrb mostly by
default, because it used to be the only mature, fully-featured CI
server in Ruby. Apparently that isn't the case anymore, which is a
Good Thing!

If using Integrity means that more people will be on board with
supporting CI (which primarily means making sure Rails Core applies
patches to keep it green), then I'm in favor of it. There are a few
things that ccrb does have going for it that I wouldn't want to lose,
though - I'll mention those later.

I also wrote Cinabox
(http://github.com/thewoolleyman/cinabox/tree/master) which makes it
almost a complete no-brainer to set up ccrb on a clean Ubuntu distro
(just two scripts and everything is running, including a working init
script)

So, here's why I like Integrity:
1. It needs less hacking to do interesting things. For example,
multiruby support would be dead simple to setup, which is something we
really need.

Is this really any easier than ccrb? Once you have the multiple
interpreters installed (with multiruby or something like cinabox's
bootstrap_ruby script, that's another discussion), most of the work is
kicking off the build with the proper environment. As Integrity's
homepage says: "as long as your build process can be run from an
unix-y environment and it returns a zero status code for success and
non-zero for failure, then integrity works for you".

For the rails build, this is just ci_build.rb
(http://github.com/rails/rails/blob/9bcf01b23c25e640da7908ac8b1b49fbf7d2e51a/ci/ci_build.rb)
- it doesn't matter what CI tool you use to invoke it, as long as you
can pass some env or param to indicate the interpreter to use. I've
hacked ccrb build scripts to dynamically take this from the ccrb
project name via env var, which is a slick way to do it. That (plus
GemInstaller) is how I handle dynamically building against multiple
rails gem versions without modifying any source code or re-freezing.

2. Adding different sorts of notifiers to it is really easy. Being
able to dump CI reports into IRC, Campfire, etc. without a lot of
hassle would be awesome.

CCRB is also fairly extensible, and it's all Ruby, so it is a wash in
terms of feasability. However, if Integrity already has all these
working and mostly bug-free, then that is a plus.

3. It's really lightweight.

This is actually pretty compelling. CCRB is in rails, and it loads
the rails env for each builder, which is a significant memory overhead
on low-memory servers running several builds.

There are a number of other reasons I prefer it, but these hit the
main technical ones. The others are mostly opinion. :slight_smile:

What do you all think?

Here's the things that I'd like to confirm before I commit to it:

* A working debian-based init script that is shipped with the project.
By working, I mean it works seamlessly on reboots, on killed servers
(no stale pids - e.g. mongrel cluster not mongrel), and standard
start-stop-restart. Cinabox + ccrb handles all this automatically -
e.g. just run setup_ci.rb and all the init script is working. Full
disclosure, the stale pid problem still isn't fixed in ccrb's init
script if the mongrel is killed.

* Updated docs in rails
(http://github.com/rails/rails/blob/9bcf01b23c25e640da7908ac8b1b49fbf7d2e51a/ci/ci_setup_notes.txt)
that explain exactly how to set up integrity. Preferably, this will
be by adding an "integrity" flavor to cinabox - currently the
ruby/ccrb server setup in the previous doc is one line: "Use cinabox
to perform rest of ruby/ccrb setup:", which consists of naught but
downloading cinabox and running two command line scripts.

* Git support: Is git support rock solid? How about for multiple
non-master branches? Because it took a lot of hacking from various
people to get all the known bugs out of ccrb's git support.

* How about git submodules/piston/braid/giternal? Specifically, can
it automatically trigger a build based on changes in a
submodule/pistonized/braidified/giternal? This is completely
unrelated to the Rails build, but something that would make me really
want to switch to Integrity for my other CI projects :slight_smile:

* I want to be disciplined about having EVERYTHING repeatable on a
clean Ubuntu server via /ci/ci_setup_notes.txt. In other words, I
DON'T want to take the approach of hacking and manually installing
crap on the CI server and nobody knows what is running or how. I want
to be able to completely blow away the server and have it back up and
running by following the ci_setup_notes.txt exactly. This lets other
people reproduce the rails CI environment on their own, which is
critical to being able to reproduce problems that "only" occur on CI
(which really means all the edge case tests, gem versions, databases,
and libraries that aren't even running on your dev box but are on CI).
It also (by leveraging GemInstaller and having a solid init script)
mostly eliminates the need for people to manually log on to the server
to debug problems - everything should be reproducible in an ubuntu VM.

* all the other mature CI features that ccrb has - reporting of scm
failures, changesets, build failures, control over custom build
artifacts, history of build output and artifacts, etc...

So, I'm totally open to try it given the above caveats. If you want
to do it, get everything set up on a fork of rails, and we'll set them
up to run concurrently for a while on ci.rubyonrails.org (following
the diff of your forked /ci/ci_setup_notes.txt exactly) at a different
nginx virtual host. If Integrity runs smoothly and does everything
ccrb does and more, then why not?

-- Chad

I'm going to chime in mostly with "what Chad said." I got interested in the CI story for Rails itself a couple of weeks ago because I simply could not get all the tests to pass on my own dev box, and thought this was due to my own stupidity. It didn't take me long to find out "oh, the tests haven't all passed in ages." This struck me as a ridiculous state of affairs, and (thanks of course to many people) we are now at a point where both the master and 2.2 branches are consistently building green on ci.rubyonrails.org.

To my mind, moving forward the most important thing is that we have a solid CI server with a reproducible setup that is as painless as possible. It also needs to build in a way that most Rails developers do not. Some things to think about:

- Testing against multiple branches of the source.
- There are tests in the Rails source that do not run if you don't have particular gems installed. Have *you* tested FCGI support lately, or SQLite2? The CI box consistently hits a large number of the test paths (though there's always room for improvement: for example, we don't have an Oracle or SQL Server testing strategy yet). This is going to get more tedious as we move forward with JRuby support (and the JRuby guys are committed to quickly fixing any bugs we expose): installed any JDBC-based AR adapters lately?
- There are test paths that are mutually incompatible: for example, testing AR changes against PostgreSQL with both the pg library and the postgres library (this just bit us last week). The CI box can handle this, while most developers are unlikely to shuffle gems to get more thorough tests.
- Testing against multiple ruby implementations (we really need matrix tests of n branches by m ruby implementations). This is not yet rolled out to ci.rubyonrails.org, but Chad and I have been tossing back and forth ideas about how to roll that in to the current painless setup, and we're close. (Meanwhile, if you're curious, I'm doing some experimental builds on my own clone of the CI server: on master, we've got 1 test failing against MRI 1.9.1RC1 [and a patch for that is just waiting to be committed] and 40 tests failing against JRuby trunk).

As Chad said, some of Integrity's benefits look interesting, especially the ability to consume less resources. I think multiruby is a 90% solved problem for our current setup; it's just a matter of finding time to finish rolling it out. And other notifiers do exist for cc.rb, though frankly, its source tree is a bit of a mess.

So, bottom line for me: Rails needs a solid CI process, because the current tree is very close to untestable by individual developers. The existing cc.rb solution can serve as that process. Integrity *may* be able to replace cc.rb with superior awesomeness, but I'd like to see someone set up a proof of concept with repro instructions. If anyone goes down that path, I'll be happy to lend a hand by testing and comparing.

Mike

I’ve been hacking on Integrity in our company. It’s very nice, but somewhat vanilla in its current state. I’d like to see Rails CI powered by Integrity one day, but from what I’ve seen now is not the time. Setting up everything that Chad and Mike listed would take much more effort than sticking with and improving the current system.

But with this many forks and people interested in Integrity, after some time (and our contributions to it) I’d like this idea to be revisited.

We’ve been looking at switching to integrity as well. One of the major benefits I see at the moment is that integrity would likely be more willing to accept features.

On a separate note, I think it would be beneficial to have an easy, built in way to run YOUR app against edge or latest of a given branch (like 2.2-stable). I think this would do 2 things:

  1. Encourage more people to have their apps in CI
  2. Cause people to see changes in the core that affect their apps. This would get more, and more immediate feedback to the core team about commits.

-Mike Gaffney

We've been looking at switching to integrity as well. One of the major
benefits I see at the moment is that integrity would likely be more willing
to accept features.

We are running the ci.rubyonrails.org off my ccrb GitHub branch, and
it's possible that ThoughtWorks would add some committers. I don't
think that's a big concern long term.

On a separate note, I think it would be beneficial to have an easy, built in
way to run YOUR app against edge or latest of a given branch (like
2.2-stable). I think this would do 2 things:
1) Encourage more people to have their apps in CI
2) Cause people to see changes in the core that affect their apps. This
would get more, and more immediate feedback to the core team about commits.

Yep, this would be great, and I've put a lot of thought into how to do
this. We (my employer) already do this on our internal projects - we
have the ability to automatically run against multiple rails versions
or edge without modifying source/unfreezing/refreezing/etc.

As I mentioned before, this can be as easy as creating an new ccrb
project with the desired rails version in the ccrb project name - e.g:
'rails_rel_2.1.0', 'rails_rel_2.2.0', 'rails_rel_edge' (although we
don't do edge yet). You can hack the cruise.rake to incorporate this
into the build, and leverage GemInstaller to activate the appropriate
rails gem via config/preinitializer.rb. Here's a live example where I
use this approach [1] to test GemInstaller against multiple RubyGems
versions [2]. Here is a gist showing the same approach for a rails
app [3]. Note again that there would be no
editing/freezing/unfreezing/installing/uninstalling involved to
continuously test against multiple versions - running CI against a new
version is a simple as creating the ccrb project with the appropriate
name.

Even cooler would be extending this to plugins and gems - ensuring
that your app works against the edge of not only rails, but all the
plugins and gems you are using. That way, you can be very proactive
about dealing with breaking changes (or sending them patches which do
things in a different way that doesn't break you).

However, this involves a much more involved infrastructure of
automatically building gems whenever source is changed, which is also
a great place to leverage CI. I've discussed this some with Koz in
the past - automatically building a nightly rails gem which would
facilitate the approach I mentioned above. You could use git
submodules/piston/braid/giternal to accomplish the same thing with
frozen edge rails/plugins, which is why I asked how Integrity handled
this.

[1] http://geminstaller.rubyforge.org/svn/trunk/cruise_config.rb
[2] http://ci.thewoolleyweb.com/
[3] http://gist.github.com/47004

Thanks,
-- Chad

I'd like to put up an open example of what I'm talking about. Does
anyone have suggestions for a Rails app that is:

* open source on github
* has a fast well-written test suite
* currently has green tests against multiple rails versions and edge
* would be willing to accept patches to accomplish the goals mentioned above

Thanks,
-- Chad

Hey all, sorry for the late response :slight_smile:

So, there's a lot of points to cover, let's go one by one.

One note though, I'm currently refactoring a large chunk of the
internals (wrt background building, mostly, but also cleaning up a bit
the database schema). I wouldn't propose we use integrity in any
"serious" environment until this is done. Basically, I agree with
Mislav that the current state it's probably not something I'd
recommend installing--specially if we have a working environment
already (even if it's using CC.rb :P).

(Integrity works fine, but it's just a bit rough around some edges --
like blocking while a project is building)

*** Script to setup an ubuntu box.

This looks like the biggest roadblock for integrity adoption in
ci.rubyonrails.org. However, since I'm not a big fan of ubuntu
I don't use ubuntu much, however, so I'm not sure I'm the correct
person to attempt this. Any help is greatly appreciated, though :slight_smile:

*** multiruby

As far as I understand, using multiruby is just calling a different
command from rake, and not much more than that, this would work out of
the box with integrity, without any hacking or modification, since
it's just changing the command you run on the shell to build the
project.

*** git support

Part of the current refactoring is easing out the tracking of multiple
branches for a given project. (Right now you can go and create a
project in integrity for each branch, which works fine, but is a bit
clunky, specially if you have many non-master branches.) Other than
that, git support is rock solid.

*** Triggering builds on submodule updates

Integrity will trigger a build on any commit to your repository. This
should include changing submodules, or changing braid config files. I
don't know about piston or giternal, but I guess it should work fine
with those too.

*** Build artifacts

This is something we don't support ATM, though we've been going back
and forth about it on #integrity and we'll support something along the
lines of this.

*** Building against all versions of rails

Has anyone tried http://github.com/ianwhite/garlic/tree/master? It
looks interesting.

Other than that, integrity doesn't provide any magic for this, it just
shells out and calls a user-provided script, recording the output and
the exit code. If you want to go crazy in the script and uninstall
gems, install other versions and run tests again (like the issue with
both postgres gems), you can, and it's trivial. (Something like
https://gist.github.com/962e6ccd1e0389265e70)

One of the ideas for integrity is to have multiple build agents
running in external servers, so you could potentially have a box setup
to test postgres, a box setup to test mysql, and each of them just
builds the necessary parts and integrity will aggregate the reports.
This is not the highest priority right now, but if you want/need this
more than having a complicated build script and want to help out, then
jump into #integrity and let's discuss how this can be done :slight_smile:

One note on this: we probably *don't* want to just run the tests with multiruby. Instead, we want to have multiple versions of ruby installed, and run the tests separately with each version. That makes it much easier to spot whether we have failures on a particular ruby implementation.

multiruby may have a place in helping install the versions in the first place, though I suspect it's overkill for that.

Mike

Hi Mike

Can you expand on this? I've been getting multiruby setup and
figuring out ways to let developers use it easily on runcoderun.com
(our hosted CI product), and it seems like the best option right now
for testing against multiple ruby versions. Multiruby does pretty
much exactly what you want -- it runs the suite against each version
of Ruby, and will tell you which ruby versions pass and which fail.

The pain really comes in when you start dealing with gems, and having
to reinstall gems (at least those with native dependencies) for each
ruby version. For example, you won't be able to really test Rails
against 1.9 until all the database drivers compile and install cleanly
against Ruby 1.9.

The gem thing gets messy no matter what approach you take, but I think
multiruby is a little cleaner then having multiple ruby's and gem
repos installed, and switching things around in the environment or via
symlinks for each version.

- Rob

http://thinkrelevance.com
http://runcoderun.com

Here's the problem: if we use the multiruby command, we end up with one pass-or-fail for "Rails across all installed versions of ruby". Instead, we need separate cc.rb projects (or integrity or runcoderun or whatever) so we can quickly see "rails passed on ruby 1.8.7" in one project and "rails failed on ruby 1.9.1rc1" in another - much more useful, especially when we've got build targets (like JRuby) where we know things are going to fail for a while. Red for JRuby shouldn't mean red for everything.

Mike