Rails internals documentation guides

I first thought about creating a feature request on LightHouse, but
then:

"Creating a feature request

Please don't. If you want a new feature in Rails, you'll have to pull
up your sleeves and get busy yourself. Or convince someone else to do
it. See the contributor guide on how to get going. But posting them
here is just going to lead to ticket root."

But that is the problem: I would like to ask for some high-level
documentation about Rails internals to easy contributions from others
than the core team. But how could I pull up my sleeves if I don't know
how Rails internals work so that I could write the documentation?

Rails has method and code snipets documentation. But if one tries to
understand how Rails initialize itself and its dependencies, how
things are structured, what is the Rails way for writing internal
code, how class reloading works in development mode and other high
level stuff, he/she will need to dive into a lot of code to try to
figure it out by himself/herself.

This makes it very hard to get new contributors to Rails core since it
seems every railer is busy and chances are small that they have enough
time to figure out the Rails internals by themselves so that they can
contribute small changes. There are a lot of plugin contributions
since its documentation is well written. I think there could be more
contributors if we had a README.internals that explained the whole
idea about how Rails is structured and the reasons behind it. Then,
each major update to Rails internal structure could be reflected in
this document...

I was just trying to add a :full_message parameter to validations, so
that I could prepare a patch to LightHouse, but was stuck trying to
figure out how ActiveRecord is loaded, how Validations are included in
ActiveRecord without being required (when I discovered they are
autoloaded), what are the differences between ActiveModel and
ActiveRecord and several other doubts. It would be much easier if such
a README.internals document exists. I spent about 2 hours doing really
nothing useful to Rails itself, while I could use this time to
actually write a useful patch...

I would like to know what is the Rails core team opinion about that.

Thanks,

Rodrigo.

This makes it very hard to get new contributors to Rails core since it
seems every railer is busy and chances are small that they have enough
time to figure out the Rails internals by themselves so that they can
contribute small changes. There are a lot of plugin contributions
since its documentation is well written. I think there could be more
contributors if we had a README.internals that explained the whole
idea about how Rails is structured and the reasons behind it. Then,
each major update to Rails internal structure could be reflected in
this document...

Same problem here, but ...

There are articles about the internals around the net. Unfortunately
they quickly get outdated. And in a way, this is a good thing, because
it means Rails is continuously evolving.

With all the changes going in to 3.0, maintaining such documentation
may be a nightmare, unless it becomes version specific. Especially
with all the refactoring going on.

Here are a few tips I found useful:

1) logging - insert logging everywhere you can when solving a problem.
With very concise code this helps understand what is going on and when
faster than any documentation.

2) reading changelogs - sometimes unrelated stuff can give insight
into the peculiar cases that Rails has to handle.

3) debugger - stopping and looking at the backtrack can teach a lot
about the architecture.

4) Lighthouse - like reading changelogs, but with more details

5) browsing through the source when you don't need to - when you are
less frustrated and you don't need to urgently find out what's wrong.

Especially the last part - hack on ruby when you are not under
pressure. Do it for fun. Learn things before you need them. Read
articles about ROR when you have spare time. It pays off eventually.
Even if this would mean investing more time in total, it will be more
pleasurable.

I was just trying to add a :full_message parameter to validations, so
that I could prepare a patch to LightHouse, but was stuck trying to
figure out how ActiveRecord is loaded, how Validations are included in
ActiveRecord without being required (when I discovered they are
autoloaded), what are the differences between ActiveModel and
ActiveRecord and several other doubts. It would be much easier if such
a README.internals document exists. I spent about 2 hours doing really
nothing useful to Rails itself, while I could use this time to
actually write a useful patch...

I know what it is like to wake up after a few hours of hacking knowing
you basically did ... nothing really.

I would like to know what is the Rails core team opinion about that.

Beware - my opinion is just from a user struggling to understand
Rails, not anyone from the core team.

Cheers

Carl and I are planning on spending some time on internals documentation once we ship an alpha (for plugin authors).

If you have a specific requests for documentation (especially in ActionPack), feel free to email me and I’ll add it to the list of priorities for when we do documentation.

– Yehuda

I’ve started a documentation effort of my own back in June, I’ve got the docs up on a Github repository: http://github.com/radar/how-rails-works. Currently it covers only timezones and respond_to for Rails 2.3. There was another guy who was going to do Dynamic Finders but he hasn’t contributed to that recently.

I like the idea of version specific documentation. Perhaps we could get the docrails team to volunteer for this kind of effort?

The docrails team has talked about internals documentation several times, and so far we’ve felt that we just don’t have the resources to cover that and still do a good job on the external-facing documentation. I don’t believe this will change any time soon.

Mike

...
There are articles about the internals around the net. Unfortunately
they quickly get outdated. And in a way, this is a good thing, because
it means Rails is continuously evolving.

Thanks God it continues to evolve every day. This is what can make it
last several years, if not forever... We should not try to stop this,
as you pointed out...

With all the changes going in to 3.0, maintaining such documentation
may be a nightmare, unless it becomes version specific. Especially
with all the refactoring going on.

If you mean that maintaining such documentation as a separate
projects, such as docrails, yes I agree. That is not what I proposed.
I propose that the Rails core team embrace the idea of maintaining
such documentation in Rails git repository, as well as Linux kernel
does. It surely requires extra work, but I think it worths if other
people will be able to contribute more...

Docrails will eventually be outdated, since it is a separated project
and it is always behind main development. I propose that this kind of
documentation should be part of a patch changing significantly the
Rails internals. Since it is part of the main repository, there should
be no need in maintaining different versions, since git would already
take care of this... I wrote about nested templates some time ago in
docrails, but I will need to take off the documentation as of Rails 3
release. That is because 'yield' return value has changed in a way
that the proposed solution won't work in Rails 3 (I didn't test on
Rails 2.3.4 yet). Since I don't know how to implement subtemplates in
a similar way for Rails 3, I'll have to drop that documentation. That
was said to state the differences about maintaining the documentation
inside the main project or as a related independent project. Both have
their pros and contras, and, although I liked the railsdoc and Rails
guides projects as they are implemented, I think that internal
documentation should be maintained altogether with Rails main source
code in the same git repository, following the Linux kernel example.

Here are a few tips I found useful:
...

These tips are really useful, but they don't apply to every
situations. They are good when dealing with most bugs, but solving
some bugs or creating new features that need more knowledge about
Rails internals would not be much benefited by these tips. Even with
all these techniques, we would waste too much time trying to figure
out how Rails works by ourselves, instead of concentrating our efforts
to actually correct the bug or add a new feature...

Especially the last part - hack on ruby when you are not under
pressure. Do it for fun.

That is exactly what I am doing, but I really don't think it helps...
Besides that, we usually really understand things only when we
actually need to do something real. When reading some arbitrary code,
we may have the feeling that we are understanding, but when having to
put it in practice, we realize that we were not...

Learn things before you need them. Read
articles about ROR when you have spare time. It pays off eventually.
Even if this would mean investing more time in total, it will be more
pleasurable.

I do it all the time. Actually I can't avoid! :wink: There are many great
articles out there and sometimes we use more time reading them than we
would be allowed to... :slight_smile:

...

Thanks for your reply, Cezary.

Regards,

Rodrigo.

Just testing... I have just written an enormous answer, that didn't
seem to arrive in the server... This google groups interface...

Sorry, I think I was too impatient... :frowning:

Carl and I are planning on spending some time on internals documentation
once we ship an alpha (for plugin authors).

That would be awesome! :slight_smile:

If you have a specific requests for documentation (especially in
ActionPack), feel free to email me and I'll add it to the list of priorities
for when we do documentation.

I tried to, in Skype, in the hope you see the messages sometime (your
state was sleep...). But after a while, you got offline and I don't
know if you got the messages. Then I went out shopping and when I
returned I saw a "hello" message on Skype (I left the computer turned
on), but then you were not online anymore :slight_smile: I think that e-mail (or
newsgroup) is being more efficient than Skype when we are not both
online :slight_smile:

Let me tell you my doubts at the moment. Firstly, I've noted that
ActionModel::Errors#full_messages method is overrided in ActiveRecord
in some way that could be dryer and I would like to know if I can
submit you a patch to make this override dryer.

Then, I'd like to know better how Rails initialization works. I have
noticed that validations.rb is required by activerecord using
autoload, while searching the source code. I guess activerecord is
also lazy loaded in Rails, but I didn't find any 'autoload' doing this
explicitly. I guess activerecord is loaded as a gem, but I didn't find
where this lazy loading occurs in gems_dependency.rb, if that is the
place where it happens. Where is lazy loading happing on gems?

And just out of curiosity, what is reloaded in development mode and
how does it occur? Where is the code that does the reloading process?

Thank you in advance,

Rodrigo.

Hi Ryan, I appreciate your effort, but I really don't think that such
approach would work, since nobody can track down all changes in Rails
core and reflect them in a separate documentation project. I
understand that we lack such documentation and that is why I am trying
to convince the Rails core team to maintain it inside git repository.
That way, if someone makes a substantial change in Rails internals, he/
she could (should?) update the internal documentation files to reflect
the changes in the same patch where it happens... It would be more
sustainable.

And we could concentrate our efforts in helping improving Rails to
show how grateful we are :slight_smile:

HI Mike. First, congratulation for your job on railsdoc project! It is
a wonderful project!

I agree with you that internal documentation should not be dealt in
railsdoc. At first, I thought in suggesting a Rails internal guide,
but then I realized that it would be outdated faster than the topic
about nested templates that I wrote some time ago... It is not just
about the docrails team having or not the resources to cover that, but
such a documentation should be maintained alongside with Rails
internal changes. That would be too difficult to achieve...

Best regards,

Rodrigo.

I don't really know what you mean here.

docrails is a branch. The Rails Documentation Team are *not* the
documenters of Rails, we are the guys that have an eye in that area of
the project, just that.

Sure, we do a lot of work on the API, but the Rails documentation
comes from patches to master. If a patch is not properly documented it
should not be accepted as if it hadn't tests.

You know docrails and master are cross merged regularly, there's no
way it can become outdated because docrails works directly against the
Rails repo.

It has its own branch to be able to offer an open policy with a little
safety net and strict rules about not touching code. With the purpose
of agilizing doc patches that should go the LH way and good luck
otherwise.

We're doing the guides as well. We commented with Pratik and Mike that
code patches should patch guides, but that is not being enforced in
practice.

As per verisioning, docrails always documents edge, both in API an
guides. If you want the docs of a release you take its snapshot
(guides are included in Rails releases). The public websites only have
stable and edge.

Let me tell you my doubts at the moment. Firstly, I've noted that
ActionModel::Errors#full_messages method is overrided in ActiveRecord
in some way that could be dryer and I would like to know if I can
submit you a patch to make this override dryer.

I'm not sure about ActionModel, but we've been working on modifications to ActiveRecord validations over on the Rails I18n list and any code regarding messages is likely going to change in 3.0. You're welcome to come over and discuss your concerns though.

Then, I'd like to know better how Rails initialization works. I have
noticed that validations.rb is required by activerecord using
autoload, while searching the source code. I guess activerecord is
also lazy loaded in Rails, but I didn't find any 'autoload' doing this
explicitly. I guess activerecord is loaded as a gem, but I didn't find
where this lazy loading occurs in gems_dependency.rb, if that is the
place where it happens. Where is lazy loading happing on gems?

Initialization is done in Rails::Initializer, and the frameworks (like ActiveRecord) are loaded by Initializer#require_frameworks. The actual frameworks that are loaded is configurable in environment.rb. So ActiveRecord is explicitely required, but all its various components are autoloaded.

And just out of curiosity, what is reloaded in development mode and
how does it occur? Where is the code that does the reloading process?

Loading and reloading is handled by ActiveSupport::Dependencies, which hooks into const_missing for all classes and modules. When cache_classes is set to false, everything that is autoloaded is unloaded at the end of each request by the Dispatcher. The exception is any file that is found in the load_once_paths, which generally contains the lib folder for all plugins. The load_once_paths, as well as the load_paths, are configurable in environment.rb.

To be honest, once you know where to look, it's relatively easy to figure out, as the code is pretty well commented. The trick is to know where to look, and this is where a high level overview would be helpful.

Let me tell you my doubts at the moment. Firstly, I've noted that
ActionModel::Errors#full_messages method is overrided in ActiveRecord
in some way that could be dryer and I would like to know if I can
submit you a patch to make this override dryer.
     
I'm not sure about ActionModel, but we've been working on
modifications to ActiveRecord validations over on the Rails I18n list
and any code regarding messages is likely going to change in 3.0.
You're welcome to come over and discuss your concerns though.
   
I'm already subscribed to i18n list but I haven't seen any topic about 3.0 recently.

Are there any repositories where I could take a look at the changes to 3.0?

Anyway, the patch I intend to prepare for submission is not directly related to I18n in my opinion.

Please, see my comments on subject at:

https://rails.lighthouseapp.com/projects/8994/tickets/1687-flexible-formatting-for-ar-validation-error-messages#ticket-1687-36

Then, I'd like to know better how Rails initialization works. I have
noticed that validations.rb is required by activerecord using
autoload, while searching the source code. I guess activerecord is
also lazy loaded in Rails, but I didn't find any 'autoload' doing this
explicitly. I guess activerecord is loaded as a gem, but I didn't find
where this lazy loading occurs in gems_dependency.rb, if that is the
place where it happens. Where is lazy loading happing on gems?
     
Initialization is done in Rails::Initializer, and the frameworks (like
ActiveRecord) are loaded by Initializer#require_frameworks. The actual
frameworks that are loaded is configurable in environment.rb. So
ActiveRecord is explicitely required, but all its various components
are autoloaded.

And just out of curiosity, what is reloaded in development mode and
how does it occur? Where is the code that does the reloading process?
     
Loading and reloading is handled by ActiveSupport::Dependencies, which
hooks into const_missing for all classes and modules. When
cache_classes is set to false, everything that is autoloaded is
unloaded at the end of each request by the Dispatcher. The exception
is any file that is found in the load_once_paths, which generally
contains the lib folder for all plugins. The load_once_paths, as well
as the load_paths, are configurable in environment.rb.

To be honest, once you know where to look, it's relatively easy to
figure out, as the code is pretty well commented. The trick is to know
where to look, and this is where a high level overview would be helpful.
   
Exactly! That is why I am asking for high level overview documentation alongside with Rails source code inside the repository.

Thank you very much for you explanations. I'll take a further look at source-code to get more details. I think it will be easier now.

Thank you,

Rodrigo.

Docrails will eventually be outdated, since it is a separated project
and it is always behind main development.
     
I don't really know what you mean here.

docrails is a branch. The Rails Documentation Team are *not* the
documenters of Rails, we are the guys that have an eye in that area of
the project, just that.
   
That is exactly what I meant. I was not complaining. But since the guides documentation are not an official documentation, there are no concerns in updating the guides when something changes in source-code in the same patch. I guess most core developers didn't even read all guides so that they are aware of changes that possible affect the guides. The consequences are that the guides will eventually be outdated until someone notes and updates its documentation to reflect the new changes.

This is a bit dangerous in my opinion if we want to welcome new developers. Documentation is one of the most important roles in any framework. Once the user is not able to follow the instructions in the recommended source of documentation, he/she might get a bad impression on the framework consistency and organization. Where I work currently, I am asked to develop in Grails. The documentation are most contradictory or incomplete. This really makes it very difficult not only for newcomers but for long date users too. The documentation states that some magic variables need to be declared in some way in the controllers under documentation, while the scaffold generator uses another way to declare the same behavior. Both documentation and framework version refer to the same version (1.1.1) but they are inconsistent. This happens a lot in Grails.

When one tries to convince some company to adopt Rails, it will be more difficult if the documentation is not clear or is incorrect. In the other side, if we low the learning curve to new developers and make the documentation consistent, we will also get more contributors to help us getting Rails better and better faster. In my specific case, if I could convince the Rails team to accept a patch to include the :full_message option to validations, it would be easier to make a clean presentation of Rails framework to my company and try to convince them to accept Rails...

In Brazil, at least, the default full_messages behavior is not that much useful for real applications. We are very concerned, in our company at least, to give users well formed Brazilian messages. Brazilian phrases usually initiate with an article, instead of the model attribute. In some cases, it makes more sense that the attribute come at the end of the phrase, and some times the attribute itself is not even cited in the message.

Having to patch around I18n files will not make any sense in a presentation and it would make it seem very difficult to do simple things in Rails.

Imagine this scenario:

validates_presence_of :login, :full_message => 'You must specify an user name.'
validates_format_of :login, :with => /[a-z]+/, allow_blank => true, :full_message => 'Your user name should contain only small case letters'

Pretty simple to get it, don't you think? But since :full_message doesn't exist, it would be much more confuse if I told them that to get these messages as of today, I would need to explain them how i18n works in Rails, even if they don't care about i18n, since the product is not intended to be commercialized outside Brazil, and they hardly would think that control of the full message should be related to i18n. And even showing them how to achieve this same result with i18n framework it would indeed be much more confusing than the above clear examples.

I am always concerned about new adopters and trying to get their experiences as smooth as possible. Regarding Rails, it usually means investing in documentation. But in this specific case of full messages control, I would prefer to submit a patch, that would still be compatible with how it currently works. I would just add an extra option to validations.

Sure, we do a lot of work on the API, but the Rails documentation
comes from patches to master. If a patch is not properly documented it
should not be accepted as if it hadn't tests.

You know docrails and master are cross merged regularly, there's no
way it can become outdated because docrails works directly against the
Rails repo.
   
Yes, I was not clear in my last message. You are talking about API documentation and I agree with you. I was talking with Rails guides in mind.

It has its own branch to be able to offer an open policy with a little
safety net and strict rules about not touching code. With the purpose
of agilizing doc patches that should go the LH way and good luck
otherwise.

We're doing the guides as well. We commented with Pratik and Mike that
code patches should patch guides, but that is not being enforced in
practice.
   
Exactly. I agree with you. It would be awesome if the guides updates were enforced in practice, but I can understand their reasoning that it would slow down main development... I really don't know what would be the best approach to this problem by now...

As per verisioning, docrails always documents edge, both in API an
guides. If you want the docs of a release you take its snapshot
(guides are included in Rails releases). The public websites only have
stable and edge.
   
Yes, I know that, and I really don't see any problems in how it is done. I really think the docrails project is an excelent idea, as it is.

I was just trying to integrate it more with core development in such a way that the guides would not be outdated...

Congratulations for your work in docrails, by the way.

Cheers,

Rodrigo.

I'm not sure about ActionModel, but we've been working on
modifications to ActiveRecord validations over on the Rails I18n list
and any code regarding messages is likely going to change in 3.0.
You're welcome to come over and discuss your concerns though.

I'm already subscribed to i18n list but I haven't seen any topic about
3.0 recently.

Are there any repositories where I could take a look at the changes to 3.0?

A bunch of stuff was changed in 2.3.4, but changes for 3.0 are still to be discussed; I was in fact planning on posting about this soon.

Anyway, the patch I intend to prepare for submission is not directly
related to I18n in my opinion.

Since error messages go through the I18n system, any changes made will necessarily involve the I18n code to some degree. i.e. it would be hard to patch any code related to full_messages without affecting I18n related code. Furthermore, I think that *any* code that relates to displaying text needs to keep I18n in mind.

Please, see my comments on subject at:

https://rails.lighthouseapp.com/projects/8994/tickets/1687-flexible-formatting-for-ar-validation-error-messages#ticket-1687-36

Right, I hadn't checked in on that thread in a while.

As of 2.3.4, you have two possibilities:

1) you can specify a separate message and full_message per validation using the I18n.

2) you can change the full_message to something other than '{{attribute}} {{message}}', like simply '{{message}}'. This would mean that whatever you set as message will also be used for the full_message.

Now, I realize this may seem like a workaround to doing it directly in the model, (although I'd argue that it's probably better to have all the text strings in one place anyway,) but I think it does cover what you're trying to do?

...

Anyway, the patch I intend to prepare for submission is not directly
related to I18n in my opinion.
     
Since error messages go through the I18n system, any changes made will
necessarily involve the I18n code to some degree. i.e. it would be
hard to patch any code related to full_messages without affecting I18n
related code.

That is why I asked if there is already any repository with these changes... :slight_smile:

Furthermore, I think that *any* code that relates to
displaying text needs to keep I18n in mind.
   
I agree, I will certainly have i18n in mind when preparing the patch, but I just wanted to enforce that I don't think this should be handled only in i18n code. This issue should not be so i18n related as it is today.

Please, see my comments on subject at:

https://rails.lighthouseapp.com/projects/8994/tickets/1687-flexible-formatting-for-ar-validation-error-messages
#ticket-1687-36
     
Right, I hadn't checked in on that thread in a while.

As of 2.3.4, you have two possibilities:

1) you can specify a separate message and full_message per validation
using the I18n.

2) you can change the full_message to something other than
'{{attribute}} {{message}}', like simply '{{message}}'. This would
mean that whatever you set as message will also be used for the
full_message.

Now, I realize this may seem like a workaround to doing it directly in
the model,

That is exactly what I think. I see this should not be a real problem in internationalized applications. But if you think about one-language application, it should hardly need to worry about i18n and the translation files. It would just be so much simpler to write the full messages directly in the models when this approach fits better.

  (although I'd argue that it's probably better to have all
the text strings in one place anyway,) but I think it does cover what
you're trying to do?
   
Rails 2.3.4 i18n module certainly allows me to do what I want. I just don't think it is the best way of approaching the full message control in validations. And I'm really concerned in trying to make Rails be as simple as possible in my demonstration to the company where I work, when I have the opportunity to talk about Rails.

Thank you for you feedback,

Rodrigo.

The goal of the 2.3.4 I18n changes wasn't to implement the ideal system, but to make possible things that were previously not possible. We plan on coming up with a better solution for 3.0, and it would be better for you to get involved in that discussion than work on a patch that may never get accepted (The reasons for which Jose and Sven both covered in the lighthouse thread you linked earlier).

That is exactly what I meant. I was not complaining. But since the
guides documentation are not an official documentation,

Why do you say that?

The guides live under rubyonrails.org, are coordinated by the Rails
Documentation Team, whose leader is a core team member. Guides are
included in the Rails distribution. They are as official as you can
get.

there are no
concerns in updating the guides when something changes in source-code in
the same patch. I guess most core developers didn't even read all guides
so that they are aware of changes that possible affect the guides. The
consequences are that the guides will eventually be outdated until
someone notes and updates its documentation to reflect the new changes.

Yeah that's a risk.

The good news is that guides are under the umbrella of docrails. So
updating them is quite easy and that's happening. It could be the case
that doesn't happen, certainly, but it can be the case it does. The
setup is streamlined and well-supportted to help making that possible.

This is a bit dangerous in my opinion if we want to welcome new
developers. Documentation is one of the most important roles in any
framework.

Agreed. For newbies, and for experienced developers also. I personally
consult the API everyday, it is very important that it is as good as
possible. Complete, with good examples, uniform conventions, etc.
That's why I joined docrails as soon as Pratik announced it.

I am always concerned about new adopters and trying to get their
experiences as smooth as possible.

That's good.

Congratulations for your work in docrails, by the way.

Thank you!