Rails internals documentation guides

rubyonrails-core
1

I first thought about creating a feature request on LightHouse, but then:

"Creating a feature request

Please don't. If you want a new feature in Rails, you'll have to pull up your sleeves and get busy yourself. Or convince someone else to do it. See the contributor guide on how to get going. But posting them here is just going to lead to ticket root."

But that is the problem: I would like to ask for some high-level documentation about Rails internals to easy contributions from others than the core team. But how could I pull up my sleeves if I don't know how Rails internals work so that I could write the documentation?

Rails has method and code snipets documentation. But if one tries to understand how Rails initialize itself and its dependencies, how things are structured, what is the Rails way for writing internal code, how class reloading works in development mode and other high level stuff, he/she will need to dive into a lot of code to try to figure it out by himself/herself.

This makes it very hard to get new contributors to Rails core since it seems every railer is busy and chances are small that they have enough time to figure out the Rails internals by themselves so that they can contribute small changes. There are a lot of plugin contributions since its documentation is well written. I think there could be more contributors if we had a README.internals that explained the whole idea about how Rails is structured and the reasons behind it. Then, each major update to Rails internal structure could be reflected in this document...

I was just trying to add a :full_message parameter to validations, so that I could prepare a patch to LightHouse, but was stuck trying to figure out how ActiveRecord is loaded, how Validations are included in ActiveRecord without being required (when I discovered they are autoloaded), what are the differences between ActiveModel and ActiveRecord and several other doubts. It would be much easier if such a README.internals document exists. I spent about 2 hours doing really nothing useful to Rails itself, while I could use this time to actually write a useful patch...

I would like to know what is the Rails core team opinion about that.

Thanks,

Rodrigo.

2

This makes it very hard to get new contributors to Rails core since it seems every railer is busy and chances are small that they have enough time to figure out the Rails internals by themselves so that they can contribute small changes. There are a lot of plugin contributions since its documentation is well written. I think there could be more contributors if we had a README.internals that explained the whole idea about how Rails is structured and the reasons behind it. Then, each major update to Rails internal structure could be reflected in this document...

Same problem here, but ...

There are articles about the internals around the net. Unfortunately they quickly get outdated. And in a way, this is a good thing, because it means Rails is continuously evolving.

With all the changes going in to 3.0, maintaining such documentation may be a nightmare, unless it becomes version specific. Especially with all the refactoring going on.

Here are a few tips I found useful:

1) logging - insert logging everywhere you can when solving a problem. With very concise code this helps understand what is going on and when faster than any documentation.

2) reading changelogs - sometimes unrelated stuff can give insight into the peculiar cases that Rails has to handle.

3) debugger - stopping and looking at the backtrack can teach a lot about the architecture.

4) Lighthouse - like reading changelogs, but with more details

5) browsing through the source when you don't need to - when you are less frustrated and you don't need to urgently find out what's wrong.

Especially the last part - hack on ruby when you are not under pressure. Do it for fun. Learn things before you need them. Read articles about ROR when you have spare time. It pays off eventually. Even if this would mean investing more time in total, it will be more pleasurable.

I was just trying to add a :full_message parameter to validations, so that I could prepare a patch to LightHouse, but was stuck trying to figure out how ActiveRecord is loaded, how Validations are included in ActiveRecord without being required (when I discovered they are autoloaded), what are the differences between ActiveModel and ActiveRecord and several other doubts. It would be much easier if such a README.internals document exists. I spent about 2 hours doing really nothing useful to Rails itself, while I could use this time to actually write a useful patch...

I know what it is like to wake up after a few hours of hacking knowing you basically did ... nothing really.

I would like to know what is the Rails core team opinion about that.

Beware - my opinion is just from a user struggling to understand Rails, not anyone from the core team.

Cheers

3

Carl and I are planning on spending some time on internals documentation once we ship an alpha (for plugin authors).

If you have a specific requests for documentation (especially in ActionPack), feel free to email me and I’ll add it to the list of priorities for when we do documentation.

– Yehuda

4

I’ve started a documentation effort of my own back in June, I’ve got the docs up on a Github repository: http://github.com/radar/how-rails-works. Currently it covers only timezones and respond_to for Rails 2.3. There was another guy who was going to do Dynamic Finders but he hasn’t contributed to that recently.

I like the idea of version specific documentation. Perhaps we could get the docrails team to volunteer for this kind of effort?

5

The docrails team has talked about internals documentation several times, and so far we’ve felt that we just don’t have the resources to cover that and still do a good job on the external-facing documentation. I don’t believe this will change any time soon.

Mike

6

... There are articles about the internals around the net. Unfortunately they quickly get outdated. And in a way, this is a good thing, because it means Rails is continuously evolving.

Thanks God it continues to evolve every day. This is what can make it last several years, if not forever... We should not try to stop this, as you pointed out...

With all the changes going in to 3.0, maintaining such documentation may be a nightmare, unless it becomes version specific. Especially with all the refactoring going on.

If you mean that maintaining such documentation as a separate projects, such as docrails, yes I agree. That is not what I proposed. I propose that the Rails core team embrace the idea of maintaining such documentation in Rails git repository, as well as Linux kernel does. It surely requires extra work, but I think it worths if other people will be able to contribute more...

Docrails will eventually be outdated, since it is a separated project and it is always behind main development. I propose that this kind of documentation should be part of a patch changing significantly the Rails internals. Since it is part of the main repository, there should be no need in maintaining different versions, since git would already take care of this... I wrote about nested templates some time ago in docrails, but I will need to take off the documentation as of Rails 3 release. That is because 'yield' return value has changed in a way that the proposed solution won't work in Rails 3 (I didn't test on Rails 2.3.4 yet). Since I don't know how to implement subtemplates in a similar way for Rails 3, I'll have to drop that documentation. That was said to state the differences about maintaining the documentation inside the main project or as a related independent project. Both have their pros and contras, and, although I liked the railsdoc and Rails guides projects as they are implemented, I think that internal documentation should be maintained altogether with Rails main source code in the same git repository, following the Linux kernel example.

Here are a few tips I found useful: ...

These tips are really useful, but they don't apply to every situations. They are good when dealing with most bugs, but solving some bugs or creating new features that need more knowledge about Rails internals would not be much benefited by these tips. Even with all these techniques, we would waste too much time trying to figure out how Rails works by ourselves, instead of concentrating our efforts to actually correct the bug or add a new feature...

Especially the last part - hack on ruby when you are not under pressure. Do it for fun.

That is exactly what I am doing, but I really don't think it helps... Besides that, we usually really understand things only when we actually need to do something real. When reading some arbitrary code, we may have the feeling that we are understanding, but when having to put it in practice, we realize that we were not...

Learn things before you need them. Read articles about ROR when you have spare time. It pays off eventually. Even if this would mean investing more time in total, it will be more pleasurable.

I do it all the time. Actually I can't avoid! :wink: There are many great articles out there and sometimes we use more time reading them than we would be allowed to... :slight_smile:

...

Thanks for your reply, Cezary.

Regards,

Rodrigo.

7

Just testing... I have just written an enormous answer, that didn't seem to arrive in the server... This google groups interface...

8

Sorry, I think I was too impatient... :frowning:

9

Carl and I are planning on spending some time on internals documentation once we ship an alpha (for plugin authors).

That would be awesome! :slight_smile:

If you have a specific requests for documentation (especially in ActionPack), feel free to email me and I'll add it to the list of priorities for when we do documentation.

I tried to, in Skype, in the hope you see the messages sometime (your state was sleep...). But after a while, you got offline and I don't know if you got the messages. Then I went out shopping and when I returned I saw a "hello" message on Skype (I left the computer turned on), but then you were not online anymore :slight_smile: I think that e-mail (or newsgroup) is being more efficient than Skype when we are not both online :slight_smile:

Let me tell you my doubts at the moment. Firstly, I've noted that ActionModel::Errors#full_messages method is overrided in ActiveRecord in some way that could be dryer and I would like to know if I can submit you a patch to make this override dryer.

Then, I'd like to know better how Rails initialization works. I have noticed that validations.rb is required by activerecord using autoload, while searching the source code. I guess activerecord is also lazy loaded in Rails, but I didn't find any 'autoload' doing this explicitly. I guess activerecord is loaded as a gem, but I didn't find where this lazy loading occurs in gems_dependency.rb, if that is the place where it happens. Where is lazy loading happing on gems?

And just out of curiosity, what is reloaded in development mode and how does it occur? Where is the code that does the reloading process?

Thank you in advance,

Rodrigo.

10

Hi Ryan, I appreciate your effort, but I really don't think that such approach would work, since nobody can track down all changes in Rails core and reflect them in a separate documentation project. I understand that we lack such documentation and that is why I am trying to convince the Rails core team to maintain it inside git repository. That way, if someone makes a substantial change in Rails internals, he/ she could (should?) update the internal documentation files to reflect the changes in the same patch where it happens... It would be more sustainable.

And we could concentrate our efforts in helping improving Rails to show how grateful we are :slight_smile:

11

HI Mike. First, congratulation for your job on railsdoc project! It is a wonderful project!

I agree with you that internal documentation should not be dealt in railsdoc. At first, I thought in suggesting a Rails internal guide, but then I realized that it would be outdated faster than the topic about nested templates that I wrote some time ago... It is not just about the docrails team having or not the resources to cover that, but such a documentation should be maintained alongside with Rails internal changes. That would be too difficult to achieve...

Best regards,

Rodrigo.

12

I don't really know what you mean here.

docrails is a branch. The Rails Documentation Team are *not* the documenters of Rails, we are the guys that have an eye in that area of the project, just that.

Sure, we do a lot of work on the API, but the Rails documentation comes from patches to master. If a patch is not properly documented it should not be accepted as if it hadn't tests.

You know docrails and master are cross merged regularly, there's no way it can become outdated because docrails works directly against the Rails repo.

It has its own branch to be able to offer an open policy with a little safety net and strict rules about not touching code. With the purpose of agilizing doc patches that should go the LH way and good luck otherwise.

We're doing the guides as well. We commented with Pratik and Mike that code patches should patch guides, but that is not being enforced in practice.

As per verisioning, docrails always documents edge, both in API an guides. If you want the docs of a release you take its snapshot (guides are included in Rails releases). The public websites only have stable and edge.

13

Let me tell you my doubts at the moment. Firstly, I've noted that ActionModel::Errors#full_messages method is overrided in ActiveRecord in some way that could be dryer and I would like to know if I can submit you a patch to make this override dryer.

I'm not sure about ActionModel, but we've been working on modifications to ActiveRecord validations over on the Rails I18n list and any code regarding messages is likely going to change in 3.0. You're welcome to come over and discuss your concerns though.

Then, I'd like to know better how Rails initialization works. I have noticed that validations.rb is required by activerecord using autoload, while searching the source code. I guess activerecord is also lazy loaded in Rails, but I didn't find any 'autoload' doing this explicitly. I guess activerecord is loaded as a gem, but I didn't find where this lazy loading occurs in gems_dependency.rb, if that is the place where it happens. Where is lazy loading happing on gems?

Initialization is done in Rails::Initializer, and the frameworks (like ActiveRecord) are loaded by Initializer#require_frameworks. The actual frameworks that are loaded is configurable in environment.rb. So ActiveRecord is explicitely required, but all its various components are autoloaded.

And just out of curiosity, what is reloaded in development mode and how does it occur? Where is the code that does the reloading process?

Loading and reloading is handled by ActiveSupport::Dependencies, which hooks into const_missing for all classes and modules. When cache_classes is set to false, everything that is autoloaded is unloaded at the end of each request by the Dispatcher. The exception is any file that is found in the load_once_paths, which generally contains the lib folder for all plugins. The load_once_paths, as well as the load_paths, are configurable in environment.rb.

To be honest, once you know where to look, it's relatively easy to figure out, as the code is pretty well commented. The trick is to know where to look, and this is where a high level overview would be helpful.

14

Let me tell you my doubts at the moment. Firstly, I've noted that ActionModel::Errors#full_messages method is overrided in ActiveRecord in some way that could be dryer and I would like to know if I can submit you a patch to make this override dryer.       I'm not sure about ActionModel, but we've been working on modifications to ActiveRecord validations over on the Rails I18n list and any code regarding messages is likely going to change in 3.0. You're welcome to come over and discuss your concerns though.     I'm already subscribed to i18n list but I haven't seen any topic about 3.0 recently.

Are there any repositories where I could take a look at the changes to 3.0?

Anyway, the patch I intend to prepare for submission is not directly related to I18n in my opinion.

Please, see my comments on subject at:

https://rails.lighthouseapp.com/projects/8994/tickets/1687-flexible-formatting-for-ar-validation-error-messages#ticket-1687-36

Then, I'd like to know better how Rails initialization works. I have noticed that validations.rb is required by activerecord using autoload, while searching the source code. I guess activerecord is also lazy loaded in Rails, but I didn't find any 'autoload' doing this explicitly. I guess activerecord is loaded as a gem, but I didn't find where this lazy loading occurs in gems_dependency.rb, if that is the place where it happens. Where is lazy loading happing on gems?       Initialization is done in Rails::Initializer, and the frameworks (like ActiveRecord) are loaded by Initializer#require_frameworks. The actual frameworks that are loaded is configurable in environment.rb. So ActiveRecord is explicitely required, but all its various components are autoloaded.

And just out of curiosity, what is reloaded in development mode and how does it occur? Where is the code that does the reloading process?       Loading and reloading is handled by ActiveSupport::Dependencies, which hooks into const_missing for all classes and modules. When cache_classes is set to false, everything that is autoloaded is unloaded at the end of each request by the Dispatcher. The exception is any file that is found in the load_once_paths, which generally contains the lib folder for all plugins. The load_once_paths, as well as the load_paths, are configurable in environment.rb.

To be honest, once you know where to look, it's relatively easy to figure out, as the code is pretty well commented. The trick is to know where to look, and this is where a high level overview would be helpful.     Exactly! That is why I am asking for high level overview documentation alongside with Rails source code inside the repository.

Thank you very much for you explanations. I'll take a further look at source-code to get more details. I think it will be easier now.

Thank you,

Rodrigo.

15

Docrails will eventually be outdated, since it is a separated project and it is always behind main development.       I don't really know what you mean here.

docrails is a branch. The Rails Documentation Team are *not* the documenters of Rails, we are the guys that have an eye in that area of the project, just that.     That is exactly what I meant. I was not complaining. But since the guides documentation are not an official documentation, there are no concerns in updating the guides when something changes in source-code in the same patch. I guess most core developers didn't even read all guides so that they are aware of changes that possible affect the guides. The consequences are that the guides will eventually be outdated until someone notes and updates its documentation to reflect the new changes.

This is a bit dangerous in my opinion if we want to welcome new developers. Documentation is one of the most important roles in any framework. Once the user is not able to follow the instructions in the recommended source of documentation, he/she might get a bad impression on the framework consistency and organization. Where I work currently, I am asked to develop in Grails. The documentation are most contradictory or incomplete. This really makes it very difficult not only for newcomers but for long date users too. The documentation states that some magic variables need to be declared in some way in the controllers under documentation, while the scaffold generator uses another way to declare the same behavior. Both documentation and framework version refer to the same version (1.1.1) but they are inconsistent. This happens a lot in Grails.

When one tries to convince some company to adopt Rails, it will be more difficult if the documentation is not clear or is incorrect. In the other side, if we low the learning curve to new developers and make the documentation consistent, we will also get more contributors to help us getting Rails better and better faster. In my specific case, if I could convince the Rails team to accept a patch to include the :full_message option to validations, it would be easier to make a clean presentation of Rails framework to my company and try to convince them to accept Rails...

In Brazil, at least, the default full_messages behavior is not that much useful for real applications. We are very concerned, in our company at least, to give users well formed Brazilian messages. Brazilian phrases usually initiate with an article, instead of the model attribute. In some cases, it makes more sense that the attribute come at the end of the phrase, and some times the attribute itself is not even cited in the message.

Having to patch around I18n files will not make any sense in a presentation and it would make it seem very difficult to do simple things in Rails.

Imagine this scenario:

validates_presence_of :login, :full_message => 'You must specify an user name.' validates_format_of :login, :with => /[a-z]+/, allow_blank => true, :full_message => 'Your user name should contain only small case letters'

Pretty simple to get it, don't you think? But since :full_message doesn't exist, it would be much more confuse if I told them that to get these messages as of today, I would need to explain them how i18n works in Rails, even if they don't care about i18n, since the product is not intended to be commercialized outside Brazil, and they hardly would think that control of the full message should be related to i18n. And even showing them how to achieve this same result with i18n framework it would indeed be much more confusing than the above clear examples.

I am always concerned about new adopters and trying to get their experiences as smooth as possible. Regarding Rails, it usually means investing in documentation. But in this specific case of full messages control, I would prefer to submit a patch, that would still be compatible with how it currently works. I would just add an extra option to validations.

Sure, we do a lot of work on the API, but the Rails documentation comes from patches to master. If a patch is not properly documented it should not be accepted as if it hadn't tests.

You know docrails and master are cross merged regularly, there's no way it can become outdated because docrails works directly against the Rails repo.     Yes, I was not clear in my last message. You are talking about API documentation and I agree with you. I was talking with Rails guides in mind.

It has its own branch to be able to offer an open policy with a little safety net and strict rules about not touching code. With the purpose of agilizing doc patches that should go the LH way and good luck otherwise.

We're doing the guides as well. We commented with Pratik and Mike that code patches should patch guides, but that is not being enforced in practice.     Exactly. I agree with you. It would be awesome if the guides updates were enforced in practice, but I can understand their reasoning that it would slow down main development... I really don't know what would be the best approach to this problem by now...

As per verisioning, docrails always documents edge, both in API an guides. If you want the docs of a release you take its snapshot (guides are included in Rails releases). The public websites only have stable and edge.     Yes, I know that, and I really don't see any problems in how it is done. I really think the docrails project is an excelent idea, as it is.

I was just trying to integrate it more with core development in such a way that the guides would not be outdated...

Congratulations for your work in docrails, by the way.

Cheers,

Rodrigo.

16

I'm not sure about ActionModel, but we've been working on modifications to ActiveRecord validations over on the Rails I18n list and any code regarding messages is likely going to change in 3.0. You're welcome to come over and discuss your concerns though.

I'm already subscribed to i18n list but I haven't seen any topic about 3.0 recently.

Are there any repositories where I could take a look at the changes to 3.0?

A bunch of stuff was changed in 2.3.4, but changes for 3.0 are still to be discussed; I was in fact planning on posting about this soon.

Anyway, the patch I intend to prepare for submission is not directly related to I18n in my opinion.

Since error messages go through the I18n system, any changes made will necessarily involve the I18n code to some degree. i.e. it would be hard to patch any code related to full_messages without affecting I18n related code. Furthermore, I think that *any* code that relates to displaying text needs to keep I18n in mind.

Please, see my comments on subject at:

#1687 Flexible formatting for AR validation error messages - Ruby on Rails - rails

Right, I hadn't checked in on that thread in a while.

As of 2.3.4, you have two possibilities:

1) you can specify a separate message and full_message per validation using the I18n.

2) you can change the full_message to something other than '{{attribute}} {{message}}', like simply '{{message}}'. This would mean that whatever you set as message will also be used for the full_message.

Now, I realize this may seem like a workaround to doing it directly in the model, (although I'd argue that it's probably better to have all the text strings in one place anyway,) but I think it does cover what you're trying to do?

17

...

Anyway, the patch I intend to prepare for submission is not directly related to I18n in my opinion.       Since error messages go through the I18n system, any changes made will necessarily involve the I18n code to some degree. i.e. it would be hard to patch any code related to full_messages without affecting I18n related code.

That is why I asked if there is already any repository with these changes... :slight_smile:

Furthermore, I think that *any* code that relates to displaying text needs to keep I18n in mind.     I agree, I will certainly have i18n in mind when preparing the patch, but I just wanted to enforce that I don't think this should be handled only in i18n code. This issue should not be so i18n related as it is today.

Please, see my comments on subject at:

#1687 Flexible formatting for AR validation error messages - Ruby on Rails - rails #ticket-1687-36       Right, I hadn't checked in on that thread in a while.

As of 2.3.4, you have two possibilities:

1) you can specify a separate message and full_message per validation using the I18n.

2) you can change the full_message to something other than '{{attribute}} {{message}}', like simply '{{message}}'. This would mean that whatever you set as message will also be used for the full_message.

Now, I realize this may seem like a workaround to doing it directly in the model,

That is exactly what I think. I see this should not be a real problem in internationalized applications. But if you think about one-language application, it should hardly need to worry about i18n and the translation files. It would just be so much simpler to write the full messages directly in the models when this approach fits better.

  (although I'd argue that it's probably better to have all the text strings in one place anyway,) but I think it does cover what you're trying to do?     Rails 2.3.4 i18n module certainly allows me to do what I want. I just don't think it is the best way of approaching the full message control in validations. And I'm really concerned in trying to make Rails be as simple as possible in my demonstration to the company where I work, when I have the opportunity to talk about Rails.

Thank you for you feedback,

Rodrigo.

18

The goal of the 2.3.4 I18n changes wasn't to implement the ideal system, but to make possible things that were previously not possible. We plan on coming up with a better solution for 3.0, and it would be better for you to get involved in that discussion than work on a patch that may never get accepted (The reasons for which Jose and Sven both covered in the lighthouse thread you linked earlier).

19

That is exactly what I meant. I was not complaining. But since the guides documentation are not an official documentation,

Why do you say that?

The guides live under rubyonrails.org, are coordinated by the Rails Documentation Team, whose leader is a core team member. Guides are included in the Rails distribution. They are as official as you can get.

there are no concerns in updating the guides when something changes in source-code in the same patch. I guess most core developers didn't even read all guides so that they are aware of changes that possible affect the guides. The consequences are that the guides will eventually be outdated until someone notes and updates its documentation to reflect the new changes.

Yeah that's a risk.

The good news is that guides are under the umbrella of docrails. So updating them is quite easy and that's happening. It could be the case that doesn't happen, certainly, but it can be the case it does. The setup is streamlined and well-supportted to help making that possible.

This is a bit dangerous in my opinion if we want to welcome new developers. Documentation is one of the most important roles in any framework.

Agreed. For newbies, and for experienced developers also. I personally consult the API everyday, it is very important that it is as good as possible. Complete, with good examples, uniform conventions, etc. That's why I joined docrails as soon as Pratik announced it.

I am always concerned about new adopters and trying to get their experiences as smooth as possible.

That's good.

Congratulations for your work in docrails, by the way.

Thank you!