New 2.7/3.0 keyword argument pain point

Yes, current ruby/ruby master has the keyword arguments properly separated, like they should be in Ruby 3 if it goes according to the original plan.

For that reason, it’s sometimes easier to debug issues with ruby master than 2.7, because it’s hard errors with a backtrace instead of warnings (potentially misplaced for delegation), and the semantics with the separation are a lot easier to reason about (*args, **kwargs-delegation actually works on master, and there are no implicit conversion of positional Hash to keywords).

2 Likes

Excellent. Thanks for the clarification Benoit, appreciated!

I want to thank @matz, @mame and the others who work so hard to perfect the language we all love. We see how much effort goes into respecting all interests.

Earlier in the thread, someone mentioned that perhaps a release as important as 3.0 should be delayed until this is perfected. I wanted to add my voice to support this, at least as a point of discussion.

We all understand that having deadlines motivates us to complete seemingly impossible goals. However, it’s also true that we have one chance to get this right and I would hope that we sacrifice Christmas as a totally arbitrary (if traditional and whimsical) deadline before we either rush out something that could do lasting damage to future language adoption or to the emotional/physical health of the people working on these details.

The only people who will be appear upset about a Christmas deadline for the Ruby language being held over are trolls.

We cannot say in good conscience that “we love you” over and over without accepting that your human happiness is more important than our developer happiness.

Please, take the time that you need to get this right. We love you.

4 Likes

Apologies in advance, this is going to be a long message. tldr: use the warning gem and Warning.process('', keyword_separation: :raise) to help debug keyword separation issues in Ruby 2.7.

I’m the main implementor of the ruby 2.7/3.0 keyword argument separation changes. My reading of this thread is that most people who posted appreciate the changes and would prefer if we do not make changes to ruby 2.7 or change the plan for ruby 3.0. This includes maintainers of very large applications, such as byroot (Shopify). I agree with them and recommend that we do not make changes to ruby 2.7 or change the plan for ruby 3.0.

As @matz is considered making things more backwards compatible in ruby 3.0, I created a patch that I think will restore most of the compatibility (https://bugs.ruby-lang.org/issues/16891), with the downside that most of the problems caused by positional hash to keyword argument converstion remain. I am not in favor of using the patch, but if we are to restore compatibility somewhat, I think it is the best way to do so. Some users who replied to this thread appeared to express favor of restoring some compatibility, such as Benoit_Daloze (though I don’t know his feelings regarding the patch).

Most of the difficulties people have with fixing keyword argument separation issues in 2.7 would be made easier by printing backtraces with warnings or raising errors instead of warnings in Ruby 2.7. This allows you to figure out places where keyword argument separation should be used (e.g. where ** should be added), when the warning message raised by Ruby is incorrect due to the use of delegation. The warning gem now makes this easy. Based on @mame’s ideas, you can do the following with the warning gem:

require 'warning'

# To turn keyword argument separation warnings into RuntimeErrors
Warning.process('', keyword_separation: :raise)

# To print backtraces after keyword argument separation warnings 
Warning.process('', keyword_separation: :backtrace)

# To turn keyword argument separation warnings into RuntimeErrors,
# but only for code in the application
Warning.process('/path/to/app', keyword_separation: :raise)

This should allow you to more easily determine the cause of the keyword argument separation issue, and once you determine the cause, it is generally simple to fix the issue. I think it’s easiest to use keyword_separation: :raise, run the tests, and fix all failures, then repeat until there are no more failures. This is basically the same advantage as testing on the Ruby master branch, in terms of fixing keyword argument separation issues.

In regards to handling delegation in Ruby 2.7, if you do not need to be compatible with older versions of Ruby, just switch to using *args, **kwargs in your delegation methods. If you want to be compatible with older versions of Ruby, or *args, **kwargs causes a performance bottleneck, continue to use *args for delegation and mark the methods with ruby2_keywords.

There are a couple concerns raised about the use of ruby2_keywords:

  1. It is ugly
  2. It will be going away at some point.

I don’t think item 1 is much of a concern, though I did prefer the original name I gave it (pass_keywords). I like beautiful Ruby code as much as anyone, but sometimes compatibility requires code somewhat lacking in beauty. Almost all of my libraries support Ruby 1.9 to Ruby 2.7, so I am very used to dealing with compatibility issues.

ruby2_keywords was specifically designed with the goal to make the same code work in older Ruby versions, Ruby 2.7, and Ruby 3.x with the minimum of changes. It requires no changes to method internals and no duplicated method definitions. As far as I know, nobody has brought up technical issues with using ruby2_keywords for handling delegation. JonRowe mentioned he would like ruby2_keywords to be more permissive, but did not specify why the current ruby2_keywords does not work for RSpec. I know Benoit_Daloze is against it as requires a minor performance decrease in some cases, but I think that is far outweighed by the performance increase in delegation cases.

In regards to 2, there is nothing requiring we (ruby-core) remove ruby2_keywords. We will definitely be keeping it at least until Ruby 2.6 goes out of support. Beyond that, it is up to @matz. ruby2_keywords has significant performance advantages over *args, **kwargs delegation in CRuby and I personally believe ruby2_keywords should be kept until there is an alternative to ruby2_keywords that is at least as efficient in CRuby. Again, how long we keep it is up to @matz, there is not a determined time where we are planning to remove it.

Some other concerns raised in this thread:

  • doudou42 was against allowing non-Symbol keyword arguments and was relying on keyword argument splitting. The splitting was deprecated in 2.7 and removed in master and is something @matz never intended to support. Even if we allow positional hash to keyword argument conversion, I do not think we should continue to support splitting of hashes. Users should use separate arguments in such cases.

  • samsaffron was mostly concerned about the performance issues with delegation. These can be fixed by using ruby2_keywords. He is hesitant to use ruby2_keywords as he feels it will need to be removed later. As I mentioned, there is no need to remove it as long as people still find it useful (either for performance or compatibility). I have a lot of respect for samsaffron, but I disagree with him that we need support for an Arguments-like object in Ruby. ruby2_keywords is enough now, and hopefully by the time it isn’t enough, there will be a replacement that is at least as fast.

  • samsaffron also was concerned that we changed some deprecation warnings from only being issued in verbose mode to always being issued. It is my opinion that verbose mode deprecations are only useful as an early warning, to be followed by warning always before raising an exception or changing behavior.I do think it is problematic to have deprecation warnings only issued in verbose mode for many years, but only because it is an indication that the deprecation and removal should have happened much sooner.

  • matthewd was concerned in general about warnings being issued to users of command line Ruby applications. You can set $VERBOSE = nil to silence all warnings, and that is probably a wise idea for all command line Ruby applications where you cannot control which Ruby version is used to run them.

5 Likes

Regarding Feature #16891: Restore Positional Argument to Keyword Conversion - Ruby master - Ruby Issue Tracking System I wrote my thoughts there, but in short I feel it’s only going to cause more confusion and delay migration, i.e., I don’t think it will actually help.

My intuition is most of the problems/pain points come from delegation, because there is no good solution to migrate:

  • ruby2_keywords: IMHO it doesn’t make sense to use ruby2_* in Ruby 3+, but we’d have to use it for delegation as long as we don’t drop Ruby 2 compatibility entirely. And it doesn’t feel like a long term solution, only a temporary one.
  • ...: much nicer, but again not available before Ruby 2.7, so we can’t use it (without eval) before dropping Ruby 2.6 support.
  • *args, **kwargs: incorrect and broken on Ruby <= 2.7. Correct and intuitive on Ruby 3+.

That’s why I think we should have *args still forwards all arguments as it did in Ruby 2.6 and before, in 2.7. In other words, having the “ruby2_keywords semantics” by default in 2.7 but without needing an explicit ruby2_keywords.
Then in a few years, when it’s reasonable for most gems to stop supporting Ruby 2, we can migrate to ... or *args, **kwargs, and at that point have accurate warnings mentioning what to change for delegation using *args. This is detailed in https://bugs.ruby-lang.org/issues/16463#note-29 notably with slides explaining the proposition.

If delegation is not a frequent pain point in practice, I think the original plan is fine.

Regarding when to remove ruby2_keywords, I am in favor to remove it as soon as possible. It’s a major hack (but the semantics of it are pretty much needed for compatibility in 2.7) and I would guess many agree using ruby2_keywords in say Ruby 3 or Ruby 4 code just doesn’t feel right. We need a natural way to do delegation, not something relying on a magical flag, at least longer term.
I don’t think performance is a good enough reason to keep ruby2_keywords, rather ... and *args, **kwargs should be optimized properly to achieve similar performance. Maybe even being a bit slower on generic delegation is acceptable if normal calls are faster, due to a stronger separation between positional and keywords arguments.

1 Like

Thank you for all the valuable input. We have investigated opinions about the keyword argument pain points very carefully, and concluded the biggest one is the overwhelming deprecations warnings regarding keyword arguments from 2.7.

So our planning actions will be:

  • stop noisy warnings regarding keyword arguments (in 2.7.2)
  • encourage use of warning gem from Jeremy. By using the gem, you can cause warnings (as it does now), or get the 3.0 behavior right now.
  • we will move on to the new keyword argument behavior in Ruby3.0 as planned, that will be released on December this year.
  • we are still considering some issues, e.g. retrieving given arguments altogether. But we need some time.

If you have any questions or concerns, feel free to post here.

Matz.

14 Likes