New 2.7/3.0 keyword argument pain point

I agree with you. However, I wonder if you have any idea how to address the issue you raise. How do you change something and get libraries to update without any kind of visible warning in user code?

Hi, I’m from MRI core team, and am an original proposer of this change in question. First of all, I’m really sorry for those who feel pain against this change, and thank you for taking time to share your feeling to this thread. I’d like to make effort to find the best way, with matz and MRI core team.

I summarized the pain points and added my comments. (Note that my comments are mine, not the consensus of MRI core team.) If I miss anything, sorry but let me know. And if your pain point is not shared yet, please write a comment and let us know.

There are too many warnings

there’s indeed a lot of them. (#7 byroot)

Over 50 000 warnings generated by our CI, thousands from gems. I didn’t know where to start. (#8 Florent)

various loud deprecations. (#15 samsaffron)

By going immediately noisy with deprecation warnings, they in practice become an immediate breaking change (#25 Matthew)

I totally hear you. I think we will make the warnings opt-in in 2.7.2.

Just silencing deprecations feels wrong to us as a path forward. If Ruby, by default, is being loud I feel we just have to deal with it. (#15 samsaffron)

You are right. Actually, we wanted you not to ignore them because your code would break on Ruby 3.0 (in the original plan). However, it is reasonable to tentatively suppress them when you are not responsible to fix them, e.g., in production. Thus, we also provided a new option -W:no-deprecated and an API Warning[:deprecated] = false to suppress the warnings, which are mentioned in the official migration guide (described later).

Note that the option stops not only keyword arguments, but also all deprecations. Shopify’s deprecation_toolkit would be a better solution for fine-grained control of suppression. Thank you, Shopify team.

Argument forwarding is difficult

capturing args to forward them to classes later, at this point I have found a bit painful to be able to capture kwargs indeed. (#2 Thibaut)

the main pain point for our library is that we transparently pass around arguments given by users, to user code, so we are not directly aware (in a lot of cases) wether keyword arguments are being used or not (#4 Jon)

the main pain point is blind delegation (#7 byroot)

Ruby is only missing two things: An easy way to forward parameters that works on all currently supported versions. (#10 byroot)

Capturing arguments “arbitrarily” to forward them on later. (#12 Jonathan)

That’s the very point that we were worrying about.

In the original plan, Ruby 3.0 would achieve very simple world; positional arguments are consistently passed as positionals, and keyword arguments are done as keywords. However, this is essentially incompatible, so compatible argument forwarding (that works from 2.6 to 3.0) became very difficult. We took so much time to design a migration path, but we couldn’t come up with a better one than ruby2_keywords. (At first glance, it looks like there are many better ways, but maybe they have very complex trade-offs.)

I admit that separating keyword arguments from positional ones makes “capturing args” less concise and less performant. IMO, we may mitigate some parts of the issue by introducing a new notion (and notation) of, say, Arguments class. But the design requires careful (and long) consideration, so we cannot do that right away.

FYI: matz plans to relax the current limitation of .... It is not decided whether it is backported to Ruby 2.7, though.

ruby2_keywords is ugly

I suppose that’s what ruby2_keywords is for, but it’s tedious to use. (#7 byroot)

Firstly I am uneasy with the whole ruby2_keywords thing, it feels like I am adding technical debt to my code when I hack it in. (#15 samsaffron)

You are perfectly right. In a sense, this is by design. This is just for backward-compatible code that needs to work on Ruby 2.6…Ruby 3.0. In the future, when you use only Ruby 3.0 or later (as byroot said, this is typical for many applications), we want people to write code that completely separates positional arguments and keyword arguments without ruby2_keywords. This intentional ugliness works too much for you, though.

Unfortunately, Ruby 2.7’s keyword arguments are “half-baked”. You MUST use ruby2_keywords if you need argument forwarding that works perfectly in Ruby 2.7. RUBY_VERSION check or some such thing will bring you some unfortunate corner cases. See the official migration guide (described later) in detail. We tried hard to design the migration path, but I couldn’t create an elegant one. And so instead we chose this “intentionally ugly” temporary directive that would make you want you to migrate to Ruby 3 as soon as possible.

IMHO, the true sadness is a long lasting “half-baked” state. This is the reason why we rushed to introduce the defaulted-on warning into 2.7, and planned the hasty change in 3.0. I’m afraid if postponing the change will not mitigate the pain but increase the pain. Giving up the change completely may somewhat solve your pain for migrating, though this means that Ruby misses the chance to fix this essentially broken language design mistake.

Some DSL cases are difficult

In the specific case of DSL… (#2 Thibaut)

I went through the kiba DSL, and noticed that it apparently represents a calculation of MyClass.new(r, k: 42) by using an Array literal: [MyClass, r, k: 42]. This deeply depends upon Ruby 2 keyword semantics, so it is essentially difficult. To make it compatible with Ruby 2, you may check if the last element is a Hash (by using Object#is_a?), and check if the target method accepts keywords (by using Method#parameters). Personally, I’d like to recommend to use a Proc instead of an Array for such a calculation, but I admit that it may reduce value of DSL.

the “user-facing” API of my library (a robotic framework) is a DSL (#13 Sylvain)

The DSL seems to hit the change of handling non-symbol keys which is accepted as keywords. I may be wrong, but I guess it is not difficult to manually separate non-symbol keys from **kwargs. (Of course, I’m really sorry for asking you to change your code.) I’ve commented it to Ruby bug tracker ticket.

MRI core team is less communicative

Some communication around the tools available as well as some kind of migration guide. (#10 byroot)

Whether it’s good or bad, I’ve posted a migration guide into the official Ruby site. And the article is referred by the release announcement of Ruby 2.7.0. I don’t blame you for missing (or forgetting) the article, but to be honest, it is sad for me that no one mentions it.

Fixing gem upstream is difficult

Dealing with gem warnings can be a nightmare. (#15 samsaffron)

The bigger challenge we have is around upstreaming fixes to gems, but we will certainly also get through that. (#20 samsaffron)

I agree with you. If you encounter any difficult case, please post it here. The Ruby core team members are willing to help the fix. I may not be in a position to say this, but I believe in the power of the Ruby community.

Again, thank you for all your comments. If you think your pain is not expressed yet, please add yours. I’d like to hear you.

16 Likes

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

Thank you heaps for thoughtful response, I feel though that one concern I have here has been lost:

There is no way to maintain the same performance level as *args in 2.0 without ruby2_keywords

It is both ugly and inefficient to implement general method memoization compared to *args.

https://bugs.ruby-lang.org/issues/16897

My proposal is either:

  1. Amend it so the class of *args is Arguments, which is a special Array like object that is kwarg aware.

OR

  1. Introduce ***args which has the class Arguments.

Given Ruby 3 is expected to be breaking, (1) is my preference, but it could be a big break. (2) would also work.

3 Likes

There are too many warnings

Is this because:

1/ There are many places generating a few warnings each, or 2/ There are a few places generating many warnings each.

In my experience it’s (2). For the purpose of fixing existing code, reporting the warning once only is sufficient for bringing attention to the issue and would reduce the volume of the output.

Argument forwarding is difficult

It seems the solution to this is ... which makes the most sense going forward. ruby2_keywords is the backwards compatible option that will be required until all code can use ....

IMHO, the true sadness is a long lasting “half-baked” state.

Agreed.

What profit does this give us?

It’s not about profit, it’s about avoiding unexpected hash -> keyword promotion which cause many pain in interface design and confusion for user, e.g. Async::Notification has trouble when signalling with a plain hash object · Issue #67 · socketry/async · GitHub

Introduce ***args which has the class Arguments.

I believe this is what we should aim for with ... syntax.

1 Like

Yeah def (...args) certainly works as well here, if it is implemented.

3 Likes

@mame I was aware of it, and it’s very good for what it is, a tactical guide: how do I fix that particular deprecation?

What I was talking about was a strategic guide: what workflow and tools should I use to make these 5k deprecations, coming from both my own code as well as some potentially not so well maintained gems, manageable.

What profit does this give us? This is not clear. I think it’s just a painful thing. The new argument layout is very complex and unreadable. I never liked. It’s look like over engineering. The lack of an interesting feature is shocking me for a while. Wasn’t there another way to make more interesting language?

I think ruby needs to be more developer friendly.

If we care about performance, I think we would have chosen other languages ​​already. If you write the clean code it usually works fast, so performance improvements at language level is not necessary. There is already fast languages. I think the focus is on something wrong.

Ruby community grew with rapid and comfortable development love by developers. That’s why we use ruby. I consider that painful days will negatively affect ruby ​​community. We are people who don’t like static types anyway. Going in this direction will upset most people.

For ruby ​​3, I think there should be 2 years more of development. Version 3 should come with a better virtual machine and include asynchronous feature.

1 Like

Thanks for the valuable input @mame! I will take that into account for future versions of Kiba.

I want to keep backward compatibility and will do by best to ensure it can be done (including providing more feedback later).

May I ask: what did you mean exactly by “using a Proc” here (in particular, how it would affect the writing of a DSL?).

All in all: thank you again, I’m happy you are providing this feedback to all of us.

1 Like

For ruby ​​3, I think there should be 2 years more of development.

Why should this be delayed for 2 years? That makes no sense. There isn’t even any objective reason for that statement and it makes even less sense considering that ruby 3.0 will be fairly minimal, compared to say 1.8.x to 2.0.

I just upgraded our codebase to Ruby 2.7 and I was pleasantly surprised by the large amount of warnings. To me the warnings indicate that Ruby is serious about moving the community to Ruby 3.0 when it comes, and that we are going to avoid the nightmare that Python 3 is (was?).

I fully expect the various gems we depend on will eventually get to updating their code. And if there’s a couple still left in a month or so we’ll probably jump in and either help with the migration or figure out if there’s a more actively maintained alternative.

Of course as a user it’s easy to say this, a couple (thousand) extra lines in the CI/CD logs doesn’t hurt me. For the library maintainers it is probably more of a headache. So thanks for dealing with stuff like this :wink:

Thank you for making the tough decisions @mame

Sorry for my late reply!

@samsaffron

I feel though that one concern I have here has been lost:

I’m aware of the performance issue. Introducing Arguments class or something may solve the issue, though the design is not going to happen overnight. BTW, It would be helpful if you can measure the overhead in a real application.

@Thibaut_Barrere

May I ask: what did you mean exactly by “using a Proc” here (in particular, how it would affect the writing of a DSL?).

Ruby 3 distinguishes between MyClass.new(r, k: 42) (passing keyword) and MyClass.new(r, { k: 42 }) (passing a hash). However, it does not distinguish between [MyClass, r, k: 42] and [MyClass, r, { k: 42 }]; the former is automatically converted to the latter. Thus, an array literal is less expressive than method arguments. You can avoid this by more explicit data structure like { receiver: MyClass, args: [r], kwargs: { k: 42 } } or by using Proc like [MyClass, -> f { f[r, k:42] }]. But both sacrifice the simplicity of DSL, unfortunately.

1 Like

Matz and I interviewed @kamipo and @a_matsuda, Rails developers who did much work to support the 3.0 keyword change. They told us that a difficult case is delegation, especially, a case where a warning is emitted within Rails code but Rails is innocent.

This is a very simplified case:

Rails code:

def target(**opt)       # warning: The called method `foo' is defined here
end

ruby2_keywords def lib_proxy(*args)
  target(*args)         # warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
end

User application code (or gem code other than Rails):

def app_proxy(*args) # actually, ruby2_keywords is required here
  lib_proxy(*args)
end

app_proxy(k: 42)

The user code attempts to pass keywords to Rails’ target method via app_proxy and lib_proxy. Rails appropriately annotates lib_proxy with ruby2_keywords, but user code does not yet for app_proxy. Calling app_proxy converts the keywords to a normal Hash, so the final target method accepts them as a positional Hash, which leads to the warning. Unfortunately, the warning points only Rails code. So, the user sends a bug report to Rails and there is no good way for Rails developer to diagnose the issue.

Kamipo said that actual cases include rails/rail #39562 and rails/rails #39227.

For this case, we plan to add a new mode to 2.7.2 so that the warning into an error. (Of course, this is opt-in mode enabled by Warning[:deprecated] = :error or something.) It will produce a backtrace when the warning occurs, which would be helpful to diagnose the issue. We are now developing a patch, and will create a ticket in Ruby bug tracker.

Matz is still considering how to mitigate the pain, and will propose a new plan maybe soon. Ruby core team is working on and experimenting some patches to help him. Please wait a little longer.

3 Likes

Don’t be sorry, no problem!

Thank you, I better understand the problem now.

One extra question for you @mame before I plan work for all this: is what you describe as “Ruby 3” available as code somewhere already (e.g. is this syntax already incorporated in Ruby master branch, or more a specification at this point?).

Knowing this will help me work on those changes long in advance. Thanks!

1 Like

Yes, current ruby/ruby master has the keyword arguments properly separated, like they should be in Ruby 3 if it goes according to the original plan.

For that reason, it’s sometimes easier to debug issues with ruby master than 2.7, because it’s hard errors with a backtrace instead of warnings (potentially misplaced for delegation), and the semantics with the separation are a lot easier to reason about (*args, **kwargs-delegation actually works on master, and there are no implicit conversion of positional Hash to keywords).

2 Likes

Excellent. Thanks for the clarification Benoit, appreciated!

I want to thank @matz, @mame and the others who work so hard to perfect the language we all love. We see how much effort goes into respecting all interests.

Earlier in the thread, someone mentioned that perhaps a release as important as 3.0 should be delayed until this is perfected. I wanted to add my voice to support this, at least as a point of discussion.

We all understand that having deadlines motivates us to complete seemingly impossible goals. However, it’s also true that we have one chance to get this right and I would hope that we sacrifice Christmas as a totally arbitrary (if traditional and whimsical) deadline before we either rush out something that could do lasting damage to future language adoption or to the emotional/physical health of the people working on these details.

The only people who will be appear upset about a Christmas deadline for the Ruby language being held over are trolls.

We cannot say in good conscience that “we love you” over and over without accepting that your human happiness is more important than our developer happiness.

Please, take the time that you need to get this right. We love you.

4 Likes

Apologies in advance, this is going to be a long message. tldr: use the warning gem and Warning.process('', keyword_separation: :raise) to help debug keyword separation issues in Ruby 2.7.

I’m the main implementor of the ruby 2.7/3.0 keyword argument separation changes. My reading of this thread is that most people who posted appreciate the changes and would prefer if we do not make changes to ruby 2.7 or change the plan for ruby 3.0. This includes maintainers of very large applications, such as byroot (Shopify). I agree with them and recommend that we do not make changes to ruby 2.7 or change the plan for ruby 3.0.

As @matz is considered making things more backwards compatible in ruby 3.0, I created a patch that I think will restore most of the compatibility (https://bugs.ruby-lang.org/issues/16891), with the downside that most of the problems caused by positional hash to keyword argument converstion remain. I am not in favor of using the patch, but if we are to restore compatibility somewhat, I think it is the best way to do so. Some users who replied to this thread appeared to express favor of restoring some compatibility, such as Benoit_Daloze (though I don’t know his feelings regarding the patch).

Most of the difficulties people have with fixing keyword argument separation issues in 2.7 would be made easier by printing backtraces with warnings or raising errors instead of warnings in Ruby 2.7. This allows you to figure out places where keyword argument separation should be used (e.g. where ** should be added), when the warning message raised by Ruby is incorrect due to the use of delegation. The warning gem now makes this easy. Based on @mame’s ideas, you can do the following with the warning gem:

require 'warning'

# To turn keyword argument separation warnings into RuntimeErrors
Warning.process('', keyword_separation: :raise)

# To print backtraces after keyword argument separation warnings 
Warning.process('', keyword_separation: :backtrace)

# To turn keyword argument separation warnings into RuntimeErrors,
# but only for code in the application
Warning.process('/path/to/app', keyword_separation: :raise)

This should allow you to more easily determine the cause of the keyword argument separation issue, and once you determine the cause, it is generally simple to fix the issue. I think it’s easiest to use keyword_separation: :raise, run the tests, and fix all failures, then repeat until there are no more failures. This is basically the same advantage as testing on the Ruby master branch, in terms of fixing keyword argument separation issues.

In regards to handling delegation in Ruby 2.7, if you do not need to be compatible with older versions of Ruby, just switch to using *args, **kwargs in your delegation methods. If you want to be compatible with older versions of Ruby, or *args, **kwargs causes a performance bottleneck, continue to use *args for delegation and mark the methods with ruby2_keywords.

There are a couple concerns raised about the use of ruby2_keywords:

  1. It is ugly
  2. It will be going away at some point.

I don’t think item 1 is much of a concern, though I did prefer the original name I gave it (pass_keywords). I like beautiful Ruby code as much as anyone, but sometimes compatibility requires code somewhat lacking in beauty. Almost all of my libraries support Ruby 1.9 to Ruby 2.7, so I am very used to dealing with compatibility issues.

ruby2_keywords was specifically designed with the goal to make the same code work in older Ruby versions, Ruby 2.7, and Ruby 3.x with the minimum of changes. It requires no changes to method internals and no duplicated method definitions. As far as I know, nobody has brought up technical issues with using ruby2_keywords for handling delegation. JonRowe mentioned he would like ruby2_keywords to be more permissive, but did not specify why the current ruby2_keywords does not work for RSpec. I know Benoit_Daloze is against it as requires a minor performance decrease in some cases, but I think that is far outweighed by the performance increase in delegation cases.

In regards to 2, there is nothing requiring we (ruby-core) remove ruby2_keywords. We will definitely be keeping it at least until Ruby 2.6 goes out of support. Beyond that, it is up to @matz. ruby2_keywords has significant performance advantages over *args, **kwargs delegation in CRuby and I personally believe ruby2_keywords should be kept until there is an alternative to ruby2_keywords that is at least as efficient in CRuby. Again, how long we keep it is up to @matz, there is not a determined time where we are planning to remove it.

Some other concerns raised in this thread:

  • doudou42 was against allowing non-Symbol keyword arguments and was relying on keyword argument splitting. The splitting was deprecated in 2.7 and removed in master and is something @matz never intended to support. Even if we allow positional hash to keyword argument conversion, I do not think we should continue to support splitting of hashes. Users should use separate arguments in such cases.

  • samsaffron was mostly concerned about the performance issues with delegation. These can be fixed by using ruby2_keywords. He is hesitant to use ruby2_keywords as he feels it will need to be removed later. As I mentioned, there is no need to remove it as long as people still find it useful (either for performance or compatibility). I have a lot of respect for samsaffron, but I disagree with him that we need support for an Arguments-like object in Ruby. ruby2_keywords is enough now, and hopefully by the time it isn’t enough, there will be a replacement that is at least as fast.

  • samsaffron also was concerned that we changed some deprecation warnings from only being issued in verbose mode to always being issued. It is my opinion that verbose mode deprecations are only useful as an early warning, to be followed by warning always before raising an exception or changing behavior.I do think it is problematic to have deprecation warnings only issued in verbose mode for many years, but only because it is an indication that the deprecation and removal should have happened much sooner.

  • matthewd was concerned in general about warnings being issued to users of command line Ruby applications. You can set $VERBOSE = nil to silence all warnings, and that is probably a wise idea for all command line Ruby applications where you cannot control which Ruby version is used to run them.

5 Likes

Regarding Feature #16891: Restore Positional Argument to Keyword Conversion - Ruby master - Ruby Issue Tracking System I wrote my thoughts there, but in short I feel it’s only going to cause more confusion and delay migration, i.e., I don’t think it will actually help.

My intuition is most of the problems/pain points come from delegation, because there is no good solution to migrate:

  • ruby2_keywords: IMHO it doesn’t make sense to use ruby2_* in Ruby 3+, but we’d have to use it for delegation as long as we don’t drop Ruby 2 compatibility entirely. And it doesn’t feel like a long term solution, only a temporary one.
  • ...: much nicer, but again not available before Ruby 2.7, so we can’t use it (without eval) before dropping Ruby 2.6 support.
  • *args, **kwargs: incorrect and broken on Ruby <= 2.7. Correct and intuitive on Ruby 3+.

That’s why I think we should have *args still forwards all arguments as it did in Ruby 2.6 and before, in 2.7. In other words, having the “ruby2_keywords semantics” by default in 2.7 but without needing an explicit ruby2_keywords.
Then in a few years, when it’s reasonable for most gems to stop supporting Ruby 2, we can migrate to ... or *args, **kwargs, and at that point have accurate warnings mentioning what to change for delegation using *args. This is detailed in https://bugs.ruby-lang.org/issues/16463#note-29 notably with slides explaining the proposition.

If delegation is not a frequent pain point in practice, I think the original plan is fine.

Regarding when to remove ruby2_keywords, I am in favor to remove it as soon as possible. It’s a major hack (but the semantics of it are pretty much needed for compatibility in 2.7) and I would guess many agree using ruby2_keywords in say Ruby 3 or Ruby 4 code just doesn’t feel right. We need a natural way to do delegation, not something relying on a magical flag, at least longer term.
I don’t think performance is a good enough reason to keep ruby2_keywords, rather ... and *args, **kwargs should be optimized properly to achieve similar performance. Maybe even being a bit slower on generic delegation is acceptable if normal calls are faster, due to a stronger separation between positional and keywords arguments.

1 Like

Thank you for all the valuable input. We have investigated opinions about the keyword argument pain points very carefully, and concluded the biggest one is the overwhelming deprecations warnings regarding keyword arguments from 2.7.

So our planning actions will be:

  • stop noisy warnings regarding keyword arguments (in 2.7.2)
  • encourage use of warning gem from Jeremy. By using the gem, you can cause warnings (as it does now), or get the 3.0 behavior right now.
  • we will move on to the new keyword argument behavior in Ruby3.0 as planned, that will be released on December this year.
  • we are still considering some issues, e.g. retrieving given arguments altogether. But we need some time.

If you have any questions or concerns, feel free to post here.

Matz.

18 Likes