New 2.7/3.0 keyword argument pain point

[Billy.Zheng (郑伟)]

I totally agree with you.

I think it will be really nice if we can do correct things on ruby 3, if Ruby 2 keyword arguments is a mistake(i think matz said on different occasions), i think update to ruby 3 is the best time to fix it.

I can guess because rails is use too many ruby magic, so, update is painful, but, rails can always select to use old ruby version 2.7. (or a more less warn version, e.g. 2.7.2) and moving to ruby 3 slowly, i even if ruby core team have enough resources, we can release ruby 2.8 with ruby 3 same time, but, i really propose to let ruby evolution toward the correct way.

Thank you.

[dogweather (Robb Shecter)]

@matz

JIT, concurrency, and static type analysis

Awesome! Very glad to hear that these are the priorities.

[samsaffron]

Note, for those interested in this topic it is now being discussed as well in the official Ruby bug tracker:

Feature #16891: Restore Positional Argument to Keyword Conversion - Ruby...

[matthewd (Matthew Draper)]

I think my main concern, personally, is a bit of a meta one.

It’s important to be able to evolve an API forwards over time, and deprecations are an excellent tool for doing so.

The point of a deprecation is that old-style code continues working for a transition period, while developers are encouraged to adopt the new form, without breaking any existing behaviour.

Any time a deprecation is shown to the developer who is responsible for the line that generated the deprecation warning, that is a good thing, and allows them to address the change, in a neat and orderly manner, before the later hard-break change arrives.

(I think all the above is pretty uncontroversial. So now we move on to:)

However, any time a deprecation is shown to someone who is not responsible for the code that caused the warning, that is a breaking change.

If my ruby-authored CLI tool one day starts spitting out warnings to my users, the tool is broken.

If my library/framework starts spitting out warnings (which are the library authors’ responsibility to fix before some future ruby release), that library is broken. And worse, it’s obscuring any warnings that might belong to code written by the local developer.

By going immediately noisy with deprecation warnings, they in practice become an immediate breaking change, and for a lot of people, you might as well have just made the real change without warning and started raising exceptions.

After that, recommending that everyone install warning-filtering code to suppress the warnings just feels tone deaf. If the warnings are immediately important, then why suggest people suppress them? If they’re not, why force everyone to update their code just to get back the previous [warning-free] functionality?

This warning seems to have been added, and defaulted-on, on the assumption that the original authors are the only people who ever see their code’s stderr. (Or that they’re sufficiently involved in ruby-core to recognise the future warning before it shipped, and adapt their code in advance – thereby illustrating that they too do not consider it acceptable for a library to cause un-actionable warnings for its consumers.)

I have mixed feelings about the actual change in question (in short: edge cases can be annoying, but don’t find myself running into the problem this is solving; in my personal opinion, this is forcing a widespread change to code that’s worked consistently since at least 1.8, and it doesn’t feel worth it). But that’s very secondary to how it’s been made. Many libraries have already adapted, small and large – including Rails. But that was forced, on a fast timeline, because downstream users (reasonably, and sometimes loudly) considered those libraries to be broken on Ruby 2.7.

“Is broken on the new Ruby release” is not a deprecation, and if users are perceiving it as broken, then it is broken.

As “just leave Rails behind on old Ruby, and let Ruby evolve on its own” has been suggested, I’ll briefly address it: that’s exactly our concern! There are limits to how far into the future things can be expected to be compatible, but if, when we release Rails, we can’t even expect Ruby+1 to run our code correctly (and here “correctly” means our library doesn’t cause any automatic output/warnings/errors we couldn’t anticipate using the previous version, and isn’t attributable to calling code), then I think the only way we can give our users a smooth experience is to apply a max ruby version as well as a minimum.

For the record, this is something we struggle with when deprecating Rails APIs too. When we deprecate things used by libraries more than application code, we generally try to run a cycle of soft (silent) deprecation first… but we absolutely do get it wrong, and force hurried changes upon our downstreams. Maybe we need our own equivalent of -v. In truth, I think we mostly get away with it because people expect the new version of Rails to break their library, either outright, or with noisy deprecations… and expect upgrading to be a planned, effort-heavy, operation. Historically, I believe people have assumed that ruby-core is more cautious, and so a minor version bump will be a safe and low-pain change. If the plan is to reverse that expectation, then you do get more freedom to change things faster… but you also get a lot more people left behind on old versions.

[walterdavis (Walter Davs)]

This is extremely well-said. The tension between Ruby for library-makers and tool-makers versus Ruby for end-users of those tools is very clear, and the rules around how version changes “break” things need to consider both.

Walter

[byroot (Jean Boussier)]

To followup on my previous post, I started clearing deprecations in our gems yesterday, the first step being to list the gems throwing deprecations.

Out of the 332 gems in our Gemfile, only 37 are printing warnings, only 23 of them are public gems, and if I also remove the gems that are public but used almost exclusively by shopify I’m down to a very small list of 17 gems:

rails # actually it's Rails code calling ours (`as_json`). The fix is to be done inside our code.
activemerchant
bugsnag
faraday
google-cloud-storage
grpc
i18n
memcached_store
minitest # same, it's our code called by minitest that expects keyword args.
mocha # same, our code that expects keywords when it is actually a hash.
rails-controller-testing
responders
sorbet-runtime
state_machines-activemodel
state_machines-activerecord
statsd-instrument
webmock

And for some of them like faraday, the work was already done upstream, we just have to upgrade.

Of course this is only one datapoint, even if our Gemfile is big it’s not representative of the community as a whole, but I think it’s fair to say that the gem ecosystem either wasn’t as much impacted as one might think, or either already started to support Ruby 2.7.

[ioquatix (Samuel Williams)]

@matthewd

The point of a deprecation is that old-style code continues working for a transition period, while developers are encouraged to adopt the new form, without breaking any existing behaviour.

I agree with you. However, I wonder if you have any idea how to address the issue you raise. How do you change something and get libraries to update without any kind of visible warning in user code?

[mame (Yusuke Endoh)]

Hi, I’m from MRI core team, and am an original proposer of this change in question. First of all, I’m really sorry for those who feel pain against this change, and thank you for taking time to share your feeling to this thread. I’d like to make effort to find the best way, with matz and MRI core team.

I summarized the pain points and added my comments. (Note that my comments are mine, not the consensus of MRI core team.) If I miss anything, sorry but let me know. And if your pain point is not shared yet, please write a comment and let us know.

There are too many warnings

there’s indeed a lot of them. (#7 byroot)

Over 50 000 warnings generated by our CI, thousands from gems. I didn’t know where to start. (#8 Florent)

various loud deprecations. (#15 samsaffron)

By going immediately noisy with deprecation warnings, they in practice become an immediate breaking change (#25 Matthew)

I totally hear you. I think we will make the warnings opt-in in 2.7.2.

Just silencing deprecations feels wrong to us as a path forward. If Ruby, by default, is being loud I feel we just have to deal with it. (#15 samsaffron)

You are right. Actually, we wanted you not to ignore them because your code would break on Ruby 3.0 (in the original plan). However, it is reasonable to tentatively suppress them when you are not responsible to fix them, e.g., in production. Thus, we also provided a new option -W:no-deprecated and an API Warning[:deprecated] = false to suppress the warnings, which are mentioned in the official migration guide (described later).

Note that the option stops not only keyword arguments, but also all deprecations. Shopify’s deprecation_toolkit would be a better solution for fine-grained control of suppression. Thank you, Shopify team.

Argument forwarding is difficult

capturing args to forward them to classes later, at this point I have found a bit painful to be able to capture kwargs indeed. (#2 Thibaut)

the main pain point for our library is that we transparently pass around arguments given by users, to user code, so we are not directly aware (in a lot of cases) wether keyword arguments are being used or not (#4 Jon)

the main pain point is blind delegation (#7 byroot)

Ruby is only missing two things: An easy way to forward parameters that works on all currently supported versions. (#10 byroot)

Capturing arguments “arbitrarily” to forward them on later. (#12 Jonathan)

That’s the very point that we were worrying about.

In the original plan, Ruby 3.0 would achieve very simple world; positional arguments are consistently passed as positionals, and keyword arguments are done as keywords. However, this is essentially incompatible, so compatible argument forwarding (that works from 2.6 to 3.0) became very difficult. We took so much time to design a migration path, but we couldn’t come up with a better one than ruby2_keywords. (At first glance, it looks like there are many better ways, but maybe they have very complex trade-offs.)

I admit that separating keyword arguments from positional ones makes “capturing args” less concise and less performant. IMO, we may mitigate some parts of the issue by introducing a new notion (and notation) of, say, Arguments class. But the design requires careful (and long) consideration, so we cannot do that right away.

FYI: matz plans to relax the current limitation of .... It is not decided whether it is backported to Ruby 2.7, though.

ruby2_keywords is ugly

I suppose that’s what ruby2_keywords is for, but it’s tedious to use. (#7 byroot)

Firstly I am uneasy with the whole ruby2_keywords thing, it feels like I am adding technical debt to my code when I hack it in. (#15 samsaffron)

You are perfectly right. In a sense, this is by design. This is just for backward-compatible code that needs to work on Ruby 2.6…Ruby 3.0. In the future, when you use only Ruby 3.0 or later (as byroot said, this is typical for many applications), we want people to write code that completely separates positional arguments and keyword arguments without ruby2_keywords. This intentional ugliness works too much for you, though.

Unfortunately, Ruby 2.7’s keyword arguments are “half-baked”. You MUST use ruby2_keywords if you need argument forwarding that works perfectly in Ruby 2.7. RUBY_VERSION check or some such thing will bring you some unfortunate corner cases. See the official migration guide (described later) in detail. We tried hard to design the migration path, but I couldn’t create an elegant one. And so instead we chose this “intentionally ugly” temporary directive that would make you want you to migrate to Ruby 3 as soon as possible.

IMHO, the true sadness is a long lasting “half-baked” state. This is the reason why we rushed to introduce the defaulted-on warning into 2.7, and planned the hasty change in 3.0. I’m afraid if postponing the change will not mitigate the pain but increase the pain. Giving up the change completely may somewhat solve your pain for migrating, though this means that Ruby misses the chance to fix this essentially broken language design mistake.

Some DSL cases are difficult

In the specific case of DSL… (#2 Thibaut)

I went through the kiba DSL, and noticed that it apparently represents a calculation of MyClass.new(r, k: 42) by using an Array literal: [MyClass, r, k: 42]. This deeply depends upon Ruby 2 keyword semantics, so it is essentially difficult. To make it compatible with Ruby 2, you may check if the last element is a Hash (by using Object#is_a?), and check if the target method accepts keywords (by using Method#parameters). Personally, I’d like to recommend to use a Proc instead of an Array for such a calculation, but I admit that it may reduce value of DSL.

the “user-facing” API of my library (a robotic framework) is a DSL (#13 Sylvain)

The DSL seems to hit the change of handling non-symbol keys which is accepted as keywords. I may be wrong, but I guess it is not difficult to manually separate non-symbol keys from **kwargs. (Of course, I’m really sorry for asking you to change your code.) I’ve commented it to Ruby bug tracker ticket.

MRI core team is less communicative

Some communication around the tools available as well as some kind of migration guide. (#10 byroot)

Whether it’s good or bad, I’ve posted a migration guide into the official Ruby site. And the article is referred by the release announcement of Ruby 2.7.0. I don’t blame you for missing (or forgetting) the article, but to be honest, it is sad for me that no one mentions it.

Fixing gem upstream is difficult

Dealing with gem warnings can be a nightmare. (#15 samsaffron)

The bigger challenge we have is around upstreaming fixes to gems, but we will certainly also get through that. (#20 samsaffron)

I agree with you. If you encounter any difficult case, please post it here. The Ruby core team members are willing to help the fix. I may not be in a position to say this, but I believe in the power of the Ruby community.

Again, thank you for all your comments. If you think your pain is not expressed yet, please add yours. I’d like to hear you.

[Mehmet_Aydogdu (Mehmet Aydoğdu)]

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

[samsaffron]

Thank you heaps for thoughtful response, I feel though that one concern I have here has been lost:

There is no way to maintain the same performance level as *args in 2.0 without ruby2_keywords

It is both ugly and inefficient to implement general method memoization compared to *args.

Feature #16897: General purpose memoizer in Ruby 3 with Ruby 2 performance -...

My proposal is either:

1. Amend it so the class of *args is Arguments, which is a special Array like object that is kwarg aware.

OR

2. Introduce ***args which has the class Arguments.

Given Ruby 3 is expected to be breaking, (1) is my preference, but it could be a big break. (2) would also work.

[ioquatix (Samuel Williams)]

There are too many warnings

Is this because:

1/ There are many places generating a few warnings each, or 2/ There are a few places generating many warnings each.

In my experience it’s (2). For the purpose of fixing existing code, reporting the warning once only is sufficient for bringing attention to the issue and would reduce the volume of the output.

Argument forwarding is difficult

It seems the solution to this is ... which makes the most sense going forward. ruby2_keywords is the backwards compatible option that will be required until all code can use ....

IMHO, the true sadness is a long lasting “half-baked” state.

Agreed.

What profit does this give us?

It’s not about profit, it’s about avoiding unexpected hash → keyword promotion which cause many pain in interface design and confusion for user, e.g. Async::Notification has trouble when signalling with a plain hash object · Issue #67 · socketry/async · GitHub

Introduce ***args which has the class Arguments.

I believe this is what we should aim for with ... syntax.

[samsaffron]

@ioquatix

I believe this is what we should aim for with ... syntax.

Yeah def (...args) certainly works as well here, if it is implemented.

[byroot (Jean Boussier)]

@mame

Whether it’s good or bad, I’ve posted a migration guide into the official Ruby site. And the article is referred by the release announcement of Ruby 2.7.0 . I don’t blame you for missing (or forgetting) the article, but to be honest, it is sad for me that no one mentions it.

@mame I was aware of it, and it’s very good for what it is, a tactical guide: how do I fix that particular deprecation?

What I was talking about was a strategic guide: what workflow and tools should I use to make these 5k deprecations, coming from both my own code as well as some potentially not so well maintained gems, manageable.

[Mehmet_Aydogdu (Mehmet Aydoğdu)]

What profit does this give us? This is not clear. I think it’s just a painful thing. The new argument layout is very complex and unreadable. I never liked. It’s look like over engineering. The lack of an interesting feature is shocking me for a while. Wasn’t there another way to make more interesting language?

I think ruby needs to be more developer friendly.

If we care about performance, I think we would have chosen other languages ​​already. If you write the clean code it usually works fast, so performance improvements at language level is not necessary. There is already fast languages. I think the focus is on something wrong.

Ruby community grew with rapid and comfortable development love by developers. That’s why we use ruby. I consider that painful days will negatively affect ruby ​​community. We are people who don’t like static types anyway. Going in this direction will upset most people.

For ruby ​​3, I think there should be 2 years more of development. Version 3 should come with a better virtual machine and include asynchronous feature.

[Thibaut_Barrere (Thibaut Barrère)]

@mame

I went through the kiba DSL, and noticed that it apparently represents a calculation of MyClass.new(r, k: 42) by using an Array literal: [MyClass, r, k: 42] . This deeply depends upon Ruby 2 keyword semantics, so it is essentially difficult. To make it compatible with Ruby 2, you may check if the last element is a Hash (by using Object#is_a? ), and check if the target method accepts keywords (by using Method#parameters ). Personally, I’d like to recommend to use a Proc instead of an Array for such a calculation, but I admit that it may reduce value of DSL.

Thanks for the valuable input @mame! I will take that into account for future versions of Kiba.

I want to keep backward compatibility and will do by best to ensure it can be done (including providing more feedback later).

May I ask: what did you mean exactly by “using a Proc” here (in particular, how it would affect the writing of a DSL?).

All in all: thank you again, I’m happy you are providing this feedback to all of us.

[shevy]

For ruby ​​3, I think there should be 2 years more of development.

Why should this be delayed for 2 years? That makes no sense. There isn’t even any objective reason for that statement and it makes even less sense considering that ruby 3.0 will be fairly minimal, compared to say 1.8.x to 2.0.

[tinco (Tinco Andringa)]

I just upgraded our codebase to Ruby 2.7 and I was pleasantly surprised by the large amount of warnings. To me the warnings indicate that Ruby is serious about moving the community to Ruby 3.0 when it comes, and that we are going to avoid the nightmare that Python 3 is (was?).

I fully expect the various gems we depend on will eventually get to updating their code. And if there’s a couple still left in a month or so we’ll probably jump in and either help with the migration or figure out if there’s a more actively maintained alternative.

Of course as a user it’s easy to say this, a couple (thousand) extra lines in the CI/CD logs doesn’t hurt me. For the library maintainers it is probably more of a headache. So thanks for dealing with stuff like this

Thank you for making the tough decisions @mame

[mame (Yusuke Endoh)]

Sorry for my late reply!

@samsaffron

I feel though that one concern I have here has been lost:

I’m aware of the performance issue. Introducing Arguments class or something may solve the issue, though the design is not going to happen overnight. BTW, It would be helpful if you can measure the overhead in a real application.

@Thibaut_Barrere

May I ask: what did you mean exactly by “using a Proc” here (in particular, how it would affect the writing of a DSL?).

Ruby 3 distinguishes between MyClass.new(r, k: 42) (passing keyword) and MyClass.new(r, { k: 42 }) (passing a hash). However, it does not distinguish between [MyClass, r, k: 42] and [MyClass, r, { k: 42 }]; the former is automatically converted to the latter. Thus, an array literal is less expressive than method arguments. You can avoid this by more explicit data structure like { receiver: MyClass, args: [r], kwargs: { k: 42 } } or by using Proc like [MyClass, -> f { f[r, k:42] }]. But both sacrifice the simplicity of DSL, unfortunately.

[mame (Yusuke Endoh)]

Matz and I interviewed @kamipo and @a_matsuda, Rails developers who did much work to support the 3.0 keyword change. They told us that a difficult case is delegation, especially, a case where a warning is emitted within Rails code but Rails is innocent.

This is a very simplified case:

Rails code:

def target(**opt)       # warning: The called method `foo' is defined here
end

ruby2_keywords def lib_proxy(*args)
  target(*args)         # warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
end

User application code (or gem code other than Rails):

def app_proxy(*args) # actually, ruby2_keywords is required here
  lib_proxy(*args)
end

app_proxy(k: 42)

The user code attempts to pass keywords to Rails’ target method via app_proxy and lib_proxy. Rails appropriately annotates lib_proxy with ruby2_keywords, but user code does not yet for app_proxy. Calling app_proxy converts the keywords to a normal Hash, so the final target method accepts them as a positional Hash, which leads to the warning. Unfortunately, the warning points only Rails code. So, the user sends a bug report to Rails and there is no good way for Rails developer to diagnose the issue.

Kamipo said that actual cases include rails/rail #39562 and rails/rails #39227.

For this case, we plan to add a new mode to 2.7.2 so that the warning into an error. (Of course, this is opt-in mode enabled by Warning[:deprecated] = :error or something.) It will produce a backtrace when the warning occurs, which would be helpful to diagnose the issue. We are now developing a patch, and will create a ticket in Ruby bug tracker.

Matz is still considering how to mitigate the pain, and will propose a new plan maybe soon. Ruby core team is working on and experimenting some patches to help him. Please wait a little longer.

[Thibaut_Barrere (Thibaut Barrère)]

@mame

Sorry for my late reply!

Don’t be sorry, no problem!

@mame

Ruby 3 distinguishes between MyClass.new(r, k: 42) (passing keyword) and MyClass.new(r, { k: 42 }) (passing a hash). However, it does not distinguish between [MyClass, r, k: 42] and [MyClass, r, { k: 42 }] ; the former is automatically converted to the latter. Thus, an array literal is less expressive than method arguments. You can avoid this by more explicit data structure like { receiver: MyClass, args: [r], kwargs: { k: 42 } } or by using Proc like [MyClass, -> f { f[r, k:42] }] . But both sacrifice the simplicity of DSL, unfortunately.

Thank you, I better understand the problem now.

One extra question for you @mame before I plan work for all this: is what you describe as “Ruby 3” available as code somewhere already (e.g. is this syntax already incorporated in Ruby master branch, or more a specification at this point?).

Knowing this will help me work on those changes long in advance. Thanks!