Webpacker presents a more difficult OOB experience for JS Sprinkles than Sprockets did

[Fjan]

This may be a controversial opinion, but most of the web apps do not need to be SPAs (Single Page Applications) and are much easier and quicker developed using standard Rails templates, perhaps with a sprinkling of JavaScript to support some advanced features. For that use case dropping a single JS file in app/assets/javascript was infinitely easier than setting up the whole web packer asset train.

I understand that webpacker is the way to go for large Rails projects, and for SPAs, but for the large majority of Rails users webpacker is just making life difficult, especially for people just starting out with Rails.

I propose that sprockets becomes a first class citizen again, or perhaps even some other asset pipeline that’s more in tune with the Ruby/Rails way. Webpacker would remain a well-supported option that can be installed when the need arises but not installed by default in a new project.

[mdh (Mark)]

I’ve thought about this a lot as well and keep coming back to these ideas:

• No question that there is a simplicity with sprockets that is way easier to use. The idea that you just have a folder that you can dump stuff into and “it just works” is wonderful.
• There is also no question in my mind that it’s substantially more limited in terms of what you can do and the speed at which you can do it.
• There is a real tension in my mind as well that Sprockets is totally out of line with where everything seems to be moving in the world of front end.

To me and I would assume many others the ideal outcome would be the ease of sprockets with the advantages of webpack.

I don’t think sprockets alone is realistically going to bridge that gap by itself. There’s way too much underlying infrastructure that has gone into Webpack from a much broader community that allows us to do all of the interesting stuff that it provides.

Instead I would really like to think about how to make Webpacker more like sprockets rather than continuing this dual use strategy where JS is processed through one pipeline and all other static assets through another.

The first obvious thing to me is approaching this from a documentation point of view. Not necessarily in the sense of “here’s what you need to know about webpacker to really get deep into it” but more “here are a bunch of essentially copy and paste recipes that will allow you to have a seamless experience in the same way you have now with Sprockets.

It’s possible (afaik) to configure things so that we can get back to the level of “just put your files in this folder and it magically works” and I think that is totally a good enough experience for most people right now. Then for those who want to use the “full power” of Webpack well we expose the config internals enough that you can do that as well if you like.

Perhaps that should be the “default” webpacker experience or put it behind a flag I don’t think it matters.

However, I feel like I have massively over simplified this and it’s super realistic I’m missing pieces of the puzzle here. Would be happy to learn what they are though

[Fjan]

These are good points and I largely agree, but while we can probably get near a “just works” experience after the initial install the whole thing remains a beast to update with each iteration. I just created an empty Rails project and just find node_modules -print | wc -l gives me 18366 files!

On a small Rails project, updating to a new version is a single bundle update and fixing a few warnings. Getting the JS infrastructure upgraded can take hours of Googling.

I appreciate that these things can do more, but we are paying dearly in terms of speed and elegance, too dearly in my opinion. In some ways we are moving backwards: why does the sassc gem take 20 times longer to install instead of the ruby sass gem, for a compile speed benefit that’s irrelevant on a regular Rails project? On the other hand why do tests run 4 times slower with webdrivers and chrome headless than on the now unmaintained capybara-webkit gem? It gives you a few new JS and CSS features that you can only use if you are willing to give up anything but the latest browsers.

Neither of these are really Rails fault, of course, but perhaps we should be stricter in saying: SASS-Rails 5 gave a better experience and we refuse to set the new version as the default until there is a solution that’s really better for our audience. If we still believe our audience includes the people looking to quickly build a small app.

[mdh (Mark)]

It sounds like we are both more or less in agreement regarding what the key issues are but maybe have differing ideas regarding how to address them.

I think the other elephant in the room for me is that Rails already has a huge image problem no matter how undeserved that might be. However, this is precisely the kind of thing that I regularly see people point at under the category of “Rails is stuck in the past” kinds of complaints you see out there.

For the record, I mostly hate the JS ecosystem and think that hype-driven-development was one of the most unfortunate things to happen to web development precisely since the 2012 time period. But I think there is also a real fallacy in the idea that things like Webpack are only for people who are trying to build SPA’s for example.

I also hear what you’re saying regarding the general feeling of lack of stability regarding the JS ecosystem. I am also one of the many people who are basically terrified of trying to do anything but the most basic changes to a build configuration tool like that and that is to say nothing of the security nightmare of dependency hell that comes with the modern JS ecosystem.

Despite this having not a lot at all to do with Rails itself, I do agree that a new project with almost 20k dependencies out of the box is not something I want anything to do with whatsoever.

But it most certainly is possible to make that reasonable. Take for example this screenshot of the node_modules folder from Hey.

The question then becomes, why don’t we make this the default (assuming that is a reasonable goal).

I guess my question to you is if hypothetically Webpacker provided an abstraction on top of Webpack itself that “just worked” for all assets and gave you a whole bunch of best practices out of the box and most importantly did so in a way that assuming you didn’t want to “do anything crazy” you could know that you will never have to look at another Webpack or JS based configuration file again would you make the switch to ditching Sprockets?

That question is also very much open to @DHH, @georgeclaghorn or @samsaffron as well I guess considering they are also big sprockets advocates as far as I am aware (apologies if I am wrong).

Again, I don’t even know if what I am proposing is even a realistic goal or not so I might be talking nonsense here but as far as I can tell I don’t yet understand why it wouldn’t be.

[Betsy_Haibel]

Hey, your friendly neighborhood moderator here.

You’re doing some things I’d like you to stop doing:

• Claiming to speak for the “majority” of Rails users when you’re just expressing your personal perspective.
• Using “controversy” as an excuse to be confrontational.
• Insisting on a specific implementation, rather than proposing a clear problem, suggesting a possible solution, and “not being married” to your initial preferred solution.

This forum has largely been a constructive place for discussion of issues with modern Rails and possible solutions to those issues. The things I’m noting in your original post tend to shut constructive conversations down and turn them into arguments.

In that spirit, I’m going to be editing your post title to be less “clickbaity” and more inviting to forum users who are interested in constructive discussion.

What I’m hearing you say, when I look for a problem rather than an implementation, is that Webpacker requires significantly more complicated end-user setup than Sprockets did and that this makes life difficult for Rails users who want to use a traditional “JS Sprinkles” frontend approach.

As one of those Rails users, I totally agree that this is a problem. But I also don’t agree with your proposed solution. When using Sprockets I’ve needed to use manual vendoring or asset gems for JS dependency management. Neither of these lends itself to staying up-to-date with dependencies.

I think @mdh’s idea about a more “omakase” Rails experience for JS Sprinkles applications sounds pretty worthwhile and I’m interested in exploring that further.

I’m having difficulties thinking of crisp technical approaches for further abstracting away Webpacker that wouldn’t make the existing “the abstraction is great up to a point, but then you fall off a cliff” issues with Webpacker even worse. But that probably just means I’m not being creative enough.

[Fjan]

Hi @Betsy_Haibel, I’m not entirely sure whether your moderation was directed at @mdh or me @Fjan? I’m not a native speaker, so I had not picked on anything in this thread that would seem too controversial or confrontational. In any case, I sincerely apologise.

Back to the issue, I entirely agree with you that Sprockets has a lot of warts (and the experience actually seemed to get a bit worse with the last version), and am certainly not proposing we should accept that as the best we can do.

What I’m challenging, is the notion that we can ever get Webpacker up to the omakase experience that made Rails/ruby so great. Even if we whittle it down to a few hundred dependencies and get the installation super smooth, I’m worried that will be putting lipstick on a pig. But perhaps that ship has sailed and I have to accept that webpacker is in every new stack for the foreseeable future.

In an ideal world someone would convert sprockets into a “webpacker light” gem that’s a drop-in replacement for the real webpacker that would have very few dependencies. Ideally, it would have a very similar file structure so an upgrade to the real thing is easy. I still think many rails projects would be happy with a less capable asset pipeline if it means a better experience.

[Betsy_Haibel]

Thanks for the apology! No need to be perfect here & I respect that there are tonal subtleties that can get misread when people aren’t writing in their native language. All I care about is that we have constructive, welcoming conversations in this forum.

I agree with you about the difficulties of getting Webpacker up to an polished omakase experience. I see this as a “culture clash” issue between the JavaScript community and the Rails community. Historically, the JavaScript community has aggressively valued the freedom of getting to configure things down to the last fiddly detail, and of course the Rails perspective is “convention over configuration.” While the JavaScript community has finally started to come around to the idea that convention is better than configuration situps, there’s a heck of a lot of JavaScript code that was written in a pro-configuration world. All that code is now technical debt that needs to be resolved. And there’s still a values clash – the average JS developer is still more pro-configuration than the average Rails developer, and the average JS developer is going to prefer JS-heavy web development practices over JS-light ones.

But I think we can’t ignore what the JS community is doing, even if it’s not to our taste. Our applications are more secure when we keep our dependencies up-to-date, and I believe that that means using the frontend community’s preferred means of package distribution. And for the most part, even CSS-only devs have converged around npm. So we need to engage with the JS community’s beliefs about application packaging. I personally don’t think Sprockets, as currently architected, does enough in that arena, and I think that it would be difficult to change in a way that would make that happen. I of course could be wrong about that.

I do think that Webpacker, or some third solution, could do a lot to improve the OOB experience for frontend development in Rails. I just personally believe that we need the Rails JS solution to do the following:

• use the package registry that the majority of the JS community overall uses
• not create too many WTFs when people try to use general frontend (rather than Rails-specific frontend) tutorials for How to Do A Thing.

[Fjan]

The question is whether we can be both the friendly-to-beginner Rails that got me hooked in Rails 1 (yes, I"m old) and the friendly-to-pro Rails that powers huge enterprises. Rails is doing a great job in the “pro” category, this whole May of WTF is to find out if we can do better in the first. I’m optimistic it can be done, this introspection is proof of a community in rude health.

Although I agree any attempt to improve the current asset train is valuable, I’m not too optimistic it will ever get to a state that makes Rails beginner-friendly again. I dare say that on the average Rails + JS sprinkles project there is no need for a JS package manager, and almost no need to keep dependencies up to date, because there should be hardly any JS packages and dependencies.

Rails still runs perfectly fine without Webpacker, but you quickly start running into the limits of Sprockets if you do want to do more than just sprinkles and the path of moving from one to the other is fraught. So offering just Sprockets may be too limiting, the full 18.000 file Webpacker dependency fest is nobody’s idea of beginner-friendly.

Now for solutions. I read in different threads around here:

• “Improve sprockets.” Easier said than done, where do you find volunteers to work on that.
• “Webpacker 5 will be so much better.” Perhaps, I’m sceptical it will get to where we aspire.

Perhaps we can add this up and divide by two (does that make sense in English?). Bend both so the migration from one to the other is smooth and well documented. New rails installs start people out with training wheels, without Webpacker. People who outgrow this or who know they want an SPA will just type rails generate asset-pipeline-pro. Does that sound reasonable / doable?

[DHH]

Happy to discuss all the concrete proposals on how to make things better. Right now Webpacker provides a out of the box experience where all the major stuff just works. Yes, that requires a shit ton of dependencies, but that’s only on the dev side. We use more-or-less vanilla Webpacker at Basecamp, but that’s just such that we get proper transpiling. Our actual dependencies in terms of what we ship is tiny, because we just use Turbolinks/Stimulus/etc. No React/Vue/Whatever.

I don’t think there’s any future in making Sprockets work with modern JavaScript. That ship has sailed. That’s why we’re on Webpack! But until Webpack provides a superior experience for css/images/fonts/whatever, Sprockets is great.

Don’t really give much a hoot of “image” problems. We diagnose problems, device solutions, and then if that appeals, great, and if it doesn’t appeal, also great .

[Fjan]

@DHH thanks for your attention. Do you think transpiling will still be needed in 2021? I completely understand why large apps need packaging, tree-shaking and what-not, but the average CRUD app needs little more than some concatenation and perhaps uglifier.

To serve both audiences, would a “Rails of two speeds” be an option? With a great experience for the quick & dirty CRUD app, and that can easily be upgraded to full pro?

[DHH]

You’re still going to want modules, npm, etc. Sprockets is never going to do all of that. All material apps need packaging. Webpacker is no more complicated than Sprockets if you have vanilla needs where the conventional configurations don’t even need to be touched. Don’t think we need two speeds.

Also, nobody is going to be interested in maintaining multiple distribution forms for Turbolinks, Trix, or anything else going forward. That ship has sailed. Modules is it.

The size of node_modules during development is really low on the list of priorities.

THAT said, who am I to deny anyone the pleasure of using Sprockets for JS . Sprockets was originally designed for JS, and it continues to do that job fine too (Basecamp 3 does its original JS on Sprockets, and anything newer on Webpack!). So you most certainly still can use Sprockets for JS, if you really wanted. But I don’t see Rails going back to recommending that.

[rossta (Ross Kaffenberger)]

@Betsy_Haibel Just want to say this is a really thoughtful response to this thread and I appreciate the hard work you’re doing to moderate.

[AndyObtiva (Andy Maleh)]

+1 for this.

I disagree with DHH as I’ve used Browserify with Sprockets before and was able to do true ES6 that way without Webpack.

And even if that’s not the best solution, this post is a cry for help that should not go unheeded. I think there is a need for a better solution than Webpack (or Webpacker). Perhaps ES6 support baked into Sprockets? Or perhaps an intelligent innovative all new solution from the guys who brought us Rails? Anything is possible and would be better than what JS folks offer.

How about finally putting serious consideration into Opal (GitHub - opal/opal: Ruby ♥︎ JavaScript)?

We’re Ruby developers after all, and expect Programmer Happiness without lame excuses.

[Betsy_Haibel]

@AndyObtiva

I’ve used Browserify with Sprockets before and was able to do true ES6 that way without Webpack.

Can you go more into how you managed dependency upgrades in that context? I believe that what OP was asking for was a holistic solution that let him not think about JS packaging, which is a reasonable thing to want, but it’s not clear to me how Browserify + manual npm configuration solves for that want.

@AndyObtiva

We’re Ruby developers after all, and expect Programmer Happiness without lame excuses.

As moderator, I would like you to rephrase this statement. I’d like you to do the following:

• Remove the ableist language.
• Clarify what you think these “excuses” are. Engage with the technical difficulties at hand rather than dismissing others’ good-faith attempts to find a solution.
• Ditch the entitlement. Rails is built by volunteers – either people working in their precious free time, or people who have fought hard to find a job which allows them to work on open-source projects that aren’t company-branded. The goal of the May of WTFs is to build Rails community and to help the existing maintainers serve the community better. It’s not a forum for making demands on people’s volunteer time.

[zerocool4u2 (Alexis Ramis)]

Hi, I’ve been working with rails for 10 years now and have to say that this transition has been the hardest yet for me. I didn’t have any experience on js frameworks, get used to use coffee instead of vanilla(I know that I can still use it, but having both seem disorganized to me) and never done anything that complicated with js except some ajax here and there, but after been working with webpacker for a couple of months, I can safely say that I have it under control. One thing I realise now, is that all my complications came from googling “rails 6 webpacker configuration”, I just got tons of differents ways of having all working out, doing all sorts of things just to enable basic common tools and all my problems could be solved by just googling “webpack configuration”. Seriously, a couple guides of people who really understand the tool gave more insight that 50 rails guides on the topic and was a lot more easy to figure it out each tool after that.

The moral of my history, this was hard for me because it was too different of what I was used to and could be solved with better documentation and guides that explains the cores of the tool rather than the problems you might encountered. I know that not knowing that webpack was a thing is on me, but it took me while to grasp it.

This days I feel a lot more comfortable with webpacker, everything is working smooth. Do I need it, honestly no, but is nice to know that I have access to more tools these days.

Some things that I still don’t like:

• I don’t know yet how to organize my assets in a clean way, currently I have all my js in the javascript folder, just import my css from packs in there and all my customs css in the old assets folder, I would like to have an easy way to import some assets from packs to my old assets folder or vice versa, mostly the ones I want to override, so I could just have one application.css on my site. Having css in the javascript folder feels so wrong for me.
• Many have pointed out the long list of dependencies, that was scary at first and I still don’t know why are they there(like fully understand it), but they doesn’t seem to be carried out into my compilated assets, so I don’t care.
• It would be nice to have the configuration side in plain ruby, rather than js, maybe a simple dsl or block that make you more comfortable tweaking it, the webpack configuration system feels really alienistick.
• I keep getting compilations errors on webpack at random times and without descriptions on where they are originating, just “webpack error compilation” or something like that, the weird part is that they would appear with me doing nothing of those files and would fix themselves just by adding some random blank line to trigger a compilation on the webpack side, granted it could be me having livereload and guard running in there.

Those are my two cents, thanks for the good work, it’s always appreciated.

[rmacklin]

@Fjan

I just created an empty Rails project and just find node_modules -print | wc -l gives me 18366 files!

This line reminded me of this post from the other thread discussing sprockets and webpacker:

Sprockets abandonment

A lot has been said already! I’ll add a few extra bits: If webpack becomes the one and only solution at some point, people need to fully realise the incredible number of sub-dependencies this brings in (to fully grasp it, check out this visualisation). For apps with limited front-end needs, this increases the attack surface, some would say for little benefit. I wonder if Rollup (Visualization of npm dependencies) could be used more in those cases, as pointed out recently to me by @Ken_Collins1 …

I’d not seen those visualizations of the dependency graphs before (webpack vs rollup), but we chose to use rollup instead of webpack on one of our projects in part because of its simpler domain (and also because it produced smaller javascript bundles). I know throwing another tool in the ring might not be what you’re looking for, but I can say anecdotally that we’ve had a great experience with rollup in that project.

[rossta (Ross Kaffenberger)]

There’s no doubt webpack uses a lot of dependencies. But that visualization comparison may lead people to believe rollup has only one dependency when in fact it has quite a few as well: rollup/package.json at aa33e4b9250ba9f2cfb216e58f31638147ce34b5 · rollup/rollup · GitHub.

The difference is in distribution. Rollup packages its dependencies into a single distribution whereas webpack expects its dependencies to be installed as separate packages. If security concerns are an argument for choosing one over the other, then all dependencies should be considered.

[vascoosx (Shunichi)]

I agree with this article Separating webpack from the Rails Asset Pipeline | Spencer Miskoviak . Webpacker was not for me because I was using rollup. Anyways, the author proposes his own method to integrate webpack but it introduces new methods. On the other hand I used the gems external_asset_pipeline and npm-pipeline-rails and now I can use anything from sprockets and anything from my own favorite framework without any difficulties. Here is a demo

[Thibaut_Barrere (Thibaut Barrère)]

@rossta

The difference is in distribution.

I noticed another difference (but I cannot tell how relevant this is, yet - I haven’t fully compared the actual content, so it is just a wild ballpark):

mkdir test-rollup test-webpack
npm install --prefix test-rollup rollup
npm install --prefix test-webpack webpack
du -s -h test*

The current output for me is:

2.8M	test-rollup
 28M	test-webpack

I will have to look closer, likely there will be some artefacts / maps / debug stuff in one which isn’t in another, but from the outside at least, it is a 10x difference.

[Betsy_Haibel]

Right now I’m not really sure what the arguments for rollup are other than “I like it better.” Could Team Rollup break down the advantages they see in a Rollup-based approach?