Webpacker presents a more difficult OOB experience for JS Sprinkles than Sprockets did

It sounds like we are both more or less in agreement regarding what the key issues are but maybe have differing ideas regarding how to address them.

I think the other elephant in the room for me is that Rails already has a huge image problem no matter how undeserved that might be. However, this is precisely the kind of thing that I regularly see people point at under the category of “Rails is stuck in the past” kinds of complaints you see out there.

For the record, I mostly hate the JS ecosystem and think that hype-driven-development was one of the most unfortunate things to happen to web development precisely since the 2012 time period. But I think there is also a real fallacy in the idea that things like Webpack are only for people who are trying to build SPA’s for example.

I also hear what you’re saying regarding the general feeling of lack of stability regarding the JS ecosystem. I am also one of the many people who are basically terrified of trying to do anything but the most basic changes to a build configuration tool like that and that is to say nothing of the security nightmare of dependency hell that comes with the modern JS ecosystem.

Despite this having not a lot at all to do with Rails itself, I do agree that a new project with almost 20k dependencies out of the box is not something I want anything to do with whatsoever.

But it most certainly is possible to make that reasonable. Take for example this screenshot of the node_modules folder from Hey.

The question then becomes, why don’t we make this the default (assuming that is a reasonable goal).

I guess my question to you is if hypothetically Webpacker provided an abstraction on top of Webpack itself that “just worked” for all assets and gave you a whole bunch of best practices out of the box and most importantly did so in a way that assuming you didn’t want to “do anything crazy” you could know that you will never have to look at another Webpack or JS based configuration file again would you make the switch to ditching Sprockets?

That question is also very much open to @DHH, @georgeclaghorn or @samsaffron as well I guess considering they are also big sprockets advocates as far as I am aware (apologies if I am wrong).

Again, I don’t even know if what I am proposing is even a realistic goal or not so I might be talking nonsense here but as far as I can tell I don’t yet understand why it wouldn’t be.

4 Likes

Hey, your friendly neighborhood moderator here.

You’re doing some things I’d like you to stop doing:

  • Claiming to speak for the “majority” of Rails users when you’re just expressing your personal perspective.
  • Using “controversy” as an excuse to be confrontational.
  • Insisting on a specific implementation, rather than proposing a clear problem, suggesting a possible solution, and “not being married” to your initial preferred solution.

This forum has largely been a constructive place for discussion of issues with modern Rails and possible solutions to those issues. The things I’m noting in your original post tend to shut constructive conversations down and turn them into arguments.

In that spirit, I’m going to be editing your post title to be less “clickbaity” and more inviting to forum users who are interested in constructive discussion.

What I’m hearing you say, when I look for a problem rather than an implementation, is that Webpacker requires significantly more complicated end-user setup than Sprockets did and that this makes life difficult for Rails users who want to use a traditional “JS Sprinkles” frontend approach.

As one of those Rails users, I totally agree that this is a problem. But I also don’t agree with your proposed solution. When using Sprockets I’ve needed to use manual vendoring or asset gems for JS dependency management. Neither of these lends itself to staying up-to-date with dependencies.

I think @mdh’s idea about a more “omakase” Rails experience for JS Sprinkles applications sounds pretty worthwhile and I’m interested in exploring that further.

I’m having difficulties thinking of crisp technical approaches for further abstracting away Webpacker that wouldn’t make the existing “the abstraction is great up to a point, but then you fall off a cliff” issues with Webpacker even worse. But that probably just means I’m not being creative enough. :slight_smile:

10 Likes

Hi @Betsy_Haibel, I’m not entirely sure whether your moderation was directed at @mdh or me @Fjan? I’m not a native speaker, so I had not picked on anything in this thread that would seem too controversial or confrontational. In any case, I sincerely apologise.

Back to the issue, I entirely agree with you that Sprockets has a lot of warts (and the experience actually seemed to get a bit worse with the last version), and am certainly not proposing we should accept that as the best we can do.

What I’m challenging, is the notion that we can ever get Webpacker up to the omakase experience that made Rails/ruby so great. Even if we whittle it down to a few hundred dependencies and get the installation super smooth, I’m worried that will be putting lipstick on a pig. But perhaps that ship has sailed and I have to accept that webpacker is in every new stack for the foreseeable future.

In an ideal world someone would convert sprockets into a “webpacker light” gem that’s a drop-in replacement for the real webpacker that would have very few dependencies. Ideally, it would have a very similar file structure so an upgrade to the real thing is easy. I still think many rails projects would be happy with a less capable asset pipeline if it means a better experience.

3 Likes

Thanks for the apology! No need to be perfect here & I respect that there are tonal subtleties that can get misread when people aren’t writing in their native language. All I care about is that we have constructive, welcoming conversations in this forum.

I agree with you about the difficulties of getting Webpacker up to an polished omakase experience. I see this as a “culture clash” issue between the JavaScript community and the Rails community. Historically, the JavaScript community has aggressively valued the freedom of getting to configure things down to the last fiddly detail, and of course the Rails perspective is “convention over configuration.” While the JavaScript community has finally started to come around to the idea that convention is better than configuration situps, there’s a heck of a lot of JavaScript code that was written in a pro-configuration world. All that code is now technical debt that needs to be resolved. And there’s still a values clash – the average JS developer is still more pro-configuration than the average Rails developer, and the average JS developer is going to prefer JS-heavy web development practices over JS-light ones.

But I think we can’t ignore what the JS community is doing, even if it’s not to our taste. Our applications are more secure when we keep our dependencies up-to-date, and I believe that that means using the frontend community’s preferred means of package distribution. And for the most part, even CSS-only devs have converged around npm. So we need to engage with the JS community’s beliefs about application packaging. I personally don’t think Sprockets, as currently architected, does enough in that arena, and I think that it would be difficult to change in a way that would make that happen. I of course could be wrong about that.

I do think that Webpacker, or some third solution, could do a lot to improve the OOB experience for frontend development in Rails. I just personally believe that we need the Rails JS solution to do the following:

  • use the package registry that the majority of the JS community overall uses
  • not create too many WTFs when people try to use general frontend (rather than Rails-specific frontend) tutorials for How to Do A Thing.
4 Likes

The question is whether we can be both the friendly-to-beginner Rails that got me hooked in Rails 1 (yes, I"m old) and the friendly-to-pro Rails that powers huge enterprises. Rails is doing a great job in the “pro” category, this whole May of WTF is to find out if we can do better in the first. I’m optimistic it can be done, this introspection is proof of a community in rude health.

Although I agree any attempt to improve the current asset train is valuable, I’m not too optimistic it will ever get to a state that makes Rails beginner-friendly again. I dare say that on the average Rails + JS sprinkles project there is no need for a JS package manager, and almost no need to keep dependencies up to date, because there should be hardly any JS packages and dependencies.

Rails still runs perfectly fine without Webpacker, but you quickly start running into the limits of Sprockets if you do want to do more than just sprinkles and the path of moving from one to the other is fraught. So offering just Sprockets may be too limiting, the full 18.000 file Webpacker dependency fest is nobody’s idea of beginner-friendly.

Now for solutions. I read in different threads around here:

  • “Improve sprockets.” Easier said than done, where do you find volunteers to work on that.
  • “Webpacker 5 will be so much better.” Perhaps, I’m sceptical it will get to where we aspire.

Perhaps we can add this up and divide by two (does that make sense in English?). Bend both so the migration from one to the other is smooth and well documented. New rails installs start people out with training wheels, without Webpacker. People who outgrow this or who know they want an SPA will just type rails generate asset-pipeline-pro. Does that sound reasonable / doable?

1 Like

Happy to discuss all the concrete proposals on how to make things better. Right now Webpacker provides a out of the box experience where all the major stuff just works. Yes, that requires a shit ton of dependencies, but that’s only on the dev side. We use more-or-less vanilla Webpacker at Basecamp, but that’s just such that we get proper transpiling. Our actual dependencies in terms of what we ship is tiny, because we just use Turbolinks/Stimulus/etc. No React/Vue/Whatever.

I don’t think there’s any future in making Sprockets work with modern JavaScript. That ship has sailed. That’s why we’re on Webpack! But until Webpack provides a superior experience for css/images/fonts/whatever, Sprockets is great.

Don’t really give much a hoot of “image” problems. We diagnose problems, device solutions, and then if that appeals, great, and if it doesn’t appeal, also great :smile:.

4 Likes

@DHH thanks for your attention. Do you think transpiling will still be needed in 2021? I completely understand why large apps need packaging, tree-shaking and what-not, but the average CRUD app needs little more than some concatenation and perhaps uglifier.

To serve both audiences, would a “Rails of two speeds” be an option? With a great experience for the quick & dirty CRUD app, and that can easily be upgraded to full pro?

2 Likes

You’re still going to want modules, npm, etc. Sprockets is never going to do all of that. All material apps need packaging. Webpacker is no more complicated than Sprockets if you have vanilla needs where the conventional configurations don’t even need to be touched. Don’t think we need two speeds.

Also, nobody is going to be interested in maintaining multiple distribution forms for Turbolinks, Trix, or anything else going forward. That ship has sailed. Modules is it.

The size of node_modules during development is really low on the list of priorities.

THAT said, who am I to deny anyone the pleasure of using Sprockets for JS :smile:. Sprockets was originally designed for JS, and it continues to do that job fine too (Basecamp 3 does its original JS on Sprockets, and anything newer on Webpack!). So you most certainly still can use Sprockets for JS, if you really wanted. But I don’t see Rails going back to recommending that.

7 Likes

@Betsy_Haibel Just want to say this is a really thoughtful response to this thread and I appreciate the hard work you’re doing to moderate.

8 Likes

+1 for this.

I disagree with DHH as I’ve used Browserify with Sprockets before and was able to do true ES6 that way without Webpack.

And even if that’s not the best solution, this post is a cry for help that should not go unheeded. I think there is a need for a better solution than Webpack (or Webpacker). Perhaps ES6 support baked into Sprockets? Or perhaps an intelligent innovative all new solution from the guys who brought us Rails? Anything is possible and would be better than what JS folks offer.

How about finally putting serious consideration into Opal (GitHub - opal/opal: Ruby ♥︎ JavaScript)?

We’re Ruby developers after all, and expect Programmer Happiness without lame excuses.

2 Likes

Can you go more into how you managed dependency upgrades in that context? I believe that what OP was asking for was a holistic solution that let him not think about JS packaging, which is a reasonable thing to want, but it’s not clear to me how Browserify + manual npm configuration solves for that want.

As moderator, I would like you to rephrase this statement. I’d like you to do the following:

  • Remove the ableist language.
  • Clarify what you think these “excuses” are. Engage with the technical difficulties at hand rather than dismissing others’ good-faith attempts to find a solution.
  • Ditch the entitlement. Rails is built by volunteers – either people working in their precious free time, or people who have fought hard to find a job which allows them to work on open-source projects that aren’t company-branded. The goal of the May of WTFs is to build Rails community and to help the existing maintainers serve the community better. It’s not a forum for making demands on people’s volunteer time.
12 Likes

Hi, I’ve been working with rails for 10 years now and have to say that this transition has been the hardest yet for me. I didn’t have any experience on js frameworks, get used to use coffee instead of vanilla(I know that I can still use it, but having both seem disorganized to me) and never done anything that complicated with js except some ajax here and there, but after been working with webpacker for a couple of months, I can safely say that I have it under control. One thing I realise now, is that all my complications came from googling “rails 6 webpacker configuration”, I just got tons of differents ways of having all working out, doing all sorts of things just to enable basic common tools and all my problems could be solved by just googling “webpack configuration”. Seriously, a couple guides of people who really understand the tool gave more insight that 50 rails guides on the topic and was a lot more easy to figure it out each tool after that.

The moral of my history, this was hard for me because it was too different of what I was used to and could be solved with better documentation and guides that explains the cores of the tool rather than the problems you might encountered. I know that not knowing that webpack was a thing is on me, but it took me while to grasp it.

This days I feel a lot more comfortable with webpacker, everything is working smooth. Do I need it, honestly no, but is nice to know that I have access to more tools these days.

Some things that I still don’t like:

  • I don’t know yet how to organize my assets in a clean way, currently I have all my js in the javascript folder, just import my css from packs in there and all my customs css in the old assets folder, I would like to have an easy way to import some assets from packs to my old assets folder or vice versa, mostly the ones I want to override, so I could just have one application.css on my site. Having css in the javascript folder feels so wrong for me.
  • Many have pointed out the long list of dependencies, that was scary at first and I still don’t know why are they there(like fully understand it), but they doesn’t seem to be carried out into my compilated assets, so I don’t care.
  • It would be nice to have the configuration side in plain ruby, rather than js, maybe a simple dsl or block that make you more comfortable tweaking it, the webpack configuration system feels really alienistick.
  • I keep getting compilations errors on webpack at random times and without descriptions on where they are originating, just “webpack error compilation” or something like that, the weird part is that they would appear with me doing nothing of those files and would fix themselves just by adding some random blank line to trigger a compilation on the webpack side, granted it could be me having livereload and guard running in there.

Those are my two cents, thanks for the good work, it’s always appreciated.

7 Likes

This line reminded me of this post from the other thread discussing sprockets and webpacker:

I’d not seen those visualizations of the dependency graphs before (webpack vs rollup), but we chose to use rollup instead of webpack on one of our projects in part because of its simpler domain (and also because it produced smaller javascript bundles). I know throwing another tool in the ring might not be what you’re looking for, but I can say anecdotally that we’ve had a great experience with rollup in that project.

1 Like

There’s no doubt webpack uses a lot of dependencies. But that visualization comparison may lead people to believe rollup has only one dependency when in fact it has quite a few as well: rollup/package.json at aa33e4b9250ba9f2cfb216e58f31638147ce34b5 · rollup/rollup · GitHub.

The difference is in distribution. Rollup packages its dependencies into a single distribution whereas webpack expects its dependencies to be installed as separate packages. If security concerns are an argument for choosing one over the other, then all dependencies should be considered.

I agree with this article Separating webpack from the Rails Asset Pipeline | Spencer Miskoviak . Webpacker was not for me because I was using rollup. Anyways, the author proposes his own method to integrate webpack but it introduces new methods. On the other hand I used the gems external_asset_pipeline and npm-pipeline-rails and now I can use anything from sprockets and anything from my own favorite framework without any difficulties. Here is a demo

1 Like

I noticed another difference (but I cannot tell how relevant this is, yet - I haven’t fully compared the actual content, so it is just a wild ballpark):

mkdir test-rollup test-webpack
npm install --prefix test-rollup rollup
npm install --prefix test-webpack webpack
du -s -h test*

The current output for me is:

2.8M	test-rollup
 28M	test-webpack

I will have to look closer, likely there will be some artefacts / maps / debug stuff in one which isn’t in another, but from the outside at least, it is a 10x difference.

2 Likes

Right now I’m not really sure what the arguments for rollup are other than “I like it better.” Could Team Rollup break down the advantages they see in a Rollup-based approach?

1 Like

This isn’t a vote for it necessarily but as far as I know it’s main selling point was having a substantially smaller learning curve. I have no idea if that’s true however.

My personal experience with Rollup is that that is only true in very specific circumstances, mostly around node rather than web JS packaging, but I’m open to others’ mileages varying.

1 Like

From the docs:

Feature webpack/webpack node-browserify rollup/rollup
Additional chunks are loaded on demand yes no no
AMD define yes deamdify rollup-plugin-amd
AMD require yes no no
AMD require loads on demand yes no no
CommonJS exports yes yes commonjs-plugin
CommonJS require yes yes commonjs-plugin
CommonJS require.resolve yes no no
Concat in require require("./fi" + "le") yes no no
Debugging support SourceUrl, SourceMaps SourceMaps SourceUrl, SourceMaps
Dependencies 19MB / 127 packages 1.2MB / 1 package ?MB / 3 packages
ES2015 import/export yes (webpack 2) no yes
Expressions in require (guided) require("./templates/" + template) yes (all files matching included) no no
Expressions in require (free) require(moduleName) with manual configuration no no
Generate a single bundle yes yes yes
Indirect require var r = require; r("./file") yes no no
Load each file separate no no no
Mangle path names yes partial not required (path names are not included in the bundle)
Minimizing Terser uglifyify uglify-plugin
Multi pages build with common bundle with manual configuration with manual configuration no
Multiple bundles yes with manual configuration no
Node.js built-in libs require("path") yes yes node-resolve-plugin
Other Node.js stuff process, __dir/filename, global process, __dir/filename, global global (commonjs-plugin)
Plugins yes yes yes
Preprocessing loaders, transforms transforms plugin transforms
Replacement for browser web_modules, .web.js, package.json field, alias configuration option package.json field, alias option no
Requirable files file system file system file system or through plugins
Runtime overhead 243B + 20B per module + 4B per dependency 415B + 25B per module + (6B + 2X) per dependency none for ES2015 modules (other formats may have)
Watch mode yes watchify rollup-watch

:diamonds: in production mode (opposite in development mode)

X is the length of the path string


Bundlephobia is nice for comparing packages.

1 Like