About a year ago I noticed various benchmarks such as running specs were running abnormally slow on macOS. Noah Gibbs started some basic investigation but did not come up with anything definitive.
Since then I crafted a version of Discourse Bench (used at: https://rubybench.org/) that is there to profile “performance” of local development machines.
It is a simple benchmark that measures performance of various tasks you are likely to run into in development:
- Booting Rails
- Running a spec
- Visiting a very common page
- Creating a topic
- Converting text to HTML via our markdown library
It also contains some micro benches to help isolate culprits including:
- PBKDF2 calculation
- Filesystem random read access
The benchmark lives here, anyone can run it.
git clone https://github.com/SamSaffron/discourse-dev-benchmark.git
bundle
./bench
We have run the benchmark on many of our development machines and noticed a trend. The same CPU tends to perform a lot better (especially boot) on either Linux native or WSL2 (virtualized Linux on Windows) than it does on MacOS
Additionally, I took the time to run the same benchmark on a dual booting Mac that I have. Here are some examples of the benchmarks we got (abbreviated, cause we have many benches):
Booting Rails
In this benchmark we simply run rails r Post.first.
| desc | seconds | variance | percent |
|---|---|---|---|
| Desktop - i9-9900K Linux Mint | 1.356 | (±1.7%) | 100.0% |
| Desktop - 9900KF - Asus Prime Z370 - nvme - Arch | 1.375 | (±1.0%) | 101.4% |
| Desktop - 9900KF - Asus Prime Z370 - nvme - WSL2 | 1.407 | (±2.7%) | 103.76% |
| Desktop - Ryzen 3900X - NVME - Ubuntu | 1.417 | (±0.4%) | 104.5% |
| Lenovo P1 - Gen 2 - 9850H - WSL2 - Ubuntu | 1.433 | (±1.2%) | 105.68% |
| Laptop - XPS 15 - 9880HK - Linux 5.5.8 | 1.439 | (±0.6%) | 106.12% |
| Lenovo P1 - Gen 2 - Arch Native - 9850H | 1.539 | (±1.0%) | 113.5% |
| Desktop: AMD Ryzen 3950X, Arch, NVMe, ruby 2.6.5 (rvm) | 1.728 | (±3.2%) | 127.43% |
| Oryx Pro - i7-9750H CPU @ 2.60GHz, 32GB, NVMe. | 1.787 | (±4.5%) | 131.78% |
| MacBook Pro (16-inch, 2019) - Intel(R) Core™ i9-9980HK CPU @ 2.40GHz | 2.51 | (±2.4%) | 185.1% |
| MBP i7-3615QM, Windows WSL 2 (2.3 quad mid 2012) | 2.52 | (±0.3%) | 185.84% |
| MBP i7-9750H @ 2.6GHz (15", 2019, macOS Catalina) | 3.046 | (±1.7%) | 224.63% |
| Lenovo P1 - Gen 2 - 9850H - WSL1 - Ubuntu | 3.06 | (±0.9%) | 225.66% |
| MBP i7-3615QM, MacOS Catalina (2.3 quad mid 2012) | 3.953 | (±0.1%) | 291.52% |
Observations
-
An ancient 2012 MacBook Pro with WSL2 boots Discourse almost as fast as the state of the art MacBook Pro!
-
9880HK CPU boots about 80% faster on Linux than it does in macOS
-
WSL1 is slower on boot, but can have adequate performance if you enable bootsnap (which we do)
-
9900K by intel is the single thread king, beating both the 3900X and 3950X by AMD
-
3950X though on paper faster than the 3900X failed to beat any of our 3900X based computers on single thread benches despite the 2x price tag.
-
Top of the range laptops perform Rails boot very close to decked up desktops
-
We stopped benching on
rbenvbased systems, everyone moved tochrubyorrvmcause the shims rbenv adds introduce significant delays on boot.
Running the Discourse Post spec
In this benchmark we run: bin/rspec spec/models/post_spec.rb --seed 1, it is one of our larger spec files and is pretty representative.
| desc | seconds | variance | percent |
|---|---|---|---|
| Desktop - i9-9900K Linux Mint | 9.907 | (±0.6%) | 100.0% |
| Desktop - 9900KF - Asus Prime Z370 - nvme - Arch | 9.994 | (±0.5%) | 100.88% |
| Desktop - Ryzen 3900X - NVME - Ubuntu | 10.524 | (±0.4%) | 106.23% |
| Laptop - XPS 15 - 9880HK - Linux 5.5.8 | 10.74 | (±2.4%) | 108.41% |
| Lenovo P1 - Gen 2 - Arch Native - 9850H | 11.346 | (±0.0%) | 114.53% |
| Desktop - 9900KF - Asus Prime Z370 - nvme - WSL2 | 11.748 | (±0.5%) | 118.58% |
| Desktop: AMD Ryzen 3950X, Arch, NVMe, ruby 2.6.5 (asdf) | 11.809 | (±0.5%) | 119.2% |
| Lenovo P1 - Gen 2 - 9850H - WSL2 - Ubuntu | 12.308 | (±2.0%) | 124.24% |
| Oryx Pro - i7-9750H CPU @ 2.60GHz, 32GB, NVMe. | 12.557 | (±2.5%) | 126.75% |
| MacBook Pro (16-inch, 2019) - Intel(R) Core™ i9-9980HK CPU @ 2.40GHz | 12.963 | (±1.7%) | 130.85% |
| Mac Mini i7-8700B @ 3.2 GHz (2018, Ubuntu 19.10, USB3 SSD) | 13.971 | (±11.3%) | 141.02% |
| MBP i7-9750H @ 2.6GHz (15", 2019, macOS Catalina) | 16.173 | (±1.0%) | 163.25% |
| Lenovo P1 - Gen 2 - 9850H - WSL1 - Ubuntu | 17.598 | (±0.6%) | 177.63% |
| MBP i7 2.6 GHz, 15’’ 2018, Catalina 16Gb RAM | 20.922 | (±2.1%) | 211.18% |
| MBP i7-3615QM, MacOS Catalina (2.3 quad mid 2012) | 23.396 | (±1.7%) | 236.16% |
| MBP i7-3615QM, Windows WSL 2 (2.3 quad mid 2012) | 23.738 | (±0.6%) | 239.61% |
| (Docker) MBP i7-8559U (13-inch, 2018) | 74.232 | (±4.0%) | 749.29% |
Observation
-
Even if we compensate for the 1 second slower boot, performance of the 9980HK under macOS is still 10% or so slower than Linux using the same CPU
-
Again, to of the range laptops are very very close to top of the range desktops for running a single spec
-
Docker on Mac is a world of pain when it comes to developing Rails
-
WSL2 and Catalina on the same machine have similar performance running our spec file. It seems that in a steady state after the slow boot, Mac can beat virtualized Linux performance but remains slower than non virtualized Linux.
Random Read
In this benchmark we read 10 random files that are 10_000 bytes long from a directory containing 5000 files.
This benchmark really highlights the slowness of WSL1 and MacOS
Raw data omitted cause graphs tell the story. It appears across the board, reading lots of smallish files randomly is significantly slower on macOS which may be a big reason for the much slower boot times.
Does anyone have any ideas how we could get boot and performance on MacOS improved?
Any other thoughts here?
