Methodology

How the benchmarks are measured, where they run, and how to read the results.

Method

Each tool is configured to behave as much like its peers as possible, but standardisation has limits — tools differ in what they install by default, and in which security checks and validations they run. So the suite counts the packages each tool actually installed, after installation, and publishes both figures: total install time, and time per package. A tool that installs fewer packages finishes sooner without necessarily being faster, which is why neither number is reported alone.

Environment

Workflow label
ubuntu-24.04-arm
VM
Linux (Ubuntu 24.04)
Architecture
arm64
Processor (CPU)
4 cores
Memory (RAM)
16 GB
Storage (SSD)
14 GB

The label is read from the workflow. The remaining figures are GitHub's published specification for standard public runners, not measurements taken here.

How to read the results

  • Total time is the measurement and the ranking: the mean wall-clock install (or, for task runners, execution) time, over the fixtures a tool completed. Lower is faster.
  • DNF: the run exited non-zero. It is excluded from every mean, chart and ranking, never counted as a time, and reported as DNF.
  • Partial: a tool that did not complete every fixture. Its mean covers fewer fixtures (shown as n/N), so it is not ranked against tools that did, and it gets no bar.
  • ±RSD: the pooled relative standard deviation across a tool's repeated runs.
  • Derived metrics are computed from total time and labelled "derived": time per installed package, change since the start of the window, and time relative to the baseline.
  • The baseline is npm: the npm client for package managers, registry.npmjs.org for registries, npm run for task runners. It is a reference point, not the tool to beat.
  • Results regenerate nightly. A run is named and cited by its date; ?date=YYYY-MM-DD pins one.

Scenarios

Registries

The same projects installed with the same npm client from each registry, so the registry is the only thing that changes.

registry-clean
A cold install, holding the package manager constant and varying the registry.
registry-lockfile
A lockfile install, holding the package manager constant and varying the registry.

Package managers

The same projects installed by each package manager, from every combination of warm and cold cache, lockfile and node_modules.

clean
A first install on a fresh machine. Every package is resolved, downloaded and linked from scratch.
  • binary preinstalled
  • cache empty
  • lockfile absent
  • node_modules removed
cache
The package cache is warm. Nothing else is: no lockfile, no node_modules.
  • binary preinstalled
  • cache present
  • lockfile absent
  • node_modules removed
lockfile
A lockfile pins the graph, but nothing is cached and nothing is installed. Resolution is skipped; download is not.
  • binary preinstalled
  • cache empty
  • lockfile present
  • node_modules removed
node_modules
node_modules is already populated. The question is how quickly a tool can decide there is little to do.
  • binary preinstalled
  • cache empty
  • lockfile absent
  • node_modules present
cache+lockfile
Cache and lockfile both warm. No node_modules to reuse.
  • binary preinstalled
  • cache present
  • lockfile present
  • node_modules removed
cache+node_modules
Cache and node_modules present, with no lockfile to pin the graph.
  • binary preinstalled
  • cache present
  • lockfile absent
  • node_modules present
lockfile+node_modules
Lockfile and node_modules present, with a cold cache.
  • binary preinstalled
  • cache empty
  • lockfile present
  • node_modules present
cache+lockfile+node_modules
Everything warm. The best case, and usually dominated by how fast a tool can decide it has nothing to do.
  • binary preinstalled
  • cache present
  • lockfile present
  • node_modules present
ci
The tool's CI-oriented install — npm ci and its equivalents. Requires a lockfile and installs strictly from it.
  • binary preinstalled
  • cache empty
  • lockfile present
  • node_modules removed
ci+setup
Published, with no description yet.
  • binary preinstalled
  • cache empty
  • lockfile present
  • node_modules removed

Task runners

The same build and the same script started by each runner, measuring what the runner adds.

build
A project build after install. Measures the build, not the install.
build-cache
The same build with a warm build cache.
run
Script execution only: process startup, reading package.json, and spawning the command.

Resources

All trademarks, logos and brand names are the property of their respective owners. All company, product and service names used on this site are for identification purposes only. Use of these names, trademarks and brands does not imply endorsement.