Two weeks ago we introduced the serverless-jetpack plugin, a drop-in replacement for Serverless Framework packaging. We thought that we had sped up serverless
CLI packaging and deployments as much as possible.
It turns out we were wrong.
We discovered we could make it even faster...
Reexamining the problem
As we discussed previously, serverless
CLI packaging slows to a crawl when reading large node_modules
directories on disk, only to later exclude most files from application packaging as devDependencies
.
Our solution with serverless-jetpack
was to avoid any introspection of node_modules
by doing a yarn
or npm
production installation in a temporary directory. This takes a bit of time, but often far less than globbing on disk with built-in serverless
packaging.
In the course of our initial release, we received great feedback from the community, including some undiscovered issues and new ideas as to how to tackle the speed problem.
Back to the drawing board
The existing design of Jetpack offers an easy means of replacing built-in serverless
packaging with a new heuristic of our choosing. Jetpack also has a full benchmark and integration test suite to ensure that the plugin is faster than serverless
while still correctly producing identical packaging output.
...so maybe we can take a different approach?
After a seemingly promising false start, we reexamined the problem. We need to not read devDependencies
. How else could we do it?
Turns out that we have already tackled this problem in depth with our inspectpack project, which does detailed bundle and dependency analysis of Webpack-built JavaScript applications.
Using techniques pioneered in inspectpack
, we created a small, focused library named inspectdep, which efficiently reads node_modules
and discovers the on-disk locations of production modules. By hooking inspectdep
into Jetpack we were able to:
- Quickly identify production dependencies before globbing.
- Use that information to skip disk I/O for
devDependencies
by reading only production dependencies in a project'snode_modules
when globbing.
In effect, Jetpack is able to do pretty much the same thing that serverless
does, just much, much faster.
A faster, more powerful Jetpack
With this refactor, the severless-jetpack
plugin is much simpler.
There are no yarn
or npm
installs anymore. All of the dependency inference is performed in pure JavaScript code. This also makes the plugin more compatible with a wider range of Serverless Framework applications.
There are no more configuration options! The plugin "just works" with existing include|exclude
Serverless packaging configurations. If it works with serverless
, it should work with serverless-jetpack
.
So, how much faster is it? Let's check out some new benchmarks!
This time around, we have Windows VM timings from our new AppVeyor CI. The speedup for Windows seems to be even more dramatic—for the huge
scenario, serverless-jetpack
is over 15 times faster than built-in serverless
packaging in our CI setup.
OS | Scenario | Type | Time | vs Base |
---|---|---|---|---|
Mac | simple | jetpack | 4338 | -46.37 % |
Mac | simple | baseline | 8089 | |
Mac | individually | jetpack | 2964 | -76.77 % |
Mac | individually | baseline | 12760 | |
Mac | huge | jetpack | 4524 | -84.03 % |
Mac | huge | baseline | 28321 | |
Windows | simple | jetpack | 2890 | -73.88 % |
Windows | simple | baseline | 11063 | |
Windows | individually | jetpack | 3110 | -83.90 % |
Windows | individually | baseline | 19312 | |
Windows | huge | jetpack | 3500 | -93.34 % |
Windows | huge | baseline | 52548 |
Fire it up!
We've been very pleased with the reception of serverless-jetpack since its introduction. Our internal refactoring should now make it better and easier to integrate in Serverless Framework applications. We hope the newer, shinier Jetpack speeds up your serverless
packaging even more! 🚀