What's better than a blazingly fast engine for Serverless Framework packaging and deploys?
More of them.
Meet the new serverless-jetpack plugin, now with parallel packaging!
Now, even fasterer
Since our initial release one month ago, we've been in the shop furiously hacking new ways to help wrangle even the gnarliest and biggest Serverless Framework projects. With our recent v0.4.x release, we are pleased to announce support for:
- Lerna monorepos and yarn workspaces
- Parallelized workers for packaging tasks
- Single-function packaging
If you're behind on the news, our introductory post and follow-on rewrite post are great places to see the motivation and potential usefulness of the serverless-jetpack plugin. Now, without further ado, all the new details!
Monorepo support
Monorepos, either via lerna or yarn workspaces, are a very popular way to organize source code with cross-cutting interdependencies. Many Serverless Framework applications structure monorepo repositories such that different functions are logically organized into directories like functions/* or package/*.
This can sometimes be a challenge to package correctly because a simple direct dependency like bar for a package like functions/foo could be placed in:
functions/foo/node_modules/bar node_modules/bar
... or even somewhere else if via a cross-linked package.
To this end, serverless-jetpack now provides features including service and function level options to specify places where a relevant function package.json is found for dependency inference. Jetpack can now "just take care of the rest", finding hoisted dependencies flattened to higher levels as well as symbolic links across packages.
One of our client monorepo projects used bespoke bash scripting to approximate Jetpack's previous (and now abandoned) approach of "yarn install --production in a temporary directory". The serverless package command took a full 44 minutes to run (lots of independent functions, lots of dependencies). Using Jetpack's new monorepo features, we were able to bring the total packaging time down to just NINE minutes!
Parallel workers
Serverless plugins run out-of-the-box in the same process as the serverless CLI application, which means all disk I/O and CPU work is shared in the main event loop. This can encounter bottlenecks when running serverless package (via Jetpack or built-in) over multiple functions that contend for both disk I/O (reading dependencies and source files) and CPU (bundling everything into a zip file).
Fortunately, the situation is "embarrassingly parallel", such that we should just be able to split independent units of packaging across different CPUs and processes/threads.
We identified a great abstraction library for parallelizing Jetpack: jest-worker. jest-worker is interesting in that it uses Node.js worker threads if available, and otherwise falls back to child processes. Hooking this into serverless-jetpack we split off the reading and zipping task into a function that could be run in serial as part of the main event loop or off event loop in parallel when desired.
We provide a simple configuration option, concurrency, to tailor the degree of parallelization to your specific environment and machines. Enabling parallelization in that same client monorepo project as before further reduced our serverless packaging time by more than half on developer machines and even more in CI!
Single-function packaging
One useful deployment strategy for Serverless Framework applications is to package and deploy single functions independently. To this end, serverless-jetpack now supports the built-in command:
$ serverless deploy -f {FUNCTION_NAME}
out of the box! Curiously, although there is a serverless package command to package the service and all independent functions into zip files, there is no corollary serverless package -f {FUNCTION_NAME}.
So we added our own.
Jetpack now provides a simple helper CLI to create all or some service and function packages to further facilitate bespoke packaging and deployment strategies.
$ serverless jetpack package $ serverless jetpack package -f {FUNCTION_NAME}
Up, up, and away!
We've been very pleased with the packaging results as we integrate the plugin into more Serverless Framework projects and a variety of different application configurations and setups. With the latest collection of features and support, we feel there's a good chance any given Serverless Framework project will have faster packaging and deploys with serverless-jetpack.
Hook up the plugin today and see what the new multi-engined Jetpack can do for you! 🚀
