💾 Archived View for capsule.adrianhesketh.com › 2021 › 05 › 11 › trying-out-npm-and-yarn-workspaces captured on 2022-06-11 at 20:56:34. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2022-01-08)
-=-=-=-=-=-=-
I'm doing most of my work with CDK at the moment, to deploy Serverless projects. These projects often consist of multiple projects in the same repo - i.e. a monorepo. There's the CDK project itself, containing the definition of the infrastructure, and one or more projects containing application code that will be deployed as AWS Lambda functions.
For example, a project I'm working on has a cdk directory and a web directory.
Within each directory is a `package.json` that defines the libraries that are used by the project.
./cdk/package.json ./cdk/index.ts ./web/package.json ./web/index.ts
In the case of a TypeScript or JavaScript CDK project, the dependenices are things like `@aws-cdk/aws-lambda-nodejs` that are used as part of the deployment process, but aren't included in any build outputs.
In the case of a TypeScript or JavaScript project being used to create AWS Lambda functions, libraries that are referenced within the `package.json` need to be bundled into the zip file or Docker container used to create the Lambda function, along with custom code.
In both cases, this means that a CI pipeline will need to install packages before a CDK deployment or packaging can occur.
This can be done by simply running `npm install` or `yarn install` in both directories one after the other.
name: GitHub Actions Demo
on: [push]
jobs:
npm-install:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v2
- uses: actions/setup-node@v2
with:
node-version: '14'
- name: web - Install npm modules
run: cd ./web && npm install
- name: cdk - Install npm modules
run: cd ./cdk && npm install
Since it takes time to run the installation, it's tempting to try and run them in parallel using background tasks in bash, but this can cause problems, and take extra time, as there are two processes racing to download (often) the same things to the same npm or yarn cache locations.
(cd ./web && npm install) & (cd ./cdk && npm install) & wait
Yarn introduced the concept of workspaces [0] a few years back to solve the problem, and `npm` introduced support for them in version 7 [1] which was released in Feb 2021, so I thought this would be an easy way to speed up the installation.
You place a `package.json` file in the root of the repo and npm or yarn uses this to work out which packages to run.
And you end up with a repo structured like this.
./package.json <-- new! ./cdk/package.json ./cdk/index.ts ./web/package.json ./web/index.ts
The `package.json` in the root has a new "workspaces" section which tells yarn or npm where the packages are.
{
"name": "workspace",
"private": true,
"workspaces": {
"packages": [
"web",
"api"
]
}
}
With this in place, you only need to run `npm install` or `yarn install` once, in the root of the directory. It sounds straightforward, but I ran into a few issues.
In Github Actions, Node 14 still uses npm 6 [2] which doesn't support workspaces. npm 6 doesn't throw an obvious error, so it can look like it's working from the root directory, but it doesn't actually install anything.
When you run `yarn install` in the root directory, all of the required packages only get added to the `node_modules` folder in the root directory, so you don't get `web/node_modules` and `api/node_modules` directories.
If you `cd` into the `web` directory and run your project, it will work fine because node will look in the parent directory and find the node module there, but if you move the `web` directory into another directory tree, then it will fail due to missing dependencie.
This behaviour isn't often what you want in a CI pipeline, because you might need to mount a child directory within your mono repo into a Docker image with the node_modules present, or bundle the directory up in some way.
Fortunately, the yarn team added the `nohoist` feature [3] which creates node_modules in all of the subdirectories.
It requires a minor update to the `package.json`.
{
"name": "workspace",
"private": true,
"workspaces": {
"packages": [
"web",
"api"
],
"nohoist": [
"**"
]
}
}
However, it's not supported in npm at the moment [4], which cost me a bit of time trying to work out why it didn't work.
The npm alternative is more tooling. lerna [5] can execute the multiple npm installation commands for you.
Instead of adding a `workspaces` section to your `package.json`, you add a `lerna.json` file to the root which accomplishes the same goal.
{
"packages": [
"web",
"api"
],
"version": "0.1.0"
}
Then, executing `npx lerna bootstrap` will populate the `node_modules` directories in the child paths.
But it's another tool that people need to know about, versus a simple shell command, so for npm projects, I'll stick with the shell command for now, until npm workspaces are a little more developed.
(cd web && npm install) && (cd api && npm install)
If you're using yarn, workspaces seem to be a quick win, but check if your CI pipeline needs that "hoisting" feature.
Using AWS X-Ray with a TypeScript Lambda