💾 Archived View for dmerej.info › blog › 0021-an-npm-story.gmi captured on 2021-12-06 at 14:29:53. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-11-30)
-=-=-=-=-=-=-
2016, Sep 17 - Dimitri Merejkowsky License: CC By 4.0
Let's say you have a cross-platform `C++` application (Linux, Mac, Windows).
Convinced that using web technologies to implement the GUI is a good idea, you start writing some `javascript`, `css` and `html` code that would run in a `QtWebEngine` window.
It's a brand new project, so you decide you're going to use:
Obviously, this means you end up with quite a few `javascript` dependencies, so you write a `package.json` and start using npm[7] to fetch them.
2: https://facebook.github.io/react
You also read about why we should stop using grunt[8], so you patch the `package.json` to have something like
8: https://www.keithcirkel.co.uk/why-we-should-stop-using-grunt/
{
"dependencies": {
"webpack": "latest",
},
"scripts": {
"build": "webpack src/*.js"
}
}
And since you do care about build reproducibility, you use shrinkwrap[9] so that you don't accidentally pick up unwanted library upgrades.
9: https://docs.npmjs.com/cli/shrinkwrap
So you run `npm install shrinkwrap`, `npm shrinkwrap`, and you push the generated `npm-shrinkwrap.json` (that contains all the libraries names and version numbers) to the remote `git` server.
Of course, because you're a front-end developer, you earn enough money to afford a shiny Apple laptop to hack on your code.
But what happens when you try to use all this shiny new technology on Windows?
One member of your teams, working on Windows, then needs to build the static assets on his machine.
"Easy", you say, "just run `npm install`, then `npm run build` and you're all set!"
But, after he installs `node`, and runs `npm install` he tells you he got an error stating that `fsevent` is not available for hist platform.
`fsevent` is actually an optional dependency, but because you generated the `npm-shrinkwrap.json` file on your mac, `npm install` tries to install it.
This is a known npm bug[10], and it seems the only workaround is to somehow patch the generated `npm-shrinkwrap.json` file to remove the dependencies you do not care about.
10: https://github.com/npm/npm/issues/2679
"All right", you say, "I'm just going to write a script that will patch the generated file, and the team members working on Windows will just have to run it from time to time".
"It does not really matter, because all those Windows devs are back-end developers anyway ..."
You first patch the `package.json` to add a list of dependencies to ignore:
{
"scripts": {
"shrinkwrapIgnore": "node ./shrinkwrapIgnore.js"
},
"shrinkwrapIgnore": [
"fsevents"
],
"devDependencies" {
"jsonfile": "^2.4.0",
}
}
And then you write a `shrinkwrapIgnore.js` file looking like:
const _ = require('lodash');
const path = require('path');
const jsonfile = require('jsonfile');
const childProcess = require('child_process');
const shrinkwrapIgnore = require('./package.json').shrinkwrapIgnore;
const SHRINKWRAP_PATH = path.join(__dirname, 'npm-shrinkwrap.json');
try {
childProcess.execSync('npm shrinkwrap --dev', {
cwd: path.dirname(SHRINKWRAP_PATH)
});
} catch (error) {
console.error(error.stderr.toString());
process.exit(1);
}
const shrinkwrapContents = jsonfile.readFileSync(SHRINKWRAP_PATH);
shrinkwrapContents.dependencies = _.omit(shrinkwrapContents.dependencies, shrinkwrapIgnore);
jsonfile.writeFileSync(SHRINKWRAP_PATH, shrinkwrapContents, {
spaces: 2
});
Satisfied, you tell your team mate: "It's fixed!, just checkout the `hack-for-windows` branch and ..."
Yup, how is he going to run *any* `npm` stuff if `npm install` fails?
That's when a third team mate comes to the rescue. "Don't worry", he says, "I'll rewrite your script in Python".
And that's what he comes up with:
import sys
import collections
import json
with open("package.json", "r") as fp:
package_json = json.load(fp)
to_remove = package_json["shrinkwrapIgnore"]
with open("npm-shrinkwrap.json", "r") as fp:
shrinkwrap = json.load(fp, object_pairs_hook=collections.OrderedDict)
for dep in to_remove:
print(sys.argv[0] + ":" ,"removing", dep)
# do not fail if key is already removed:
shrinkwrap["dependencies"].pop(dep, None)
with open("npm-shrinkwrap.json", "w") as fp:
json.dump(shrinkwrap, fp, indent=2)
1/ Loading a value from a `.json` config file in Python is quite long:
with open("foo.json", "w") as fp:
data = json.load(fp)
value = data["value"]
In `javascript`, it's a one-liner:
const value = require("./foo.json").value
2/ We are using a small hack to make sure the order of the keys is preserved:
import collections import json shrinkwrap = json.load(fp, object_pairs_hook=collections.OrderedDict)
This makes it possible to have a meaningful `diff` after we've run the Python script. (Hopefully this will no longer be necessary starting with Python 3.6)
3/ Note how we use the `pop()` method with an `default` argument:
my_dict.pop(key, None)
This is shorter and more readable than:
if key in my_dict:
del my_dict[key]
That's all for today, see you later!
----