💾 Archived View for d.moonfire.us › blog › 2023 › 02 › 13 › package-management-dependencies captured on 2024-02-05 at 09:53:59. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-12-28)
-=-=-=-=-=-=-
While packages can be self-contained, usually they have dependencies with other packages or even frameworks to avoid duplicating code. Without dependencies, packages would be forced to contain common logic that other packages share, and that code may become stale or not updated as frequently introducing potential bugs.
This is going to be a series of posts, but I have no idea of how fast I'll be writing them out. I want to work out my ideas, maybe have a few conversations, and then start to move to more technical concepts.
2023-02-07 Package Management - Introduction
2023-02-08 Package Management - Versions
2023-02-12 Package Management - Identifiers
2023-02-13 Package Management - Dependencies
2023-09-20 Package Management - Identifiers 2
2023-11-30 Package Management - Formats and Registries
For most systems, dependencies are very simple: this package requires these packages in this range. When working with Bakfu, this is almost identical to the package identifier but without the `bakfu:version` and instead a range of values that are acceptable.
Using MfGames.Nitride?version=0.15.1[1] as an example, here is a fragment of what that could look like:
1: https://src.mfgames.com/mfgames-cil/-/packages/nuget/mfgames.nitride/0.15.1
{
// Basically this is "bakfu:nuget:MfGames.Nitride?framework=net6.0"
type: "nuget",
id: "MfGames.Nitride",
attributes: {
"framework": "net6.0",
},
content: {
version: "0.15.1",
dependencies: [
"bakfu:nuget:Autofac?framework=net6.0": "[6.4.0, 7.0.0)",
"bakfu:nuget:FluentValidation?framework=netstandard2.1": "[11.2.1, 12.0.0)",
"bakfu:nuget:MfGames.Gallium?framework=net6.0": "[0.4.0, 0.5.0)",
],
},
}
While the above works in most cases, there are situations when it doesn't. Probably the easiest one to explain in in Minetest with multiple mods that replace the built-in `beds` module: TenPlus1's[2] and SorceryKid's[3]. In all three of the cases, they are installed as `beds` which is used for the dependency, but they are distinguished enough for our package system despite the same name.
2: https://content.minetest.net/packages/TenPlus1/beds/
3: https://content.minetest.net/packages/sorcerykid/beds/
In the C# world, I've encountered this with the BouncyCastle libraries. In specific, we have:
Both of these libraries have a 1.8.9 version and both of them have exactly the same DLL once it is build, `BouncyCastle.dll`. In the past, this has caused us a lot of problem because while the two versions are API compatible (they have the same application binary interface, or ABI), they also overlap each other and have different packaging systems (one is only for .NET Framework, portable is more universally usable and currently maintained).
Likewise, if someone abandoned a package in a fit of anger, say the leftpad drama in the JavaScript[4], a strict package identifier means a replacement cannot be find. The same thing happens if a project maintainer dies without planning for a successor[5], there is a schism in the community causing a soft- or even hard-fork, or even someone using a Git repository or local copy to add some customization they need. Regardless, there are reasonable reasons for having two packages that proclaim to be the same version.
4: https://www.theverge.com/2016/3/24/11300840/how-an-irate-developer-briefly-broke-javascript
5: https://d.moonfire.us/garden/exit-planning/
To handle this, packages should list the aliases of what they provide (including themselves if needed) and that is what dependent packages would use. Using the BouncyCastle for an example:
// BouncyCastle
{
content: {
id: "BouncyCastle",
version: "1.8.9",
attributes: {
"framework": "net",
},
provides: [ "bakfu:nuget:BouncyCastle?bakfu:version=1.8.9&framework=net" ],
},
}
// Portable.BouncyCastle
{
content: {
id: "Portable.BouncyCastle",
version: "1.8.9",
attributes: {
"framework": "net40",
},
provides: [
"bakfu:nuget:Portable.BouncyCastle?bakfu:version=1.8.9&framework=net40",
"bakfu:nuget:BouncyCastle?bakfu:version=1.8.9&framework=net",
],
},
}
// A third package
{
content: {
id: "Something.Else",
version: "1.0.0",
requires: {
"bakfu:nuget:BouncyCastle?framework=net": "[1.8.9,2.0.0)",
},
// A missing provides would just produce the identity package
// in this case, it would imply:
// provides: { "bakfu.nuget:Something.Else?bakfu:version=1.0.0" }
},
}
In the above example, if a module doesn't have a `provides` element, then we could assume that it is just the identity version. “Something.Else” version 1.0.0 becomes `bakfu.nuget:Something.Else?version=1.0.0`.
The above example doesn't handle optional dependencies. For example, Minetest modules can check for the presence of other modules and then hook up to them. Atom (one of my favorite editors that was sadly killed) had the idea of services where a package could provide a service (that had a version) and other packages could “consume” them. Using a fragment of my own `spell-check`, it looked like this:
// from spell-check
"consumedServices": {
"spell-check": {
"versions": {
"^1.0.0": "consumeSpellCheckers"
}
}
},
// from spell-check-project
"providedServices": {
"spell-check": {
"versions": {
"1.0.0": "provideSpellCheck"
}
}
},
In the above case, the editor would call use the callback method for every package that “provided” a service which would return an opaque object, which would then be handed to `spell-check` consumed callback (`consumeSpellCheckers`) as an array. That allowed optional dependencies by publishing a service and then having other packages subscribe to them.
Even if the publisher (`spell-check` in the above example) would get every subscriber, then doesn't mean that the publisher then could call something back into every subscriber that would let you wire the two together to allow for a reverse- or even bi-directional flow of data.
Since this also has system-specific elements, we use the same `attributes` construct to give flexibility to extend outside of Bakfu.
{
content: {
id: "spell-check",
version: "0.77.1",
publish: {
"bakfu:atom-service:spell-check": {
accept: "[1.0.0,2.0.0)",
attributes: {
callback: "consumeSpellCheckers",
},
},
},
},
}
{
content: {
id: "spell-check-project",
version: "0.7.2",
subscribe: [
"bakfu:atom-service:spell-check": {
accept: "[1.0.0,2.0.0)",
attributes: {
callback: "provideSpellCheck",
},
},
],
},
}
How the application handles those services isn't in the domain of Bakfu, just the gathering and providing the information that links the two. It would also be based on the application if there can be multiple conflicting publishers (can two packages both publish the same service and version?) or not.
The last major topic to cover for this post is how to categorize dependencies. NPM and C# both have the concept of development and production dependencies. NPM is a little easier to work with:
// package.json
{
dependencies: {
"package-1": "1.2.3",
},
devDependencies: {
"package-2": "4.5.6",
},
}
Rust is more complicated because enabling a feature also changes the dependencies of the package. Many crates don't include serialization, but if you add the `serde` feature, it would include `serde` and any other requirements. In effect, this is an optional dependency but determined at build item.
It also changes how we organize dependencies above by adding an extra layer inside the `requires` element that provides an application-specific set of keys. In Rust, this may also include every feature as a separate element of dependencies.
(I could have started with this, but I wanted to build the blocks and work out the ideas on a single dependency before getting into the more complex situations.)
{
content: {
id: "Something.Else",
version: "1.0.0",
requires: {
production: {
"bakfu:nuget:BouncyCastle?framework=net": "[1.8.9,2.0.0)",
},
development: {
"bakfu:nuget:DoesNotValidateInput": "[1.0.0,2.0.0)",
},
},
},
}
Another reason to have this categorization is to allow us to identify what packages are vulnerable in an intelligent mannner[6] instead of just assuming any and all vulnerabilities are important. In other words, if you trust your developers not to use a DDoSing regular expression in their code, you don't have to patch that code.
6: https://overreacted.io/npm-audit-broken-by-design/
One of the areas I'm not sure about is now to describe dependences and provides. In the above examples, I'm using the URI format but my gut feeling is that it should be the more verbose form to allow for extensions.
The previous example would then look like this:
{
content: {
id: "Something.Else",
version: "1.0.0",
requires: {
production: [
{
type: "nuget",
id: "BouncyCastle",
accept: "[1.8.9,2.0.0)",
attributes: {
framework: "net",
},
},
],
development: [
{
type: "nuget",
id: "DoesNotValidateInput",
accept: "[1.0.0,2.0.0)",
},
},
},
},
}
I'm just not entirely sure about which one is the “best” but I'm also trying to work out ideas for a complete system, so I will probably use the URI unless the expanded makes sense. Ultimately, I really won't know until I've gotten through all the analysis and ideas before I can say one way or the other.
(Though, I do lean toward more strict schema which means that the expanded form will probably be the final one.)
I'm also not sure if the Bakfu format really needs to have `bakfu:` in front of identifiers. I mean, it is inside the “bakfu” file.
Categories:
Tags:
Below are various useful links within this site and to related sites (not all have been converted over to Gemini).
https://d.moonfire.us/blog/2023/02/13/package-management-dependencies/