💾 Archived View for d.moonfire.us › blog › 2022 › 08 › 07 › semantic-release-and-woodpecker-ci captured on 2023-01-29 at 02:40:27. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Semantic Release and Woodpecker CI

Up a Level

With the recent drama of GitLab[1], both with the CI/CD changes and then more recent possible threat of deleting old repositories, I continue my migration to a local Gitea[2] instance, https://src.mfgames.com/ for the bulk of my code and writing.

1: /tags/gitlab/

2: /tags/gitea/

For the most part, migrating is just a matter of shuffling data. I have a lot of repositories, both active and inactive, and it will take me months to move them over. Plus I haven't decided if I'm going to purge them from my GitLab account so there is a single source of truth or just mirror back to them.

Currently, the most difficult task was figuring out how to handle the build processing. I've mentioned previously that I use Conventional Commits[3] and Semantic Release[4] fairly heavily. I've branched out a little from there using Lefthook[5] and my project layout[6].

3: /tags/conventional-commits/

4: /tags/semantic-release/

5: /tags/lefthook/

6: /garden/project-layout/

Currently, the CI does the following:

Build the project
Test various conditions including valid commit messages
If the commits indicate a new build:

Tag it
Build the release version
Create a release on Gitea

This changes over time, but it is the basic pattern.

Tags and Git Depth

Woodpecker does not automatically download the needed tags for `semantic-release` (and GitVersion[7]). This means that the `.woodpecker.yml` file needs to include tags.

7: /tags/gitversion/

clone:
    git:
        image: woodpeckerci/plugin-git
        settings:
            tags: true
pipeline:
    # The pipeline elements

Unike GitLab, which only limits to the last ten commits, it appears that Woodpecker downloads the full repository[8] by default which is also needed by GitVersion because it calculates every version. Not entirely sure about `semantic-release` logging indicates it doesn't need the full repository, just enough back to find a version.

8: https://woodpecker-ci.org/plugins/plugin-git

Building and Testing

To support task branches, I have a basic build and test code that runs on pushes and pull requests. This lets me identify bugs earlier and catch typos with my commits.

build:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/build.sh
    when:
        # We need both "tag" for the next section.
        event: [push, pull_request, tag]
        tag: v*

test:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - nix develop --command scripts/test.sh
    when:
        event: [push, pull_request]

From the build tasks, you can see that I'm using my current project layout which uses scripts in the `scripts/` folder instead of `npm run` or `dotnet run`. This is to make it easier to work with polyglot plus works around the issue that I need to use `nix develop` to get into my reproducible environment since the Docker image doesn't automatically do that. This is because both Gitlab and Woodpecker use the image which bypasses initialization files and I couldn't have it run `direnv allow` automatically to set up environment variables.

One thing that is missing is that Woodpecker doesn't have a clean mechanism for temporary build artifacts. I can't upload the build files and then download them so I can see the final results. Instead, I have to script it out or use a S3 plugin.

Building on Versions

With most cases, I build the release version of the project when the conventional commits indicate that there is a new version (`feat` and `fix`). This is an additional pipeline that comes after the `test:` line.

release-main:
    image: registry.gitlab.com/dmoonfire/nix-flake-docker:latest
    commands:
        - export DRONE="true" # Required to convince `env-ci`
        # semantic-release needs this locally
        - git branch $DRONE_BRANCH origin/$DRONE_BRANCH
        - nix develop --command scripts/release.sh
    secrets:
        - gitea_token
        - git_credentials
    when:
        event: push
        branch: main

There are a number of things in this block that took me a while. The first is the `event` and `branch`. We only do releases on the `main` (I'm still moving away from `master` as racist language).

The second is the `export DRONE` line. At the time I set this up, env-ci[9] wasn't aware of Woodpecker, but it was recently added[10] thanks to 6543 on the Woodpecker Matrix channel, `#woodpecker-ci:matrix.org`. I don't know when the latest `semantic-release` will have it, but it shouldn't be needed soon, if not already.

9: https://www.npmjs.com/package/env-ci

10: https://github.com/woodpecker-ci/woodpecker/pull/1035

The third is the `git branch` line in the above script. Woodpecker creates a detached head, as does Gitlab. But when it doesn't do is also create a local branch for the one being created. This causes a problem because the release process appears to “jump” to the branch to figure out the changes between the detached head (the commit being built) and the actual branch.

Finally, we have the secrets. `semantic-release` automatically picks up $GITEA_TOKEN for the release process but also needs $GIT_CREDENTIALS to verify Git access.

The token is easy, that is what given by Gitea for the user.

GIT_CREDENTIALS[11] is slightly harder, it is a colon-separate tuple of the user name and the Gitea access token. From observations, the Gitea is basically just a bunch of URL-safe characters, so the URL-escaping isn't needed in my case (you need to URL-escape the left and right of the colon but not the colon itself).

11: https://github.com/semantic-release/semantic-release/blob/master/docs/usage/ci-configuration.md

$ export GITEA_TOKEN=9fc6d72c72e4b149f07491a0b2d3ec9215d57caf
$ export GIT_CREDENTIALS="dmoonfire:$GITEA_TOKEN"

These need to be set on a per-project basis since Woodpecker, unlike GitLab and sourcehut[12], there doesn't appear to be a good way of having a shared set of secrets for projects (GitLab has organization/group level secrets, sourcehut has the secret storage). This means I have to set the same GITEA_TOKEN and GIT_CREDENTIALS for all 80+ of my Fedran[13] repositories†.

12: /tags/sourcehut/

13: /tags/fedran/

† Woodpecker has a CLI, `woodpecker-cli` which will let me automate that. I will use that.

Create Release on Tag

One thing I'm moving toward is creating a release entry on the forge. Since this happens after the build process and Woodpecker uses a different Docker image, I need it to be a post-release event so I hang it off the tagging process instead of the push to `main`.

6543 came to my rescue again with this one, so that is why there are those two “tag” elements in the script above. I also have a new stanza for the release process:

release-gitea:
    image: plugins/gitea-release
    settings:
        base_url: https://src.mfgames.com
        files:
            - "*.pdf"
            - "*.epub"
        api_key:
            from_secret: gitea_token
    when:
        event: tag
        tag: v*

If I didn't have the `event:` and `tag:` in the `build:` stanza, it wasn't working for the tags. This caused me some difficulties because I usually treat hashes as being unordered, but Woodpecker uses file order for processing pipelines. So, I needed to have the `build:` target build the file (with the correct version because it was tagged) and then `release-gitea:` to use that output for the release process. The `test:` and `release-main:` are skipped because they don't have those events listed.

In addition, secrets are handled differently when done as a parameter for a plugin. That is why I have the `from_secret:` element in the above script. This inconsistency threw me for a few days.

Putting it Together

If you want to see the final version, check out this example[14] which has my current version as a single file.

14: https://src.mfgames.com/dmoonfire-garden/project-layout/src/branch/main/.woodpecker.yml

Conclusion

I'm happy to move over to Woodpecker (you know, except for the cost of hosting) both because of the control and the challenge. I also don't have a need for speed, so if it takes a while to get through the queue, I'm okay.