Skip to content

ci: prepare release drafts before tagging - #11569

Merged
Roasbeef merged 2 commits into
lightningnetwork:masterfrom
yyforyongyu:task-release-preflight
Nov 13, 2026
Merged

Roasbeef merged 2 commits into
lightningnetwork:masterfrom
yyforyongyu:task-release-preflight

Conversation

@yyforyongyu

@yyforyongyu yyforyongyu commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Prepare release artifacts before publishing signed tags so notes or build
failures do not consume release versions.

Change Description

  • Release CI runs automatically when a version-bump PR merges into a release
    branch and changes build/version.go.
  • If CI fails, merge the fixes, then manually trigger the release workflow on
    the updated release branch.

CI builds using a local unsigned tag and uploads assets to a draft tied to the
tested commit. Maintainers then sign and push that tag and publish the draft.

Add focused workflow tests and update the release procedures. Actionlint,
ShellCheck, local fixture tests, and signed-commit audits pass.

Build release-version commits before maintainers publish their signed
tags. Stage the successful build as a draft and cover preparation and
skipping behavior with focused local workflow tests.
@github-actions github-actions Bot added the ci-hold CI is held; remove this label to run CI label Nov 12, 2026
@yyforyongyu yyforyongyu removed the ci-hold CI is held; remove this label to run CI label Nov 12, 2026
@ziggie1984

Copy link
Copy Markdown
Collaborator

Thanks for working on this. I think this improves the release process, but it does not fix the race we actually encountered.

The failure mode was roughly:

  1. The version-bump PR passed CI against release-branch state A.
  2. Another PR merged into the release branch, advancing it to B.
  3. The version-bump PR then merged as C, without its previously successful CI proving the exact final release composition we intended.

This workflow prepares a draft from C, so the draft commit, branch head, and eventual tag can all agree on C. If C passes the release build, nothing here identifies that the version-bump PR was originally approved/tested against A, or that the additional change entering through B was not intended as part of the release. In other words, comparing the draft's target_commitish with the tag target would not catch this particular race; all of those values can match after the unexpected change is already included.

What this PR does protect against is a different and useful failure mode: publishing a permanent tag and only then discovering that release-note validation, packaging, or artifact upload fails. That is worthwhile, but I don't think we should describe it as fixing the intervening-merge/stale-base incident.

To cover that incident, I think the release cut needs to be serialized:

  1. Freeze the release branch (or otherwise stop new merges) before the final version bump is admitted.

  2. Merge the version bump through a merge queue with a single-PR group, or require strict up-to-date checks, so required CI runs against the latest release-branch base rather than reusing an older successful result. The CI workflow must handle merge_group if a merge queue is used.

  3. Prepare the draft from that resulting merge commit.

  4. Before creating/pushing the signed tag, enforce:

    successful final CI commit
        == successful draft preparation commit
        == current upstream release-branch HEAD
        == signed tag target
    
  5. Keep the branch frozen until the tag is pushed, then reopen it for backports.

The draft's explicit target_commitish: ${{ github.sha }} is useful for step 4, but that check is not currently performed by scripts/tag-release.sh, and by itself it only catches the branch advancing after draft preparation. The merge queue/strict-base check plus the short release freeze are what cover the earlier window where another PR lands after the bump's PR CI but before the bump merges.

Could we either add that enforcement as part of this change, or narrow the stated goal to "prepare and validate release artifacts before publishing the tag" and track the stale-base/release-freeze fix separately?

Wait for the draft preparation run before signing and pushing the exact
tested release commit. Route RC and patch releases through the same
procedure and explain cleanup before retrying stale draft preparation.
@yyforyongyu
yyforyongyu force-pushed the task-release-preflight branch from 9014786 to 9c415d2 Compare November 12, 2026 14:05
@yyforyongyu

Copy link
Copy Markdown
Member Author

Thanks for working on this. I think this improves the release process, but it does not fix the race we actually encountered.

The failure mode was roughly:

  1. The version-bump PR passed CI against release-branch state A.
  2. Another PR merged into the release branch, advancing it to B.
  3. The version-bump PR then merged as C, without its previously successful CI proving the exact final release composition we intended.

This workflow prepares a draft from C, so the draft commit, branch head, and eventual tag can all agree on C. If C passes the release build, nothing here identifies that the version-bump PR was originally approved/tested against A, or that the additional change entering through B was not intended as part of the release. In other words, comparing the draft's target_commitish with the tag target would not catch this particular race; all of those values can match after the unexpected change is already included.

What this PR does protect against is a different and useful failure mode: publishing a permanent tag and only then discovering that release-note validation, packaging, or artifact upload fails. That is worthwhile, but I don't think we should describe it as fixing the intervening-merge/stale-base incident.

To cover that incident, I think the release cut needs to be serialized:

  1. Freeze the release branch (or otherwise stop new merges) before the final version bump is admitted.
  2. Merge the version bump through a merge queue with a single-PR group, or require strict up-to-date checks, so required CI runs against the latest release-branch base rather than reusing an older successful result. The CI workflow must handle merge_group if a merge queue is used.
  3. Prepare the draft from that resulting merge commit.
  4. Before creating/pushing the signed tag, enforce:
    successful final CI commit
        == successful draft preparation commit
        == current upstream release-branch HEAD
        == signed tag target
    
  5. Keep the branch frozen until the tag is pushed, then reopen it for backports.

The draft's explicit target_commitish: ${{ github.sha }} is useful for step 4, but that check is not currently performed by scripts/tag-release.sh, and by itself it only catches the branch advancing after draft preparation. The merge queue/strict-base check plus the short release freeze are what cover the earlier window where another PR lands after the bump's PR CI but before the bump merges.

Could we either add that enforcement as part of this change, or narrow the stated goal to "prepare and validate release artifacts before publishing the tag" and track the stale-base/release-freeze fix separately?

we should fix one thing as a time, this PR aims at preventing wasting tags we push.

@litbot-9000 litbot-9000 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/gateway review

Verdict: LGTM — no issues found, approval warranted. Posted as a comment rather than an approval because the review requester on my side is not the workspace approver; the technical verdict is a clean pass.

What was verified (at head 9c415d2)

Trigger/mechanism review

  • New triggers are exactly push to v0.*-branch with paths: build/version.go plus workflow_dispatch; the old v* tag trigger is fully removed, so pushing the signed tag cannot rerun the build. fetch-depth: 0 keeps tags available, so the git show-ref --verify refs/tags/$tag skip actually works in CI.
  • lnd --version prints lnd version <ver> commit=<c> (config.go:933), so awk '{print $3}' extracts the version; field position is robust to the temp binary name under go run.
  • make release tag=$RELEASE_VERSION runs release.sh check-tag, which requires $tag == $(git describe --tags). Confirmed with make -n release tag=v0.21.9-beta and a local tag experiment: untagged HEAD describes as <nearest>-<n>-g<sha> (check would fail), a local tag at HEAD describes exactly as the tag (check passes). The local unsigned tag in the prepare step is what satisfies this.
  • check-release-notes-consistency.sh release reads the version from build/version.go and has no pushed-tag dependency, so it works on the untagged bump commit.
  • docs/ci-runs.md trigger description matches the workflow; no stale "tag push triggers release build" references remain; scripts/tag-release.sh does verify HEAD-in-sync-with-upstream as the docs claim.

Execution of the new guards (fixture replication of scripts/release-workflow_test.sh)
No Ruby in my environment, so I extracted the prepare block from the YAML verbatim (indent-aware awk, compared against the file) and ran the test's exact scenarios in fixture git repos:

  • new release → exports RELEASE_VERSION, creates only a local lightweight tag at HEAD, nothing pushed: pass
  • dev version 0.21.99-beta → skips, no tag, no export: pass
  • already-tagged v0.21.9-beta → skips, original tag object preserved: pass
  • extra: 0.21.0-beta.rc1 prepares v0.21.0-beta.rc1: pass
  • extra: 0.21.9-beta.99 is not caught by the .99 dev regex: pass

Mutation tests — both new guards are pinned:

  • dev regex neutralized (.99→.98) → dev version gets tagged; fixture catches it.
  • tag-existence check neutralized (if false && ...) → re-tag attempted, git tag fails "already exists"; fixture catches it.

Lint/syntax

  • actionlint 1.7.7: clean on release.yaml.
  • shellcheck 0.10.0: clean on scripts/release-workflow_test.sh and on the extracted prepare block; bash -n clean on all touched shell.
  • 80-col: three added lines exceed 80 (release-body URL, markdown table row, one 83-char shell line) — none in lint-covered Go code; CI lint agrees (green).

CI at head: all checks green (Lint, Static Checks, both unit suites, Check commits incl. signed-commit audit, cross compilation, release-notes checks).

What was not verified

  • An end-to-end run of the modified workflow (only triggers on release branches / dispatch); the draft-creation step itself has not executed here.
  • The exact draft-reuse semantics of the pinned lightninglabs/gh-actions/action-gh-release@c7149b6a (softprops fork) behind the retry doc's pagination claim — the guidance (stop runs, delete stale drafts before retry) is conservative and safe either way.
  • The Ruby version of the fixture test was not run as-is (no Ruby in my env); it was replicated step-for-step instead.

Required changes

None.

Non-blocking nit

One inline: the fixture test guards a real contract but is not wired into CI.

— claudell ⚡ (OpenClaw dev agent; this review posts as litbot-9000 and does not count toward branch protection)

@@ -0,0 +1,123 @@
#!/bin/bash

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this guards the workflow contract (triggers, draft settings, prepare-step semantics), but nothing in CI runs it, so a future edit to release.yaml could silently break the contract it pins. Worth wiring into an existing shell/static-check job (it only needs bash, git, ruby — no Docker) in a follow-up.

@ziggie1984 ziggie1984 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK. The build-before-tag flow makes sense: the version bump prepares artifacts in a commit-pinned draft first, while the real signed tag remains an explicit maintainer step before publication.

@ziggie1984
ziggie1984 added this pull request to the merge queue Nov 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 13, 2026
@Roasbeef
Roasbeef merged commit b21b727 into lightningnetwork:master Nov 13, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants