Skip to content

chainntnfs: don't miss a confirmation when an earlier-hinted watch registers second - #30

Merged
paulscode merged 9 commits into
blake2bfrom
conf-hint-backport
Nov 13, 2026
Merged

paulscode merged 9 commits into
blake2bfrom
conf-hint-backport

Conversation

@paulscode

Copy link
Copy Markdown
Owner

On a node whose bitcoind has no transaction index, such as a pruned one, a channel the node funded can stay pending after its funding transaction confirms, and a close can wait forever for its confirmations. This is the stuck channel reported in paulscode/lightning-fork-startos#1, which 0.21.8's hint fix (lightningnetwork#11481) did not cure.

What happens

On startup the funder republishes the funding transaction. lnwallet.PublishTransaction then watches for its confirmation on TxOut[0] from the current tip. When the funding output is output 0, that is the same request as the funding manager's watch, which starts from the broadcast height. The rebroadcaster registers first, a couple of milliseconds earlier, and TxNotifier.RegisterConf gave the later, earlier-hinted watch no scan of its own. The empty scan then stored the tip as the cached hint, so every later start skipped the confirmation too. With a transaction index, the lookup by txid finds it regardless of heights, so only nodes without one are hit.

The change

  • Backport of chainntnfs: don't miss historical events for subscribers with earlier height hints lightningnetwork/lnd#11247 (open upstream, targeting 0.22): hints are stored with the origin of the scans behind them, and a later subscriber below a set's coverage gets a scan of the uncovered prefix. The conflicts with channeldb: scope confirmation hints by script lightningnetwork/lnd#11481's script-keyed hints are resolved in the first commit; its upstream tests and asynchronous state machine model come along, with this tree's btcd import paths.
  • Fork only: confirm hints stored without an origin, as every earlier release stored them, are ignored, so each confirmation watch scans once from its own hint after the update. Upstream trusts such a hint for whichever subscriber registers first, which on an affected node is a race between two goroutines. Spend hints keep upstream's legacy rule.
  • Fork only, from review: when the first subscriber's hint is too late to use the cached hint, a later prefix scan still skips the blocks the cached hint proves clean, instead of rescanning from the broadcast height on every restart while a transaction is unconfirmed. A disconnected block stops that use.
  • An itest that opens a channel with its funding at output 0, stops the funder before the confirmation and restarts it after, and a CI job running it, with the breach hint case, on bitcoind without txindex.

Testing

  • The new itest fails 5 of 5 runs on v0.21.8-beta-blake2b.24 and passes 8 of 8 with this change, on bitcoind without txindex (Knots 29.4.3, BLAKE2b from height 10).
  • chainntnfs and channeldb pass under -race; the state machine model passes 20,000 checks and now also asserts that no hint is ever stored without an origin. New unit tests for the legacy rule and the cached-hint prefix fail when either change is reverted.
  • fork.yml's package set passes locally.

Upgrade and downgrade

An older binary reads the first four bytes of a new hint, its height, so going back to .24 loses the fix and nothing else. On the first start after the update, confirmation watches (pending opens, closes, splices, resolvers) each rescan once from their own hint; on a pruned node, blocks past the prune window are fetched from peers.

Not addressed here

Narrower cases in the same code, present before this change and tracked upstream in lightningnetwork#11248 and lightningnetwork#11234: a stale scan result credited to a request set recreated after maturity, a script-only request whose tip match is reorged out while a prefix scan is in flight, failed historical scans that are never retried, and Neutrino progress hints written by a prefix scan. Each needs a reorg or a failed scan plus a restart at the wrong time, and none affects txid confirmations or outpoint spends in normal operation.

Roasbeef and others added 9 commits November 13, 2026 09:31
A cached height hint only records where the next historical scan should
begin, not where the scans that produced it started. So if a subscriber
with a hint later than the actual confirmation or spend is the one
watching a request, its scans never cover that block, yet the hint it
leaves behind is trusted by the next registration. After a restart, a
subscriber with an earlier, correct hint that registers first then
skips straight past its event.

In this commit, we store a HeightHint of both the height and its origin,
and a cached hint is only used by a registration whose own hint is at
or above that origin. Otherwise, the scan starts at the subscriber's
hint. Each set tracks the coverage its hints are persisted with, and
Neutrino tags its incremental spend progress with the scan's origin.
We also query the cache under the notifier's lock, so updateHints can't
change the hint between the read and the set being created.

In channeldb, the hint is encoded as its height followed by its origin.
A legacy four byte value decodes with an unknown origin, which we trust
only for the first subscriber, same as before, so no migration is
needed. The height comes first, so an older binary reading the value
still recovers it. The cache interfaces now take a map of hints so each
request keeps its own origin while updateHints still writes every hint
in a single transaction.
Duplicate confirmation and spend registrations for the same request
share a single historical scan. If a later subscriber supplied an
earlier height hint than the one the scan started from, it just waited
on that scan, so a confirmation or spend below the first subscriber's
hint was never found.

In this commit, we track how far down each set's coverage reaches, and
a later subscriber whose hint is below it gets a supplemental scan of
only the uncovered prefix. The set counts its pending scans, so an
empty result from one scan can't advance the shared hint while another
is still checking an earlier range. The persisted hint needs no change
when a prefix is dispatched: its origin is above the new subscriber's
hint, so a restart before the prefix completes scans it again.

Supplemental dispatches are flagged explicitly, and Neutrino only
extends a dispatch's end height to its own tip for an initial scan, so
a prefix scan keeps its fixed boundary. A prefix also never ends above
the notifier's tip: after a reorg, the blocks above it are connected
again and seen at tip, and a backend can't scan blocks it doesn't have
yet. A backend that fails a scan
never reports it, which leaves the set pending and its hint frozen,
failing safe until a restart scans the range again.
The existing model completes every historical scan inside its
registration and gives every subscriber the same height hint, so the
earlier hint bugs fixed in the previous commits weren't reachable from
it.

In this commit, we add a rapid state machine that queues historical
scans and completes, fails, or reports Neutrino progress for them in
any order relative to registrations, blocks, bounded reorgs, and
restarts that re-register subscribers in a random order. Subscribers
draw their own hints, which can be earlier than a prior subscriber's or
later than the event itself, and the model covers spends alongside
confirmations.

Instead of predicting each notification, the oracle checks invariants
after every action: no subscriber receives an event that isn't in the
chain, a subscriber whose hint precedes its event is notified once all
scans for it have reported, and the hint cache never lets such a
subscriber skip its event if it registered first after a restart.
Reverting the prefix scans for either kind, the pending scan
accounting, or hint origins makes the model fail within a few thousand
runs.
A release without hint origins stored the tip as a request's confirm hint
whenever a late subscriber's historical scan came up empty. The wallet's
rebroadcaster is such a subscriber: on startup it republishes a pending
channel's funding transaction and watches its first output from the tip,
which is the funding output when it is output 0. Upstream trusts a hint
without an origin for the first subscriber, so a node that ran such a
release can still skip its confirmation when the funding manager happens to
register first. Ignore those confirm hints and scan once from the
subscriber's own hint instead; the hint stored afterwards carries an origin.
Spend hints keep the legacy rule, since rescanning every channel's funding
output from its open height is expensive on a pruned node.
When the first subscriber's hint is too late to use the cached hint, for
example the wallet rebroadcaster registering a republished transaction at
the tip, the cached hint was dropped and a later subscriber's prefix scan
covered blocks it already proved clean. On a node without a transaction
index that rescanned every block since a pending funding transaction's
broadcast on each restart. Keep a hint with an origin on the request set and
start the prefix at its height where it covers the subscriber's hint, with no
dispatch when it covers the whole prefix. A disconnected block drops it, since
it may vouch for replaced blocks.

Also: the test model's restart rule ignores origin-less confirm hints as
RegisterConf does, the property model asserts no hint is ever committed
without an origin, and the legacy spend hint comment says where the cost
lies.
@paulscode
paulscode merged commit 1e41d72 into blake2b Nov 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants