Repository navigation
chainntnfs: don't miss a confirmation when an earlier-hinted watch registers second - #30
Merged
Merged
Conversation
A cached height hint only records where the next historical scan should begin, not where the scans that produced it started. So if a subscriber with a hint later than the actual confirmation or spend is the one watching a request, its scans never cover that block, yet the hint it leaves behind is trusted by the next registration. After a restart, a subscriber with an earlier, correct hint that registers first then skips straight past its event. In this commit, we store a HeightHint of both the height and its origin, and a cached hint is only used by a registration whose own hint is at or above that origin. Otherwise, the scan starts at the subscriber's hint. Each set tracks the coverage its hints are persisted with, and Neutrino tags its incremental spend progress with the scan's origin. We also query the cache under the notifier's lock, so updateHints can't change the hint between the read and the set being created. In channeldb, the hint is encoded as its height followed by its origin. A legacy four byte value decodes with an unknown origin, which we trust only for the first subscriber, same as before, so no migration is needed. The height comes first, so an older binary reading the value still recovers it. The cache interfaces now take a map of hints so each request keeps its own origin while updateHints still writes every hint in a single transaction.
Duplicate confirmation and spend registrations for the same request share a single historical scan. If a later subscriber supplied an earlier height hint than the one the scan started from, it just waited on that scan, so a confirmation or spend below the first subscriber's hint was never found. In this commit, we track how far down each set's coverage reaches, and a later subscriber whose hint is below it gets a supplemental scan of only the uncovered prefix. The set counts its pending scans, so an empty result from one scan can't advance the shared hint while another is still checking an earlier range. The persisted hint needs no change when a prefix is dispatched: its origin is above the new subscriber's hint, so a restart before the prefix completes scans it again. Supplemental dispatches are flagged explicitly, and Neutrino only extends a dispatch's end height to its own tip for an initial scan, so a prefix scan keeps its fixed boundary. A prefix also never ends above the notifier's tip: after a reorg, the blocks above it are connected again and seen at tip, and a backend can't scan blocks it doesn't have yet. A backend that fails a scan never reports it, which leaves the set pending and its hint frozen, failing safe until a restart scans the range again.
The existing model completes every historical scan inside its registration and gives every subscriber the same height hint, so the earlier hint bugs fixed in the previous commits weren't reachable from it. In this commit, we add a rapid state machine that queues historical scans and completes, fails, or reports Neutrino progress for them in any order relative to registrations, blocks, bounded reorgs, and restarts that re-register subscribers in a random order. Subscribers draw their own hints, which can be earlier than a prior subscriber's or later than the event itself, and the model covers spends alongside confirmations. Instead of predicting each notification, the oracle checks invariants after every action: no subscriber receives an event that isn't in the chain, a subscriber whose hint precedes its event is notified once all scans for it have reported, and the hint cache never lets such a subscriber skip its event if it registered first after a restart. Reverting the prefix scans for either kind, the pending scan accounting, or hint origins makes the model fail within a few thousand runs.
A release without hint origins stored the tip as a request's confirm hint whenever a late subscriber's historical scan came up empty. The wallet's rebroadcaster is such a subscriber: on startup it republishes a pending channel's funding transaction and watches its first output from the tip, which is the funding output when it is output 0. Upstream trusts a hint without an origin for the first subscriber, so a node that ran such a release can still skip its confirmation when the funding manager happens to register first. Ignore those confirm hints and scan once from the subscriber's own hint instead; the hint stored afterwards carries an origin. Spend hints keep the legacy rule, since rescanning every channel's funding output from its open height is expensive on a pruned node.
When the first subscriber's hint is too late to use the cached hint, for example the wallet rebroadcaster registering a republished transaction at the tip, the cached hint was dropped and a later subscriber's prefix scan covered blocks it already proved clean. On a node without a transaction index that rescanned every block since a pending funding transaction's broadcast on each restart. Keep a hint with an origin on the request set and start the prefix at its height where it covers the subscriber's hint, with no dispatch when it covers the whole prefix. A disconnected block drops it, since it may vouch for replaced blocks. Also: the test model's restart rule ignores origin-less confirm hints as RegisterConf does, the property model asserts no hint is ever committed without an origin, and the legacy spend hint comment says where the cost lies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a node whose bitcoind has no transaction index, such as a pruned one, a channel the node funded can stay pending after its funding transaction confirms, and a close can wait forever for its confirmations. This is the stuck channel reported in paulscode/lightning-fork-startos#1, which 0.21.8's hint fix (lightningnetwork#11481) did not cure.
What happens
On startup the funder republishes the funding transaction.
lnwallet.PublishTransactionthen watches for its confirmation onTxOut[0]from the current tip. When the funding output is output 0, that is the same request as the funding manager's watch, which starts from the broadcast height. The rebroadcaster registers first, a couple of milliseconds earlier, andTxNotifier.RegisterConfgave the later, earlier-hinted watch no scan of its own. The empty scan then stored the tip as the cached hint, so every later start skipped the confirmation too. With a transaction index, the lookup by txid finds it regardless of heights, so only nodes without one are hit.The change
Testing
chainntnfsandchanneldbpass under-race; the state machine model passes 20,000 checks and now also asserts that no hint is ever stored without an origin. New unit tests for the legacy rule and the cached-hint prefix fail when either change is reverted.Upgrade and downgrade
An older binary reads the first four bytes of a new hint, its height, so going back to .24 loses the fix and nothing else. On the first start after the update, confirmation watches (pending opens, closes, splices, resolvers) each rescan once from their own hint; on a pruned node, blocks past the prune window are fetched from peers.
Not addressed here
Narrower cases in the same code, present before this change and tracked upstream in lightningnetwork#11248 and lightningnetwork#11234: a stale scan result credited to a request set recreated after maturity, a script-only request whose tip match is reorged out while a prefix scan is in flight, failed historical scans that are never retried, and Neutrino progress hints written by a prefix scan. Each needs a reorg or a failed scan plus a restart at the wrong time, and none affects txid confirmations or outpoint spends in normal operation.