Skip to content

Fixed SQLite automation wait deadlines - #32151

Draft
JoaoMarcos44 wants to merge 2 commits into
TryGhost:mainfrom
JoaoMarcos44:fix/sqlite-automation-date-storage
Draft

JoaoMarcos44 wants to merge 2 commits into
TryGhost:mainfrom
JoaoMarcos44:fix/sqlite-automation-date-storage

Conversation

@JoaoMarcos44

@JoaoMarcos44 JoaoMarcos44 commented Oct 21, 2026 •

Copy link
Copy Markdown

fixes #32140

Summary

SQLite can select an automation wait before its deadline when new repository 2 writes bind Date objects as INTEGER epoch milliseconds and the existing poller compares ready_at with a TEXT datetime cutoff. SQLite sorts numeric storage classes before TEXT, so even a future integer timestamp can pass the due-time predicate. Mixed representations also make ready_at ordering incorrect.

This change keeps stored values and the poller's due-time cutoff in one comparable SQLite text representation, and normalizes existing integer dates. MySQL keeps its existing Date bindings.

Implementation and rationale

  • The date utility exports a UTC SQL datetime formatter with millisecond precision: yyyy-MM-dd HH:mm:ss.SSS. Repository 2 uses it for created_at and updated_at on automation runs and for created_at, updated_at, and ready_at on run steps when the connection is SQLite. For MySQL and MySQL2, it continues passing Date objects as before.
  • Migration 2026-10-21-21-22-05-convert-automation-dates-to-sqlite-text.js converts integer epoch-millisecond values on five columns: automation_runs.created_at and updated_at; automation_run_steps.created_at, updated_at, and ready_at. The SQLite-only guard leaves other database engines untouched. Each UPDATE is restricted to rows where typeof(column) is integer, so text rows are skipped on repeat execution. The migration is data-idempotent and is marked irreversible by the migration helper.
  • The legacy fetchAndLockSteps poller uses a separate readyAtCutoff for both ready_at <= predicates: the candidate selection and the guarded lock update. SQLite uses the millisecond formatter; MySQL uses the existing second formatter. Both phases therefore apply the same deadline boundary.
  • nowString remains second-precision and continues to populate locked_at, started_at, and updated_at. The stale-lock cutoff also remains second-precision. The existing lock-related writes use toDatabaseDate(), and this change has no evidence that those values are integer-backed; they are intentionally not migrated or reformatted.
  • findNextPendingReadyAt does not compare ready_at with the current time. It selects the earliest unlocked pending ready_at for scheduling and only compares locked_at with the stale-lock cutoff, so no new ready_at cutoff is needed there.

The legacy writer still stores ready_at as a whole-second UTC text value. That representation remains comparable to the SQLite millisecond cutoff: a same-second value without a fractional suffix sorts before the cutoff with its fractional suffix, consistent with the legacy writer's second-level precision. The new repository writer and migrated data preserve milliseconds.

Files changed

  • ghost/core/server/lib/db-types/date.ts
  • ghost/core/server/services/automations/database-automations-repository-2.ts
  • ghost/core/server/services/automations/database-automations-repository.ts
  • ghost/core/server/data/migrations/versions/6.73/2026-10-21-21-22-05-convert-automation-dates-to-sqlite-text.js
  • ghost/test/integration/services/automations/automation-run-date-storage.test.ts

No table schema or client/API contract changes are introduced.

Verification

The regression test uses in-memory SQLite configured through Ghost's real database helper. At a frozen time of 2026-10-21 12:00:00.500 UTC, it verifies that a due .499 step is returned and a future .501 step is excluded, alongside a legacy due step.

A RED run retained the formatter and migration support but used the original repository 2 writer and original legacy poller. The integration assertion failed: the returned IDs included future-step, proving the original writer-to-poller defect. Both supporting migration cases passed.

With the proposed repository changes, the focused GREEN run passed 104 tests across two files: three SQLite integration cases and 101 existing repository unit tests. The temporary sandbox configuration ran:

pnpm exec vitest run --config .c1-vitest.config.ts --maxWorkers=1 --no-file-parallelism

The integration cases verify conversion of old integer values with millisecond precision, idempotent rerun, the non-SQLite no-op guard, and repository 2 writer-to-legacy-poller deadline selection. The MySQL branch was represented by a mocked non-SQLite guard, not a live MySQL connection.

Scope and limits

This patch addresses a private experimental automations data-layer path on SQLite. It does not claim end-to-end behavior for a running Ghost service or browser behavior, and it does not establish that MySQL has this storage-class failure. The migration touches only the five integer-backed run and step date columns listed above; lock/start/finish timestamps remain outside its scope.

Infographic

Architecture, problem, solution and generation report.

Conceptual root-cause explanation

Preserved evidence and reproducibility

Focused commands, exit codes and captured output are pinned to a documentation commit. Historical investigation records are separately labeled; their source revisions and runtimes should not be confused with the current-base regression runs. The validation/ filenames above refer to this archive, which remains available after the local sandbox is deleted.

The validation report records the environment adjustments, shared RED commands, hook warning, combined checks and unresolved gates.

Final verification and remaining gates

pnpm check was attempted on the combined source/test worktree and exited 130. Formatting passed on 7,110 files and Oxlint passed. Nx then reported 30 successful tasks, 33 failures and four skipped tasks. Several lint scripts failed because Windows cmd did not interpret the quoted regex/Unix commands; Shade also failed with Rolldown INVALID_OPTION / missing options.input. That build failure remains unresolved and was not established as a clean-base failure. The chained full test stage was not reached. No full monorepo success is claimed.

A separate combined regression run passed 239 Core tests across eight files; the two combined navigation files passed three tests. These are additional focused runs, not a full suite and not distinct-test totals to add to the individual PR counts.

Commit hooks remained enabled. The nonblocking changeset reminder encountered ERR_PNPM_FROZEN_LOCKFILE_WITH_OUTDATED_LOCKFILE in its temporary checkout; that reminder is not counted as a passing versioning check. The successful GitHub labeling action is metadata automation. CodeRabbit skipped draft review in the captured historical snapshot; that historical status is not review approval. Product build/test CI remains unverified, and merge approval is not established. The user requested a ready-for-review transition, but GitHub denied the markPullRequestReadyForReview permission for account JoaoMarcos44 on this transition; the associated PR remains a draft due to this platform permission constraint and requires maintainer intervention. The concrete correction and focused results are available for review.

C1 additionally requires migration review. The checklist-posting job succeeded; its unchecked boxes do not establish staging performance, full up/down migrator lifecycle safety or reviewer sign-off. Only fixture-level repeatable up() conversion was exercised. The migration uses the irreversible helper. No staging/production database or live MySQL was used.

The temporary SQLite validation config can be copied into ghost/.c1-vitest.config.ts to reproduce the focused command above without the MySQL-backed project. The migration was initialized with the canonical repository generator using Git Bash for the Windows git describe quoting workaround. A migration lint failure on for iteration was corrected with five explicit awaited updates; hooks were kept enabled and the 104-test run passed again after that correction.

ref TryGhost#32140

The regression selected a future step with the original writer and poller. The generated migration support and formatter allow the fixture to cover existing data and capture the failing repository interaction.
ref TryGhost#32140

The real SQLite writer/poller deadline regression and 101 existing repository cases pass (104 tests).
@coderabbitai

coderabbitai Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the migration [pull request] Includes migration for review label Oct 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

It looks like this PR contains a migration 👀
Here's the checklist for reviewing migrations:

General requirements

  • ⚠️ Tested performance on staging database servers, as performance on local machines is not comparable to a production environment
  • Satisfies idempotency requirement (both up() and down())
  • Does not reference models
  • Filename is in the correct format (and correctly ordered)
  • Targets the next minor version
  • All code paths have appropriate log messages
  • Uses the correct utils
  • Contains a minimal changeset
  • Does not mix DDL/DML operations

Schema changes

  • Both schema change and related migration have been implemented
  • For index changes: has been performance tested for large tables
  • For new tables/columns: fields use the appropriate predefined field lengths
  • For new tables/columns: field names follow the appropriate conventions
  • Does not drop a non-alpha table outside of a major version

Data changes

  • Mass updates/inserts are batched appropriately
  • Does not loop over large tables/datasets
  • Defends against missing or invalid data
  • For settings updates: follows the appropriate guidelines

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

migration [pull request] Includes migration for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLite can select an automation wait before its deadline

1 participant