The encrypted secret was bytes. It round-tripped perfectly in the in-memory store. The unit tests passed. In Postgres, the column was JSON, and byt...
For further actions, you may consider blocking this person and/or reporting abuse
D-5 is why running SQLite or in-memory dicts for test harnesses always bites you later. Python dictionaries happily store raw bytes, custom classes, or unpickled objects without complaint. You only discover Postgres jsonb or Pydantic validation rejects the wire format once psycopg actually tries to serialize the query.
D-8 also hits anyone doing audit tables or event logs. If you query the history table instead of folding the events or checking an active state flag, the mere existence of a disable event gets treated as the current truth. I had a runner report workers as stopped for three days because the healthcheck was reading the historical incident log instead of the live lease table.
Yes — that's exactly the D-5 seam. The model stays
bytes; the boundary base64-encodes on the way to Postgres and decodes on the way back, so the in-memory test still passes and still proves nothing about the column. And your D-8 example (healthcheck reading the historical incident log instead of the live lease table) is the same shape — the fix here was to report the derived state, not the record alone. Great point on freshness vs existence being two different controls.This is a great reminder that passing unit tests doesn’t always mean a system is reliable. For example, an app might successfully save a date in memory but fail when PostgreSQL expects a different format or timezone handling. The logic works, but the boundary breaks. I also like the point about turning database constraint violations into meaningful API responses—good error handling is part of good system design, not an afterthought.
What’s your go-to strategy for catching these issues before production: integration tests, contract tests, or testing against the real infrastructure?
My go-to is the layer that crosses the boundary: Postgres-gated tests (the ones CI runs with a real database) plus the Docker field test that runs the actual container stack — 67/67 container tests, 36/36 scenarios. Contract tests help for interfaces, but these defects were serialization mismatches (bytes→JSON, id→uniqueness), and those only show when the value actually hits the real store. So: test against the real infrastructure at the seam, and keep unit tests for the logic on either side.
The fix for #573 makes the manifest honest about the build, and the restore still compares it with the build rather than with the database.
SCHEMA_HEADis now the head of the migrations folder shipped in the running build, read withScriptDirectory.get_current_head();create()stamps it into the manifest, andrestore()checks the manifest against the same value. The restore endpoint passes noexpected_head, so nothing on the backup path readsalembic_versionfrom the database the rows go into. The issue asked for that half too: its reproduction has the guard accepting the archive "regardless of real DB revision", and it says "the restore guard should compare against the actual target schema head."On a single build the two cannot drift, because startup upgrades to head, and an older build will not even start against a newer database: Alembic stops with a CommandError because the version row names a revision its scripts do not contain, which I checked on a scratch SQLite with the version row set one revision ahead. They drift during an upgrade. The chart sets no update strategy, so the Deployment rolls: the new pod migrates while it starts, and the old pod keeps serving until the new one passes /readyz. A restore that lands on the old pod in that window matches the old build's head and writes into a database one revision ahead, which is the 0034 against 0035 case from the post. Reading
version_numinsiderestore(), the same query readiness.py already runs, and refusing when it differs from the manifest would close that window, and a test that sets the version row one past the build and expectsBackupSchemaMismatchErrorwould keep it closed.You've read it exactly right.
SCHEMA_HEADis the head of the migrations folder shipped in the build (ScriptDirectory.get_current_head()),create()stamps it into the manifest, andrestore()refuses whenmanifest.alembic_head != (expected_head or SCHEMA_HEAD)→BackupSchemaMismatchError. So the guard is build-vs-manifest, and the restore path never readsalembic_versionfrom the target DB. Your rolling-upgrade window is the real gap: the old pod can restore into a DB one revision ahead. Readingversion_numinsiderestore()(the same queryreadiness.pyruns) and refusing on a mismatch would close it — noting that for the v0.2.0 write-up. Thank you for the precise repro.D-7 is the one I'd sit with, because we had the same failure with an API in place of a primary key, and it took a reader's correction before I could see it.
We had a checker comparing a hand-maintained list of our published post IDs against the platform's own list of published posts. Two sources, one comparison, green. What we had not noticed is that the second list was fetched through the same endpoint the first one had been built from, and that endpoint silently caps its response at 15 items. A truncation hit both sides. The check passed and we filed it as verification. A reader named the shape of it and the wording stuck with me: an independently enumerated list is only independent if it does not page through the same capped API.
That is your bytes-through-JSON problem, moved out one layer. If both stores are the same store, the value round-trips fine in both of them. A value can cross a boundary you drew yourself while never crossing one that exists.
The test I took from it, and the reason I think it travels past your table: if the dependency both sources route through fails, do they fail in the same direction? Two readers of one truncated list are not two readers, they are one reader twice. Your in-memory store and your Postgres store share an interface but not a serialization, which is exactly why D-5 was visible in the field test and invisible in the unit test. Two comparators can share a failure mode the same way, and it hides in the same place.
D-8 deserves its own name, I think. The record existed and the listing was wrong. That is not a logic bug. It's a freshness bug, and it never throws. We shipped a post about our version of it. A table with 19,518 rows, every tool call in it correct, zero errors, and every result two days stale, because the writer had moved to a different table and the reader hadn't. The query returned rows. The rows looked current. We now keep two separate controls: an existence control, which is a row we know is there, and a freshness control, which is a row we know is new. The first catches the store that stopped writing. The second catches the store that kept writing somewhere else. Only the second one would have caught ours.
This is the sharpest framing of D-7 I've seen. The bug was id-vs-uniqueness — one source worked, a second source applying the same desired state hit a PK collision. The fix is a composite key
(source_id, workload_id)with an idempotent upsert. Your "two readers of one truncated list are one reader twice" is the same lesson as the in-memory/Postgres seam: they share an interface but not a serialization, which is exactly why D-5 was visible in the field test and invisible in the unit test. The existence-control + freshness-control pair is a keeper — I'm stealing it.Composite key is the right fix for the collision. The part I'd watch is what it does to the signal.
Before, the second source announced itself with a 500. Ugly, but unmissable. With (source_id, workload_id) and an idempotent upsert, that same attempt resolves into a quiet merge. That's the correct behaviour for a retry and the wrong behaviour for a different source. If the two ever disagree about the value rather than the row, the upsert keeps one and the loser leaves nothing behind. The crash and the evidence that two sources were contending for the same row go away together.
We hit the same trade from the other direction. A sweep of ours reported success while sampling 25 posts out of 26. One id had been dropped from a hand-maintained list, and a post that's missing produces exactly the same line as a post that hasn't changed. Nothing could flag it, because the output was identical either way. What fixed it is recording how many posts the sweep expected to see, and refusing to report a verdict below that count.
So with the composite key in, the control I'd add is a counter of how often the second source actually lands on an existing row. If that counter stays at zero, either you genuinely have one source or the second one stopped reaching you, and from the write path those two look identical. Your freshness control, applied to conflicts instead of rows.
Good luck with v0.2.0. The rolling-upgrade window in your other reply is the one I'd keep open.
The restore check is the seam I’d test next: restore into an empty database, read its migration head there, then run the target app revision against it. Comparing only with the current live head could reject an intentionally older rollback without proving the backup is usable. Does the field test include an older-but-supported backup alongside an incompatible one?
The field test covers restore plus an incompatible-head case, but not the "older-but-supported" backup you're describing — that's a better usability test than comparing live heads, because it proves the backup is usable rather than merely consistent. Adding it. And agreed that comparing only against the current live head could reject a legitimate rollback; that's the argument for treating the build head as the reference and testing the restored DB by running the app revision against it.