I run two small Apify actors that pull U.S. congressional stock trading disclosures (the STOCK Act "PTR" filings) for the House and the Senate and turn them into clean JSON. A handful of people pay for them. Revenue had plateaued, so I did what I should have done months ago: I stopped looking at the marketing and started auditing my own output.
It was worse than I thought.
The number that ruined my evening
I took the 50 most recent House filings and checked how many rows each one produced.
Almost half of them produced zero rows. Not errors. Not warnings anyone would see. Just nothing. The run finished green, the dataset looked healthy, and a big chunk of Congress's trades simply weren't there.
After fixing it, the same 50 filings went from 238 rows to 416.
If you're a paying user, that's the kind of thing you'd never notice. You'd just assume your data was complete. That's what bothered me most.
Why it happened
There wasn't one bug. There were several small ones that all failed the same way: quietly.
1. An amount format I never planned for. House filings report amounts as ranges like $1,001 - $15,000. My regex required a range. Then one filing showed up with an exact amount, $2,722.50, with cents. The regex didn't match, the parser returned an empty array, and the filing vanished.
2. Exchange transactions. Some rows are exchanges (swap one holding for another), not buys or sells. I didn't have a type for them, so they got dropped.
3. Scanned paper filings. About 14% of House filings in the last 90 days aren't digital at all. Someone printed the form, filled it in, and hand-delivered it. The PDF is just an image. My parser saw no text, returned nothing, moved on.
4. The Senate had the same problem, and I had a comment saying it didn't. There was literally a code comment in my Senate actor saying "Senate has no scanned case, source is HTML." Wrong. The Senate also accepts paper filings, they just live at a different URL. In the last 30 days: 3 out of 49.
5. Network failures. When a PDF download failed after retries, I incremented an error counter and... that's it. The filing was known, it was in the index, and it disappeared from the output.
The pattern is obvious in hindsight: every one of these paths ended in return []. And an empty array is the most polite way a program can lie to you.
The fix: every filing leaves a trace
The rule I landed on is simple. Every filing that appears in the official index must show up in the output. Either as real transaction rows, or as a clearly marked placeholder that tells you why there's no data.
So now every row has a parse_status:
| parse_status | What it means |
|---|---|
ok |
Parsed normally |
scanned_unparsed |
Paper filing, image only, no text layer |
parse_failed |
Text exists, but my parser couldn't extract rows (my bug, not yours) |
fetch_failed |
The source couldn't be downloaded after retries |
Placeholders keep the politician, filing date, and a link to the original document, so you can go look yourself. One catch: the platform bills every row in the dataset, placeholders included, so if you're counting or paying per result, filter on parse_status = "ok" for the rows that carry real trade data.
The run output now also includes counters (paper filings, parse failures, fetch failures), so a silent drop can't hide in a green run anymore.
Fixing things broke other things
A few smaller lessons from the same week:
My ticker fix broke Mastercard. Some rows had the ticker only inside the asset name, like Electronic Arts Inc. (EA). I added a fallback to extract it, with a stoplist to avoid false positives like (LLC) or (NY). I put all U.S. state codes on that stoplist. Which meant MA (Mastercard), MS (Morgan Stanley), DE (Deere), and MO (Altria) all quietly became null. Removed the state list, added tests for exactly those tickers.
Not every "bug" was a bug. I was sure duplicate content_hash values meant I was losing data. They didn't. The hash fingerprints the transaction content on purpose, so identical trades in the same filing share it, and the unique id handles the rest. It was already documented. I was the one who hadn't read it.
I tried OCR. I stopped.
Since 9 out of 10 scanned forms I checked were printed (not handwritten), OCR seemed doable. It mostly was.
The forms turned out to be checkbox grids: the amount isn't written as text, it's an X in one of eleven lettered columns. Full-page OCR turned that into garbage. So I built a hybrid: OCR for text cells, pixel density for checkboxes.
It got close. On one 6-page filing, date errors went from 29 of 111 rows down to 4. The remaining 4 were Tesseract reading a printed 2 as 7, consistently, on digits a human reads instantly. 06/02/2026 became 06/02/7026.
I could have "fixed" that with rules like "the year must be 2026, so 7 is probably 2." But that same guess on a month or day turns 02 into 07, and now you have a wrong trade date that looks perfectly real. So the policy is all or nothing: if any row in a filing fails validation, the whole filing stays a scanned_unparsed placeholder.
The OCR code is sitting behind a disabled flag. Maybe a different engine later. For now, an honest placeholder beats a confident wrong answer.
What I'd tell anyone building a data product
- Count what goes in, not just what comes out. Index size vs output size, per run.
-
return []in a parser is a smell. Make failure produce something visible. - Treat "the run succeeded" and "the data is complete" as two different claims.
- When in doubt, emit less data with a clear flag rather than more data you can't vouch for. The actors are here if you want to poke at them: House and Senate. If you find a filing that isn't accounted for, I genuinely want to know.
Top comments (8)
238 rows to 416 on the same 50 filings. That's a brutal number to find on your own.
We had the mirror image of this in a product data pipeline. Instead of returning nothing, a fallback returned something: when no evidence page was found, it filled in a URL from a targets file, and for 1 product that URL was a source we'd banned. The record looked complete. We added an assertion right before the file is written, and it fired on its very first run. That's how we found the fallback at all.
Your
parse_statuscolumn is the better version, and I'd take a placeholder row over a missing one every time, since it keeps the filing and says why. Do you watch the ratio week to week? A jump inparse_failedseems like the earliest warning you'd get that the House changed its PDF format.That fallback story is scarier than mine, honestly. A missing row at least leaves a hole you might eventually notice. A filled-in row with a banned source looks finished, so nobody goes looking. I like that the assertion fired on its very first run. That's the best argument for adding one I've heard.
To answer your question: not yet, and you're right that I should. Right now every run writes the counts (parse_failed, scanned_unparsed, fetch_failed, paper filings) to its output, but I only look at them when I go digging. So they're a snapshot, not a trend.
Your point about parse_failed as an early warning is spot on. Scanned filings run at a pretty steady ~14% on the House side, so they won't tell me much. But parse_failed should sit near zero. If it jumps, either the Clerk changed the PDF layout or I broke something, and I'd want to know within a day, not a month later when someone complains.
Next step is probably a simple threshold alert: if parse_failed or fetch_failed goes above a small percentage in a run, it pings me. Thanks for the nudge.
A steady ~14% for scanned filings is a great baseline to have, because it means a jump in that one is also news. If paper filings suddenly drop to 2%, the Clerk probably changed how scans are posted, not Congress.
On the alert: pick the number first. Before you see a bad run. Once a run has gone wrong it's tempting to set the threshold just above whatever that run did. Where do you think parse_failed should trip, 1 percent or 5?
Fair point on the baseline. If scanned filings suddenly dropped to 2%, that would tell me something changed on the Clerk's side, so I'll alert on that too, in both directions.
And I'll commit to a number now. For parse_failed it's one filing, not a percentage. The first bug I found in this audit was a single filing out of 133 over 90 days, about 0.75%. A 1% threshold would have missed it, and 5% wouldn't have been close. On real data the baseline is zero, so even one is worth looking at. It usually means a new format variant, and those are cheap to fix when you catch them early.
fetch_failed is different because network blips are normal. I'll trip that at 3 or more filings in a single run, or if it shows up in two runs in a row.
I'm writing both numbers down before the next run, so I can't move them after the fact.
One filing out of 133 is a great argument for "one" over any percentage. At 0.75% a 1% rule stays quiet forever, and that bug would still be there.
Writing the numbers down first is the whole trick. Good call.
The two-runs-in-a-row rule for fetch_failed is nice too, because a single network blip and a source that has actually gone away look identical inside one run and look nothing alike across two. Will the alert go to you directly, or to the people paying for the data as well?
Me, directly. I can't reach the people paying for the data. Apify doesn't give me contact details or per-user visibility, so a proactive "heads up" to users isn't possible.
What they do get is the signal inside the data. Every row carries parse_status, so a user's own pipeline can check for non-ok rows, and each run's summary has the counts. If you consume this data, that's the thing to alert on yourself. I'd rather say that plainly than promise a notification I can't send.
Your point about two runs is why I picked it. One blip and a dead source look identical within a run, and completely different across two.
That's the honest version. I'd trust a data source more for saying it.
A parse_status on every row means the buyer's own pipeline can do the alerting you can't do from Apify, and the per-run counts give them something to compare week over week without ever talking to you. Promising a notification with no way to send it would have been worse.
Thanks for the thread. I took more from it than I gave. Is the Senate actor getting the same placeholders?
Yes. Senate paper filings come through as placeholder rows now, same parse_status field. A 30-day run showed 3 paper filings out of 49, and before the change all three would have vanished silently. A Senate detail page that loads but yields no rows also gets a flagged row instead of disappearing.
One thing I should have said earlier: the platform bills every row in the dataset, placeholders included. So if you pay per result, filter on parse_status = "ok" for the rows that carry real trades.