Why our product data pipeline refuses Amazon as a source

I run a small site called Inch & Drawer. It answers one question: will this kitchen organizer actually fit in your drawer or cabinet? You type your inside measurements, and the site compares them against the published dimensions of every product it lists. Ten are live right now. Five more sit in draft.

That sentence hides the whole problem. "Published dimensions" is doing a lot of work. Where do the numbers come from, and how do I know they describe the exact product someone might buy?

Here is the rule set we ended up with, and the two bugs that shaped it.

Two sources, and Amazon is never one of them

Every product record needs two independent pages that agree on the model. The dimensions that get published on the site can only come from two kinds of pages: the manufacturer's own product page, or a retailer listing that prints the numbers itself.

Amazon pages are allowed in the record, but only as evidence for the model match. They never supply a published number. The site links to Amazon through the Associates program, and I did not want a page that earns a commission to also be the page that vouches for the measurement. Those are different jobs. Mixing them would have been the wrong call, and I'd rather list ten products with traceable numbers than fifty with numbers I copied from a marketplace.

Concretely, each product file carries a list of matched pages with a kind on each one: maker_site, retailer_listing, brand_storefront, amazon_listing. The importer on the site side only reads width, depth and height from the first two kinds. The other two can confirm "yes, this is the same model," and that is all they can do.

The fallback nobody looked at

The generator that builds these records had a fallback. If no evidence page was found for a product, it fell back to a URL from a targets file, so the record would at least point somewhere.

For one shelf riser, that fallback URL was an Amazon brand storefront. The rest of the pipeline treated it like any other source URL, and the record went out marked as importable. The site's catalog test caught it on the other side: it rejects any record whose source URL is on Amazon, drafts included. The importer had been stricter only for publishable records, so we tightened it to match the test.

The fix on the generator side was two lines of thinking. First, the fallback can no longer produce an Amazon URL. Second, there is now an assertion right before the file is written: if a primary source URL resolves to an Amazon domain, the write fails loudly. On its first run that assertion fired, which is how we found the fallback in the first place.

Midnight in the wrong time zone

Each record has a checked_at date. The site's freshness check reads that date as UTC midnight and refuses anything in the future. Our records are checked in Korea, nine hours ahead of UTC. Every morning between midnight and nine, a record checked "today" looked like it came from tomorrow, and the canary release failed on it.

We now write a full timestamp with the offset, checked_at_ts, next to the date. The importer converts that to a UTC date. If only the plain date exists, it takes the earlier of that date and today in UTC. Boring. It was also the only thing standing between the first automated product and the live site for about a day.

A sentence is not a number

One product page said the organizer needs a drawer that is "at least 3" high." The first import copied that whole sentence into the field meant for the printed value, so the card read like a quote inside a label. We changed it: the field holds the value as printed, 3", and the sentence lives in the evidence file that the card cites. Rules like "print the value, keep the sentence" are easy to write down and easy to forget. The importer now rejects any requirement value longer than 24 characters or without a digit, which is a crude way of saying "that's a sentence, not a number."

The pipeline reads pages logged out, at human speed, with a cap of 40 pages a day and a few seconds between requests. It is deliberately slow. Three products went live this week under these rules, and none of them needed a correction. The site publishes a handful of new products a week, and every one of them can be traced back to a page that printed the number.

If you want to see what the output looks like, the fit-check page is here: https://www.inchanddrawer.com/fit-check/. Each card shows where its dimensions came from, and marks the Amazon link as a paid link.

This post was drafted with AI help. The numbers and facts come from my own logs and repo.