Two providers changed the rules, and my app went back to the hangar
Showtime on the Balcony (Gone Wrong)
"I like planes too. I wish...
For further actions, you may consider blocking this person and/or reporting abuse
"The error message had been lying to me all along, and I'd written it myself" is the line I'd pin above my desk. Ours was the same lie one level down. SEC publishes a file of every listed ticker, and our nightly sweep treated "not in the file" as "delisted". Then the file quietly dropped QQQ, one of the busiest ETFs there is, and it went to a delisted page overnight while it was trading tens of millions of shares a day. Absence was never evidence, we'd just written it that way. Now a missing ticker only gets delisted if the market agrees too, no volume for a few sessions or an actual delisting form. Did the OpenSky block show up on a specific day, or had Railway been cut off from the start and the old token was hiding it?
Apparently, I also need to pin it above my desk... how on earth did I not make it more detailed?
Regarding OpenSky, I'm not really sure. I redeployed my the app and all of a sudden the planes were gone.
Ha, it's the kind of line you only get to write after rotating the credentials twice. If the planes vanished right on a redeploy, I'd check whether the new container came up with a different outbound IP. That would make the block look sudden even if OpenSky had it in place for weeks.
Yup, I though about that, cause this issue had occured before, but after 2-3 redeployments, it'd usually work. So I thought changing the hosting region would help, as the traffic would go out of a different IP address. But it still didn't do the trick.
Well, at least now I know the root cause. All I need to do right now is to find a suitable replacement for that provider.
That's a rough one to lose. A block by network range with no allowlist leaves nothing to fix on your side. I'd be curious which provider you end up with.
I'll share another post when I get my app running again. I need to do some deep research, I don't want to pick just any provider.
But I guess it's gonna have to wait for a while, cause I have plenty of other priorities ATM. But it's OK, I like the maintenance page too haha.
For real i think The misleading "Authentication failed" message is probably the biggest lesson here.
Two credential rotations happened because the error hid the real TCP timeout.
Good reminder that bad observability can waste more time than the actual bug.
1000% Proper error handling should never be neglected. Doesn't matter if it's a hobby project or an enterprise application.
the misleading error message part really hit home.
"authentication failed" sounds like a useful diagnosis, but if network timeouts, dns errors and actual credential problems all end up with the same message, it can send you in completely the wrong direction. rotating the credentials twice before discovering that the request couldn't even establish a connection is a painful example of that.
i also like the provider adapter approach. external services changing their rules is something you can't completely prevent, but keeping the provider-specific code isolated makes the eventual replacement much less painful.
That's exactly why proper error handling and logging are critical for any application.
And yes, a generic provider adapter makes life much easier, whether you need to swap one provider for another or diagnose issues.
Thanks!
That's a rough one to lose
It is, but it’s OK. I’ll get it back.