Sergey Shinder - DEV Community

Sergey Shinder

Oct 2

Our users were logged out for two weeks before Redis refused to write anything

#sergeyshinder #sre #redis #reliability

2 min read

Sergey Shinder

Oct 2

Our beta became the version every new customer installed

#sergeyshinder #releaseengineering #npm #javascript

2 min read

Sergey Shinder

Oct 2

The SMS provider told us when to come back and we heard now

#sergeyshinder #api #integration #http

2 min read

Sergey Shinder

Oct 2

Our error graph went blank every time we zoomed in on the outage

#sergeyshinder #observability #prometheus #grafana

1

2 min read

Sergey Shinder

Oct 2

A reporting job outranked our ingress controllers and the scheduler agreed

#sergeyshinder #kubernetes #scheduling #containers

2 min read

Sergey Shinder

Oct 2

Our bucket policy let the office in and locked out every server we own

#sergeyshinder #terraform #aws #iam

2 min read

Sergey Shinder

Oct 2

Our invoices lost their accents when we moved to a new CI system

#sergeyshinder #cicd #git #docker

1

2 min read

Sergey Shinder

Oct 1

A connector we switched off in May kept every change our database made

#sergeyshinder #sre #postgres #reliability

2 min read

Sergey Shinder

Oct 1

A migration merged on Tuesday was never going to run in production

#sergeyshinder #releaseengineering #database #migrations

2 min read

Sergey Shinder

Oct 1

Our HR sync decided José had left and then hired him again

#sergeyshinder #api #integration #unicode

2 min read

Sergey Shinder

Oct 1

Our synthetic check spent five weeks testing the login page

#sergeyshinder #observability #monitoring #synthetics

2 min read

Sergey Shinder

Oct 1

Our node pool rotation hit Docker Hub's limit and new nodes could not start their pods

#sergeyshinder #kubernetes #docker #containers

1

2 min read

Sergey Shinder

Oct 1

A refactor that touched no database changed our database password

#sergeyshinder #terraform #iac #postgres

2 min read

Sergey Shinder

Oct 1

We made checkout shallow and every build became version 0.1.0

#sergeyshinder #cicd #git #versioning

2 min read

Sergey Shinder

Sep 30

We added a fourth server and made our cluster easier to break

#sergeyshinder #sre #reliability #distributedsystems

2 min read

Sergey Shinder

Sep 30

Our new region launched with thirty one flags we had switched off everywhere else

#sergeyshinder #releaseengineering #featureflags #automation

2 min read

Sergey Shinder

Sep 30

Our customs broker changed an XML prefix and every shipment stayed on the dock

#sergeyshinder #api #xml #integration

2 min read

Sergey Shinder

Sep 30

Thirty four pods each failed a little and our alert judged them one at a time

#sergeyshinder #observability #monitoring #alerting

2 min read

Sergey Shinder

Sep 30

The node was fifty eight percent full and could not create a single file

#sergeyshinder #kubernetes #docker #linux

2 min read

Sergey Shinder

Sep 30

Every apply during the sale shrank our web tier back to six instances

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 30

Pip found a newer version of our internal package on the public index

#sergeyshinder #cicd #python #security

2 min read

Sergey Shinder

Sep 29

Every cache entry we warmed at deploy expired in the same minute

#sergeyshinder #sre #reliability #caching

2

1

2 min read

Sergey Shinder

Sep 29

Our deploy tool decided that 2.9.0 was newer than 2.10.0

#sergeyshinder #releaseengineering #versioning #automation

1

2 min read

Sergey Shinder

Sep 29

Every order paid in yen looked a hundred times underpaid

#sergeyshinder #api #payments #integration

1

2 min read

Sergey Shinder

Sep 29

Which service lost its logs depended on who wrote first after midnight

#sergeyshinder #observability #logging #elasticsearch

1

2 min read

Sergey Shinder

Sep 29

Every pod on the node was ready and none of them could open a connection

#sergeyshinder #kubernetes #linux #networking

1

2 min read

Sergey Shinder

Sep 29

Our disaster recovery copies were in the same region as the originals

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 29

A hung test held the only runner allowed to deploy our hotfix

#sergeyshinder #cicd #githubactions #testing

1

2 min read

Sergey Shinder

Sep 28

The partner recovered in four minutes and our circuit breaker kept them out for three hours

#sergeyshinder #sre #reliability #resilience

2 min read

Sergey Shinder

Sep 28

We switched traffic to green and blue kept doing the work

#sergeyshinder #releaseengineering #deployment #devops

2 min read

Sergey Shinder

Sep 28

We moved our webhook and the partner's POSTs arrived as empty GETs

#sergeyshinder #api #webhooks #integration

1

2 min read

Sergey Shinder

Sep 28

Four in ten of our requests were our own health checks

#sergeyshinder #observability #monitoring #sre

2

2 min read

Sergey Shinder

Sep 28

Our service became PID 1 and never collected its dead children

#sergeyshinder #docker #kubernetes #linux

1

2 min read

Sergey Shinder

Sep 28

The plan said update in place and our database failed over at half past ten

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 28

Our migration test passed because the last job left its database behind

#sergeyshinder #cicd #docker #testing

3

2 min read

Sergey Shinder

Sep 26

Our database was fast for six hours and then it ran out of credit

#sergeyshinder #sre #reliability #database

2 min read

Sergey Shinder

Sep 26

The app release was the one deploy we could not take back

#sergeyshinder #releaseengineering #mobile #automation

2 min read

Sergey Shinder

Sep 26

The carrier kept the first thirty five characters of every address

#sergeyshinder #api #integration #logistics

1

2 min read

Sergey Shinder

Sep 26

Our p99 could never go above two and a half seconds

#sergeyshinder #observability #monitoring #prometheus

2 min read

Sergey Shinder

Sep 26

Our cluster upgrade waited two days for one pod that could not leave

#sergeyshinder #kubernetes #containers #devops

2 min read

Sergey Shinder

Sep 26

A cleanup pull request scheduled the key our backups needed

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 26

A pull request from a stranger ran with our publishing token

#sergeyshinder #cicd #security #githubactions

2 min read

Sergey Shinder

Sep 25

One message we could not parse stopped four hours of dispatches

#sergeyshinder #sre #reliability #kafka

2 min read

Sergey Shinder

Sep 25

We pressed the button again and the release came out half published

#sergeyshinder #releaseengineering #automation #devops

2 min read

Sergey Shinder

Sep 25

Our integration was a select statement against somebody else's tables

#sergeyshinder #api #integration #database

2 min read

Sergey Shinder

Sep 25

Our service got forty percent more expensive and nothing could say where

#sergeyshinder #observability #performance #profiling

2 min read

Sergey Shinder

Sep 25

The volume we mounted hid the files our image had shipped

#sergeyshinder #kubernetes #containers #helm

2 min read

Sergey Shinder

Sep 25

Our code was correct and the account said no

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 25

Twelve test containers and one of them ran for twenty six minutes

#sergeyshinder #cicd #testing #devops

2 min read

Sergey Shinder

Sep 24

The incident ended and the spare never came back

#sergeyshinder #sre #reliability #incident

2 min read

Sergey Shinder

Sep 24

The deploy took four minutes and the release took six days

#sergeyshinder #releaseengineering #database #automation

2 min read

Sergey Shinder

Sep 24

A third of our calls came from an address our carrier had never been given

#sergeyshinder #api #integration #networking

2 min read

Sergey Shinder

Sep 24

We spent thirteen weeks about to buy a bigger database

#sergeyshinder #observability #tracing #performance

2 min read

Sergey Shinder

Sep 24

The node was ready in seventy seconds and our pod was not

#sergeyshinder #kubernetes #docker #autoscaling

2 min read

Sergey Shinder

Sep 24

We forked a module for one line and owned all of it

#sergeyshinder #terraform #iac #devops

2 min read

Sergey Shinder

Sep 24

Thirty seven tests failed the day our runner stopped running them in order

#sergeyshinder #cicd #testing #devops

2 min read

Sergey Shinder

Sep 23

The order was committed and nothing else ever heard about it

#sergeyshinder #sre #reliability #messaging

2 min read

Sergey Shinder

Sep 23

The fix we backported to our oldest branch did nothing

#sergeyshinder #releaseengineering #versioning #automation

2 min read

Sergey Shinder

Sep 23

Twenty workers refreshed one token and invalidated each other

#sergeyshinder #api #oauth #integration

1

2

2 min read

Sergey Shinder

Sep 23

The requests our customers gave up on were in none of our numbers

#sergeyshinder #observability #monitoring #latency

2 min read

Writing Debut