Every AI rollout comes with a dashboard. PRs merged per week, time from ticket to prototype, model accuracy on the eval set. All of those numbers g...
For further actions, you may consider blocking this person and/or reporting abuse
I understand the wish to find a number for a developer's work, and I agree velocity is the wrong one. But the problem is older than AI. Every developer KPI I've seen broke the same way:
So AI didn't create the measurement problem. It made the wrong numbers grow faster.
I like that your three tests are questions, not numbers. Test 3 is actually a rephrasing of a principle every professional developer knows: dependency inversion from Clean Architecture, where business logic doesn't depend on frameworks, databases or vendors. To me that's the best part of your argument. The principles that made code last before AI are the same ones that make it last now. AI just makes it easier to skip them.
To answer your question, the signal I trust most is how long it takes a new person to make a safe change, and whether someone on the team can still explain why the code is the way it is.
That last point is probably the best answer to the question. How long does it take a new person to make a safe change, and can someone explain why the code works this way? I agree the measurement problem existed long before AI. AI just makes it much easier to produce impressive numbers while hiding the actual cost of maintaining what we build.
I think we should put more effort into automating testing and review, because that's where the hidden cost gets caught early. You said AI makes it easy to produce impressive numbers while hiding the maintenance cost. A test or a review check that runs on every change can't be impressed by a demo. I've started asking the agent to write more types of tests, not just unit ones. Golden screen tests, and e2e tests for the MCP server (when tested, it is connected as a client). Maybe that helps in your field too.
The "year two" framing is the right test. Almost any codebase looks healthy in the first month, and AI assistance mostly compresses that first month further. The bill arrives later, when someone has to change a feature nobody fully understood when it was generated.
In client builds I have found it helps to treat maintainability as an explicit acceptance criterion rather than a hope. If the definition of done for a piece of work includes tests for the critical path, a short note on why the main decisions were made, and a deploy that someone other than the author can run, the speed gains from AI stop being borrowed against the future.
The other thing that helps is keeping each unit of work small and bounded. A single, well defined problem with a clear outcome is much easier to review and hand over than a sprawling feature that grew one prompt at a time. Curious whether you track any metric for this, such as time to first change by a new contributor.
I like the idea of making maintainability part of the definition of done instead of something we hope for later. Small, bounded changes also make a huge difference when AI makes it so easy to generate more than anyone can realistically review. I don't track time to first change by a new contributor yet, but honestly, that's a metric worth trying.
Test 3 is the one I'd push on, because the interface fixes the compile-time half of a swap and not the half that bites. Summarizer.summarize() keeps the same signature across vendors, but output length, tone and what gets dropped all change, and nothing in the type system notices. So the signal I'd trust more than velocity is whether a dependency swap comes with something that can fail: a dozen saved inputs with properties the output must keep, run against both providers before the switch. Without that, a clean boundary mostly makes it easier to swap in a regression.
Fair point. An interface makes the swap easier mechanically, but it doesn't guarantee equivalent behavior. I like the idea of keeping representative inputs and checking what must be preserved across providers. I'd add that to the boundary itself, because otherwise we've only solved half the problem.
Đúng trọng điểm. Dashboard đẹp mắt (PRs/week, cycle time giảm, prototype nhanh) chỉ đo lường velocity, không đo sustainability. Year two mới bộc lộ chi phí thực: technical debt tích lũy từ code gen không review kỹ, test coverage giả vờ, architecture bị ép vào pattern LLM phù hợp demo chứ không phù hợp domain, và team mất khả năng debug khi black-box model thay đổi behavior.
Thấy nhiều team celebrate "shipped in 2 days" rồi month 6 vẫn đang chase flaky test, refactor spaghetti prompt chain, và explain cho stakeholder tại sao feature "done" lại break khi model update minor version. Engineering maturity là biết khi nào không dùng AI, và xây guardrail cho những chỗ dùng — không phải maximize metric dashboard (site: labagent .tech)
Exactly, especially the part about knowing when not to use AI. Shipping in two days means very little if the next six months go into debugging and patching it. The tool should help us make better engineering decisions, not become an excuse to skip them.
I don't think there's a hard signal for this, but rather a "soft" one. Ask the developers what they created and why it matters to the business. If they can clearly explain what they built, why it exists, and the value it provides, that's probably a pretty good sign.
I like this take. If a developer can explain what they built, why they made those choices, and how it helps the business, that tells you a lot more than a PR count. The real test is whether they can still explain it six months later when something breaks or the requirements change.
The manual fixes per week metric hits home. I've shipped enough small tools to know that "it works" and "I never have to touch it" are very different things. The second one takes more thought upfront.
Exactly. Getting something to work is often the easy part. Getting it to keep working without someone constantly checking on it is a completely different engineering problem. The hidden maintenance hours are what make the real difference.