I recently audited a release page for a popular open-source project and found a 140-line changelog that read like this:
* fix: typo in auth hand...
For further actions, you may consider blocking this person and/or reporting abuse
This is such a great write-up! πΈ
I really like how you explained both sides of ChangelogBot.
Why release notes should explain the WHY, not just repeat commit messages, and what it actually takes to make that work reliably in real CI environments.
The bounded concurrency and degraded metadata handling were important improvements for the project, so thank you again for contributing them and for documenting the reasoning behind them so clearly! π
Iβm really happy to see ChangelogBot evolving through contributions like this. π
Thanks so much, @nyaomaru! It was a real pleasure collaborating on this and helping make the metadata pipeline rock-solid for bigger releases. I really appreciate your thorough reviews across the PRs and look forward to seeing the project continue to grow.
Iβm really happy I got to build this together with you too, John! πΈπ
Letβs keep building and improving it! ππ
The part that stuck with me is the dry-run report as a gate, not just a log. Asserting degradedLookups === 0 before publishing turns a silent failure into a blocked release, which is exactly the difference between knowing something went wrong and catching it in time. I also liked that a missing WHY stays visibly missing instead of being filled in with a confident guess β that honesty is what makes the rest of the changelog trustworthy.
Glad those two pieces stood out to you. Turning the dry-run JSON into a hard CI gate and refusing to guess missing context were both about making sure the output is actually reliable in production. Once an AI tool sneaks in a single plausible-sounding hallucination, developers quickly lose trust in the entire changelog.
This is especially true for desktop end-users, who are not reading your commit history: an Electron app's release note that says 'refactor IPC layer' tells a user nothing about whether their crash is fixed or whether the update is safe to install. The shift from implementation breadcrumbs to user-impact notes is the same one we made for auto-update changelogs β grouping changes under fixed, changed, and known issues with one-line plain-language summaries. The rate-limit hardening detail is also a real gotcha: a release-note generator that silently skips a failed fetch produces a thinner changelog with no signal that anything went wrong.
Desktop and Electron apps are a prime example of where commit-driven changelogs fall flat. End-users deciding whether to click "Update" only care if a release fixes their issue or changes their workflow, not how the IPC layer was refactored under the hood. And catching those silent rate-limit drops was huge for us. A partial changelog that looks complete is one of the easiest ways to accidentally hide important release notes.
We went the other way round at work: the change log is written before the code. Each release gets its change log, product update and press release at the start of development, and they go out to everyone client-facing, the devs and QA together. The why is decided first, so nobody has to dig it out of PR bodies later. Your trust score covers the cases where nobody wrote it down. What does ChangelogBot write when a PR body fails it: a guess, or a visible gap?
I really like that approach! πΈ
Writing the WHY before implementation is probably the nice solution when the whole team can work that way.
One small clarification about ChangelogBot though: the trust score doesnβt try to fill in a WHY that nobody wrote down. Itβs actually there to prevent that kind of guessing.
If a PR body is empty, has no usable WHY candidate, or has a low local trust score, it gets skipped before reaching the model. And if the modelβs result doesnβt meet the required confidence, that WHY is skipped too.
So the changelog entry remains, but
ChangelogBotwonβt invent a rationale for it. In dry-run diagnostics, the skip/fallback reason is visible so you can see why enrichment was omitted.Iβd much rather have a visible missing WHY than a confident-sounding hallucination. πΉ
A visible skip beats a confident guess, agreed. Our agent pipeline leans the same way: every tool call its guard refuses comes back with the reason, so the agent corrects course instead of guessing around a silent no. Does the skip reason only show in dry-run, or does a normal run surface it too, say in the CI summary?
Right now, per-PR skip reasons (
fallbackReasons) only show up in--dry-runand--dry-run-json-report. A normal run stays quiet so it doesn't clutter the logs, and it only drops a"Generated without LLM. Reason: ..."note into the changelog PR body if the entire LLM pass falls back. For CI, the usual setup is running--dry-run-json-reportas a quick check before the actual release step.