Stop Generating Changelogs from Git Commits: How I Fixed the 'WHY' Problem in ChangelogBot

I recently audited a release page for a popular open-source project and found a 140-line changelog that read like this:

* fix: typo in auth handler (#412)
* refactor: rewrite query builder (#415)
* feat: increase internal timeout to 15s (#418)

As a developer upgrading that package, those lines tell me almost nothing. Why was the query builder rewritten? Did it change memory consumption or SQL dialect support? Why was the timeout tripled? Was there an underlying socket leak under load?

Generating release notes directly from raw git commits or PR titles is one of the most widespread anti-patterns in modern DevOps. We treat commit logs—which are micro-level implementation breadcrumbs meant for authors—as user-facing documentation.

In this article, I will explain why traditional automated changelogs fail, introduce ChangelogBot—an open-source tool built to extract the actual intent ("WHY") behind changes using AI—and walk through how I re-engineered its GitHub metadata engine to survive secondary rate limits and silent CI failures.

The Mistake: Git Commits Are Not Release Notes

Conventional Commits (feat:, fix:, chore:) solved automated semantic versioning, but they created a massive documentation blind spot.

When developers open pull requests, the richest context lives in the PR description, linked issues, and discussion threads. That is where engineers explain edge cases, breaking behavioral nuances, and performance trade-offs.

However, standard release generators throw all of that away:

1. They prioritize the what over the why: A commit title like fix: cache invalidation on tenant switch describes code motion, not the customer-facing impact.
2. They flood readers with internal noise: Squash merges, dependency bumps, and formatting fixes bury critical behavioral changes.
3. They force consumers to hunt through PR history: If a changelog does not summarize the rationale, developers must click dozens of GitHub links just to evaluate an upgrade.

A real changelog should communicate intent and impact.

Enter ChangelogBot: Extracting the "WHY" with AI

To solve this problem, ChangelogBot takes a radically different architectural approach:

Instead of dumping commit messages into Markdown, ChangelogBot:

• Maps release commits to their original GitHub Pull Requests.
• Fetches the PR body, author, and associated discussions.
• Runs a local heuristic trust score to filter out trivial or low-quality descriptions.
• Leverages LLMs (Anthropic Claude or OpenAI) to extract and summarize the architectural "WHY" behind the change.
• Categorizes changes cleanly into human-readable sections, linking back to authors and pull requests.

Here is what the difference looks like in practice:

The Production Trap: Where Release Automation Fails Under Load

When I began contributing to ChangelogBot, the core AI extraction worked brilliantly on small repositories. But when I ran it against real production releases containing 40+ merged PRs, the tool hit two critical failure modes in CI/CD.

1. The Unbounded Promise.all Trap (Secondary Rate Limits)

The original commit resolver fetched pull request metadata for every commit in parallel:

// ANTI-PATTERN: Firing unbounded requests simultaneously
const prResults = await Promise.all(
  commits.map(commit => fetchPRDetails(commit.sha))
);

If a release contained 60 commits, this line fired 60 HTTPS requests to api.github.com in the exact same millisecond.

GitHub has a generous primary rate limit (5,000 requests/hour for authenticated users). However, GitHub also enforces Secondary Rate Limits (abuse detection mechanisms designed to prevent concurrent traffic spikes).

The result? The job failed with HTTP 403 Forbidden or HTTP 429 Too Many Requests—even though the token had over 4,800 hourly requests remaining!

2. Silent Degradation in CI

The second problem was defensive error handling gone wrong. To prevent crashes, previous code wrapped API lookups in broad try/catch blocks and returned empty arrays on failure:

// DANGEROUS: Silent fallback hiding network & auth errors
try {
  return await fetchPRDetails(commitSha);
} catch {
  return []; // CI exits 0, but changelog loses all PR context!
}

When GitHub throttled the pipeline or an API token lacked permissions, the bot did not fail. Instead, it silently dropped the PR associations, skipped the LLM enrichment, and generated a degraded changelog containing only bare commit SHAs. The GitHub Action finished with a green checkmark (exit 0), and the team only noticed the ruined changelog after shipping the release.

How I Re-Engineered the Metadata Pipeline

To make ChangelogBot rock-solid for large enterprise repositories, I designed and implemented a three-part architectural fix.

Step 1: Lightweight Bounded Concurrency (mapConcurrent)

I avoided pulling heavy external dependencies like p-limit and instead implemented a native, lightweight bounded concurrency worker directly in TypeScript:

/**
 * Maps items through an asynchronous mapper with a strictly bounded pool of in-flight executions.
 */
export async function mapConcurrent<Item, Result>(
  items: readonly Item[],
  limit: number,
  mapper: (item: Item, index: number) => Promise<Result>
): Promise<Result[]> {
  const boundedLimit = Math.max(1, Math.min(limit, items.length || 1));
  const results = new Array<Result>(items.length);
  let nextIndex = 0;

  const workers = Array.from({ length: boundedLimit }, async () => {
    while (nextIndex < items.length) {
      const currentIndex = nextIndex++;
      results[currentIndex] = await mapper(items[currentIndex]!, currentIndex);
    }
  });

  await Promise.all(workers);
  return results;
}

By capping the concurrency at GITHUB_METADATA_CONCURRENCY = 4, API lookups remain blisteringly fast while completely eliminating secondary rate-limit triggers.

Step 2: Discriminated Metadata Results

Instead of returning null or empty arrays on failure, I introduced typed result discrimination. Every API interaction now explicitly declares whether it succeeded or degraded, along with a structured failure reason:

export type MetadataLookupFailureReason =
  | 'not-found'
  | 'rate-limited'
  | 'unauthenticated'
  | 'invalid-response'
  | 'network-error';

export type MetadataLookupResult<T> =
  | { status: 'success'; data: T }
  | { status: 'degraded'; reason: MetadataLookupFailureReason; error?: unknown };

Now, the engine cleanly distinguishes between expected missing data (like a 404 for a commit that was rebased or cherry-picked) and critical operational failures (such as rate limits or invalid tokens).

Step 3: Transparent Diagnostic Reporting

I surfaced these metrics directly into the --dry-run console output and the --dry-run-json-report schema:

Metadata lookups: total=24, successful=23, degraded=1
Metadata fallback reasons: Commit 3a1b2c4: PR lookup returned 404 (not found)

In automated CI pipelines, teams can inspect the JSON payload and assert that degradedLookups === 0 before publishing a release to production.

Quick Start: Using ChangelogBot in Your Project

You can run ChangelogBot locally or wire it directly into your GitHub Actions release workflow.

Local Dry-Run

# Clone and build, or run via npx
npx @necoz-oss/changelog-bot \
  --release-tag v1.4.0 \
  --release-name 1.4.0 \
  --provider anthropic \
  --dry-run

GitHub Actions Workflow

Here is a minimal workflow that automatically updates CHANGELOG.md and creates a pull request when a release tag is pushed:

name: Update Changelog

on:
  push:
    tags:
      - 'v*'

jobs:
  changelog:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Generate Changelog with ChangelogBot
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          REPO_FULL_NAME: ${{ github.repository }}
        run: |
          npx @necoz-oss/changelog-bot \
            --release-tag ${{ github.ref_name }} \
            --release-name ${{ github.ref_name }} \
            --provider anthropic

The Takeaway

Your changelog is often the only piece of documentation your users and downstream developers read during an upgrade.

1. Stop using commit headers as release notes. Raw commits explain file modifications; your users need to understand technical and product intent.
2. Never fire unbounded Promise.all against third-party APIs in CI. Secondary rate limits will penalize concurrent bursts even when your hourly quota is full.
3. Say no to silent try/catch fallbacks. Discriminated union types and structured diagnostics ensure that your automation alerts you when upstream metadata degrades.

Check out the project on GitHub: necoz-oss/changelog-bot. Give it a star, test it on your next release, and see what automated release notes look like when they actually explain the "WHY".

What failure mode has broken an automated release in your CI/CD pipeline—and how did you catch it?

Connect with Me

If you found this article helpful, let's connect!

• 💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.