🎃 The October We Almost Burned Out
Last Hacktoberfest, our repo (a mid-sized CLI tool with about 4k stars) got hit with 380 pull requests in three weeks. Our normal monthly average is around 25. By October 12th, we had two maintainers spending every evening triaging PRs instead of writing code.
Here's the breakdown of what we actually received:
- ~140 were genuine, reviewable contributions
- ~90 were low-effort README/typo edits submitted purely for the swag
- ~110 were auto-generated or bot-assisted spam (changing whitespace, reordering imports, adding a single blank line)
- ~40 were duplicate PRs fixing the same "good first issue" because people didn't check if it was already claimed
That second and third category is what pushed us to change our process mid-month. This post is about what we built, what we automated, and what we'd do differently next year.
🔍 The Problem With "Just Review Everything"
The standard advice is "just review every PR on its merits." That works fine at 25 PRs/month. It does not work at 15 PRs/day when half of them are structurally identical.
The real cost wasn't rejecting bad PRs — GitHub makes that a two-click operation. The cost was triage time: opening each PR, reading the diff, checking if it touched anything meaningful, checking if an issue was linked, and then writing a polite rejection instead of just hitting close. Fifteen PRs a day at even three minutes each is 45 minutes of pure overhead before you've reviewed a single line of real code.
So the goal wasn't "reject more spam." It was "spend zero human minutes on obvious spam, and spend full attention on the stuff that matters."
🤖 Automating the First Filter
We built a GitHub Action that runs on every pull_request event and does a first-pass triage before a human ever looks at it. It doesn't auto-close anything by itself — it labels, and labels drive the rest of the workflow.
yaml
name: pr-triage
on:
pull_request_target:
types: [opened, synchronize]
jobs:
triage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Compute diff stats
id: diffstats
run: |
git fetch origin ${{ github.base_ref }} --depth=1
FILES_CHANGED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD | wc -l)
LINES_CHANGED=$(git diff --shortstat origin/${{ github.base_ref }}...HEAD | awk '{print $4+$6}')
echo "files=$FILES_CHANGED" >> "$GITHUB_OUTPUT"
echo "lines=$LINES_CHANGED" >> "$GITHUB_OUTPUT"
- name: Flag trivial-only changes
id: trivial
run: |
# Matches PRs that only touch markdown/whitespace, nothing in src/
CODE_TOUCHED=$(git diff --name-only origin/${{ github.base_ref }}...HEAD | grep -E '^(src|lib)/' | wc -l)
echo "code_touched=$CODE_TOUCHED" >> "$GITHUB_OUTPUT"
- name: Apply labels
uses: actions/github-script@v7
with:
script: |
const codeTouched = parseInt('${{ steps.trivial.outputs.code_touched }}');
const linesChanged = parseInt('${{ steps.diffstats.outputs.lines }}');
const labels = [];
if (codeTouched === 0 && linesChanged < 5) {
labels.push('needs-human-review:low-signal');
}
if (linesChanged > 500) {
labels.push('needs-human-review:large-diff');
}
await github.rest.issues.addLabels({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
labels,
});
That low-signal label became our single most useful signal. It doesn't mean "bad PR" automatically — some genuinely good doc fixes are five lines. But it means a human should skim it in under 30 seconds rather than giving it a full review pass.
🧹 The Spam-Specific Checks
Beyond trivial-diff detection, we added three checks aimed specifically at Hacktoberfest-pattern spam:
1. No linked issue, on a repo that requires one. We added a CONTRIBUTING.md rule that any non-trivial PR must reference an issue with Closes #123. A bot comment checks for that pattern and nags once:
python
import re
import sys
def has_issue_reference(pr_body: str) -> bool:
pattern = r'(closes|fixes|resolves)\s+#\d+'
return re.search(pattern, pr_body, re.IGNORECASE) is not None
if name == "main":
body = sys.stdin.read()
if not has_issue_reference(body):
print(":⚠️:No linked issue found. PRs without a linked issue "
"may be closed without review during Hacktoberfest.")
sys.exit(1)
2. Account age and history heuristics. We don't auto-reject new accounts — plenty of first-time contributors are genuine. But an account created in the last 48 hours, with zero other public activity, submitting a one-line change to an unrelated file, got an automatic spam:likely label for faster human triage. We used the GitHub REST API's created_at field on the user object plus public_repos count as a quick heuristic. It's crude, but it correctly flagged about 70% of what turned out to be spam, with a false-positive rate we were comfortable eating.
3. Duplicate detection on "good first issue" labels. We added a step that checks if an issue already has a linked, open PR before letting a second PR reference the same issue without a comment warning. This alone saved us dozens of redundant reviews.
⚡ The Actual Policy Change: Close Faster, Explain Less
The automation only gets you halfway. The bigger shift was cultural: we gave ourselves explicit permission to close PRs with a templated, non-personalized response when they hit spam:likely or low-signal with no linked issue.
Before:
Thanks so much for your contribution! After reviewing, we feel this change doesn't quite fit the direction of the project because X, Y, Z. We really appreciate you taking the time, and please feel free to look at our other open issues...
After:
Closing — this PR doesn't include a linked issue and doesn't modify
any source files. See CONTRIBUTING.md for what qualifies as a
valid Hacktoberfest contribution in this repo. Feel free to reopen
with more context.
That felt harsh the first few times we used it. But the alternative was maintainers writing a paragraph of empathetic feedback for a PR that added a single period to the README, forty times a week. We reserved the long, kind, detailed response for PRs that were clearly a genuine effort but missed the mark technically — those contributors deserve real feedback, and giving it generously is part of why people come back next year.
📊 What Changed, Numerically
After two weeks running this setup:
- Median time-to-first-response dropped from 36 hours to 4 hours
- Maintainer triage time dropped roughly 60%
- Genuine contributor PRs (the ~140 category) got reviewed faster because they weren't queued behind spam
- We did get a handful of complaints about the terse closing message — a fair trade-off in our view
We also opted our repo out of the official Hacktoberfest topic for about a week when spam spiked past a threshold we'd set informally, then opted back in once the automation was tuned. That's an option a lot of maintainers don't realize exists — you can add and remove the hacktoberfest topic from your repo at will, and DigitalOcean's spam-detection on their end does scan for it.
🙋 Where's the Line for You?
If you maintain anything with real traffic during October, I'm curious where you draw the line between "could be a genuine first-time contributor" and "this is noise." Do you rely more on automated heuristics like ours, or do you still hand-review everything on principle? And if you've tried bot-based triage — what signal ended up being the most reliable one for you?
We're going to keep iterating on this before next October, so I'd genuinely like to steal good ideas from other maintainers' playbooks.
Top comments (0)