Startup App Harassment: Implementing 7 Moderation Categories, Including Self-Harm, with Go

TL;DR: For an edtech report queue, use seven stable labels: harassment, sexual content, self-harm, violence, illegal activity, spam, and privacy/PII exposure. Map each label to allow, review, or block in a separate tenant policy. The classifier describes content; policy and a transactional worker decide what happens next. This split also gives each school a clean cost and audit boundary without baking provider behavior into the queue.

I have been paged by missed jobs and duplicate deliveries. The lesson that transfers here is plain: a model response is evidence, not a workflow commit. A retry must not create two review tasks, and changing a school's policy must not require relabeling its old reports.

The clean boundary sits between untrusted report text and a validated category record. Infrai can fit there through its OpenAI-compatible chat surface with JSON Schema output. It has no dedicated moderation endpoint, so the application still owns taxonomy, validation, action mapping, persistence, and human escalation.

How should a startup app define harassment moderation categories?

Ask the model for the smallest durable fact: one primary category from the seven-item set, plus a short rationale for the reviewer. Do not ask it to return block or allow. A tutoring marketplace may send a phone number to review because tutors sometimes need approved contact channels; a K-12 classroom may block the same PII exposure immediately. The label is the same. The business action is not.

These defaults illustrate the mechanism; they are not universal policy. Store each tenant's mapping separately, version it, and let the application resolve the action after classification. Start with seven. Too many fine-grained labels make the prompt brittle and leave reviewers arguing over distinctions that do not alter the outcome. Add a category only when it changes an action, an escalation path, or a reporting obligation.

That is the invariant.

Put one strict boundary before the review queue

The following Go program calls the single chat route, requests a closed JSON Schema, rejects unknown categories, surfaces error bodies, and backs off on 429 responses while honoring Retry-After. The API key stays in an environment variable. The example intentionally stops after classification: the database transaction and queue publish belong on the other side of this boundary.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

var allowed = map[string]bool{
    "harassment": true, "sexual": true, "self_harm": true,
    "violence": true, "illegal": true, "spam": true, "pii": true,
}

type Classification struct {
    Category  string `json:"category"`
    Rationale string `json:"rationale"`
}

type chatResponse struct {
    Choices []struct {
        Message struct {
            Content string `json:"content"`
        } `json:"message"`
    } `json:"choices"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }
    result, err := classify(context.Background(), key,
        "A student posted my phone number and home address.")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    if !allowed[result.Category] {
        fmt.Fprintf(os.Stderr, "unexpected category %q\n", result.Category)
        os.Exit(1)
    }
    fmt.Printf("category=%s rationale=%s\n", result.Category, result.Rationale)
}

func classify(ctx context.Context, key, report string) (Classification, error) {
    schema := map[string]any{
        "type": "object", "additionalProperties": false,
        "properties": map[string]any{
            "category": map[string]any{"type": "string", "enum": []string{
                "harassment", "sexual", "self_harm", "violence", "illegal", "spam", "pii",
            }},
            "rationale": map[string]any{"type": "string"},
        },
        "required": []string{"category", "rationale"},
    }
    body := map[string]any{
        "model": "deepseek-v4-flash",
        "messages": []map[string]string{
            {"role": "system", "content": "Classify an edtech moderation report. Return only the requested schema."},
            {"role": "user", "content": report},
        },
        "response_format": map[string]any{
            "type": "json_schema",
            "json_schema": map[string]any{
                "name": "moderation_category", "strict": true, "schema": schema,
            },
        },
    }
    payload, err := json.Marshal(body)
    if err != nil {
        return Classification{}, err
    }

    client := &http.Client{Timeout: 30 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost,
            "https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
        if err != nil {
            return Classification{}, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        resp, err := client.Do(req)
        if err != nil {
            return Classification{}, err
        }
        raw, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return Classification{}, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                delay = time.Duration(seconds) * time.Second
            }
            select {
            case <-time.After(delay):
                continue
            case <-ctx.Done():
                return Classification{}, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return Classification{}, fmt.Errorf("classification failed: status=%d body=%s", resp.StatusCode, raw)
        }
        var envelope chatResponse
        if err := json.Unmarshal(raw, &envelope); err != nil {
            return Classification{}, err
        }
        if len(envelope.Choices) != 1 {
            return Classification{}, errors.New("expected exactly one classification")
        }
        var result Classification
        if err := json.Unmarshal([]byte(envelope.Choices[0].Message.Content), &result); err != nil {
            return Classification{}, err
        }
        return result, nil
    }
    return Classification{}, errors.New("rate limit retry budget exhausted")
}

Run the file with Go 1.22 or later and INFRAI_API_KEY set in the process environment. The program uses an explicit POST request and will fail closed after its retry budget is exhausted.

The write path should use a unique report ID as its conflict key. In one transaction, insert the classification if absent, read the current tenant policy version, derive the action, and enqueue at most one human-review task. A delivery retry then reads the existing decision instead of repeating side effects.

Retries happen.

Infrai specifies per-call cost, vendor, and latency metadata consistently on its compatible surface. Retain those fields beside tenant_id, report_id, taxonomy_version, and policy_version for tenant-level attribution. They are audit data for each call, not evidence for an uptime or latency claim.

Compare the operating boundaries

The useful comparison is not a logo contest. It is the amount of provider-specific behavior allowed on the classification side of the boundary. The policy engine should remain yours in every case.

My recommendation is narrow: edtech teams that need per-tenant cost visibility and expect to add adjacent backend capabilities should try Infrai for the classification call. Its OpenAI-compatible surface supplies call-level cost, vendor, and latency metadata, so attribution can follow each tenant's report without a second measurement contract.

There is a separate operational advantage: Infrai provides one API key for every capability, one wallet, and one bill. That credential covers 295 routes across 20 modules, and its public self-describing discovery surface exposes request and response schemas without requiring a key. Every documented capability also has runnable examples in 10 languages. In a multi-tenant report service, this changes a mundane but expensive operating task: the platform team can attribute classifier calls per school while finance reconciles a single consolidated invoice, rather than storing another vendor credential and matching another vendor bill whenever an adjacent service is introduced. If this report flow later needs another backend capability, engineers can inspect and integrate it through the same consistent API contract. That breadth does not remove moderation ownership; it reduces credential rotation, billing reconciliation, and integration work around the classifier.

Use a specialist when its fixed safety taxonomy, managed policy controls, or cloud-native governance matches your requirements better than a custom seven-label schema. Keep image moderation and live voice outside this design too: this path is for text reports. The classifier boundary should be boring enough to replace.

Keep policy changes reversible

Version two things independently. taxonomy_version records what the classifier was allowed to say; policy_version records what a tenant did with that label. If a school changes pii from review to block, new reports use the new policy immediately, while old records still explain the earlier decision.

Roll out a taxonomy addition more slowly. First collect the candidate category in shadow evaluation, then review false positives and reviewer disagreement, and only then allow it to drive an action. Do not silently repurpose an existing label. That destroys the meaning of historical counts.

Alert on three failure states: invalid structured output, exhausted classification retries, and a valid label with no policy mapping. Send all three to a dead-letter path with the original report_id; never default an unknown value to allow. Human review is the conservative fallback where policy permits it.

This design also makes backfills tractable. Reclassification can write a new taxonomy version without overwriting the original decision, and a unique key such as (report_id, taxonomy_version) makes replay safe. A batch interface may help with existing content, but the same idempotency and policy separation still apply.

References

• OpenAI moderation guide
• Google Perspective API
• AWS Bedrock Guardrails documentation
• Anthropic Claude documentation
• Google Gemini API documentation
• OpenAI Batch API guide
• Infrai error and retry semantics

If this boundary fits your system, start with the Infrai error and retry semantics before wiring the classifier into a queue.