ValorD33

Posted on Oct 3

GDPR-Friendly App Logging: Compare 3 Hosted Services Through Marketplace Incident Cost

#observability #gdpr #logging

Short answer: retain a small, pseudonymous incident ledger for longer than full application logs, and make tenant, purpose, retention class, and deletion state first-class fields. For an EU marketplace, that is the least complex design that can reconstruct a buyer-seller incident without paying to index every debug event for the same period. Compare hosted services by the controls they expose for this model, not by an EU-region checkbox.

The bill is usually made from four levers: bytes ingested, bytes indexed, time retained, and work performed to query, rehydrate, or export them. Egress and duplicate copies can matter too. The dominant term depends on the contract and traffic shape, so calculate it from an invoice and usage export before changing architecture. A service that charges primarily on ingestion calls for aggressive source filtering; one that separates hot indexed data from archives may reward moving evidence to a colder tier. No universal winner follows from the pricing page.

Count bytes first.

For cost attribution, every accepted event should carry a marketplace tenant or business-domain key. Without it, finance can report a logging total but cannot explain which seller, workflow, or incident class created the spend. Do not use an email address or phone number as that key. Those identifiers leak into indexes, alerts, saved searches, and exports with surprising ease.

What are you actually paying to preserve?

Start with a byte-flow model, not a vendor shortlist. Let N be accepted events, S their average encoded size, I the fraction sent to the searchable tier, R_h hot-retention time, and R_c cold-retention time. Your workload dimensions are approximately N * S for ingestion, N * S * I * R_h for hot storage, and N * S * (1 - I) * R_c for cold storage. Billing units differ, so this is an attribution model rather than a price calculator.

In a marketplace, verbose request logs often dwarf the evidence needed to answer the real question: which authenticated actor changed an order, which policy version ran, which outbound notification was attempted, and what state transition followed? OTP bodies, bearer tokens, raw message content, email addresses, and phone numbers add exposure without improving that reconstruction. Drop them at the source. Keep a delivery outcome, provider-neutral message class, attempt number, and correlation ID instead. Spam-filter and rate-limit investigations need state and timing; they rarely need the secret itself.

A workable event envelope looks like this:

from dataclasses import asdict, dataclass
from datetime import datetime, timezone
import hashlib
import hmac
import json

@dataclass(frozen=True)
class EvidenceEvent:
    occurred_at: str
    tenant_id: str
    subject_ref: str
    trace_id: str
    action: str
    outcome: str
    purpose: str
    retention_class: str
    schema_version: int = 1

def subject_ref(secret: bytes, internal_user_id: str) -> str:
    return hmac.new(
        secret, internal_user_id.encode(), hashlib.sha256
    ).hexdigest()

def encode(event: EvidenceEvent) -> bytes:
    return json.dumps(asdict(event), separators=(",", ":")).encode()

now = datetime.now(timezone.utc).isoformat()

The HMAC is pseudonymization, not anonymization: anyone holding the key and candidate identifiers can reproduce the reference. Treat the key and lookup capability as personal-data controls. Rotate keys under a documented plan, and keep the mapping boundary outside the general log search path.

That boundary matters.

The first cost move is to stop indexing evidence that nobody searches interactively. Keep the compact transition ledger searchable according to its operational need. Send bulky diagnostic context to a shorter-lived tier, or do not collect it. Sampling is acceptable for performance telemetry; it is a poor default for state-changing audit evidence because the one omitted event may be the event under dispute.

Make deletion and export properties of the schema

GDPR Article 5 establishes storage limitation. Articles 15, 17, and 20 cover access, erasure, and portability, while Articles 28, 30, and 32 matter to processor terms, records of processing, and security. These are separate obligations with conditions and exceptions; an engineering design should not collapse them into a single "delete everything" button. Legal counsel and the controller's policy determine which records must be erased or retained. The logging system must make that decision executable and provable. My decision rule is narrower: if the team cannot state a field's purpose, expiry, and subject-lookup path, that field does not enter the durable ledger.

Use a stable internal subject identifier at the application boundary, derive the pseudonymous subject_ref, and maintain a subject-to-event index with an expiry no longer than the data it locates. A user request can then produce a deterministic manifest across the hot store, archive, alert payloads, and any replay queue. Export should use a documented schema and include timestamps, purposes, and understandable event meanings, not an opaque dump of internal stack traces.

Deletion needs an asynchronous job with idempotency, bounded retries, and an evidence record that contains the request ID, affected stores, completion time, and policy basis, but not the deleted payload. Expired archives and immutable backup media need a documented deletion lifecycle. Claiming immediate erasure while a recoverable copy remains for an unspecified period is a policy gap, even if the primary index is clean.

This is also where account-level cost attribution becomes reliable. The same deletion manifest can count accepted bytes and retained objects by tenant and retention class. Keep that accounting aggregate after payload expiry only if it cannot be used to re-identify a person and the policy permits it.

How should you compare GDPR-friendly hosted app logging services?

Datadog, Better Stack, and Axiom all publish material relevant to EU logging, but their public control surfaces are organized differently. Datadog documents separate sites, including an EU site, and a log-archive workflow. Better Stack documents regional data storage and log-retention controls. Axiom documents deployment regions and dataset retention. Those are boundaries to verify, not endorsements.

Decision	Datadog	Better Stack	Axiom	Contract test
EU processing boundary	Select and verify the EU site and every connected integration	Select and verify the documented region	Select and verify the documented deployment region	Send a canary, then inspect destination, subprocessors, and support path
Retention unit	Validate index and archive policies separately	Validate source retention and any archive path	Validate dataset retention	Apply two classes and prove expiry at each boundary
Subject erasure	Confirm how indexed logs, archives, monitors, and derived data are covered	Confirm how stored logs and derived data are covered	Confirm how datasets and derived data are covered	Insert a synthetic subject, delete it, then search and export
Export boundary	Test search export plus archive retrieval	Test the documented query or export surface	Test the documented query surface	Reconstruct one incident from a clean account with rate limits enabled
Cost attribution	Map usage dimensions to tenant tags before procurement	Map usage dimensions to source and tenant metadata	Map usage dimensions to dataset and tenant metadata	Reconcile sampled usage records with the invoice

Public documentation can establish that a control exists. It cannot establish that your DPA, account region, archive bucket, alert destination, or support arrangement uses it correctly. Ask each provider the same written questions: where data and backups reside, which subprocessors can access them, what deletion reaches, how long deletion takes, how exports are rate-limited, and what happens after contract termination. Record the answers alongside the data-processing agreement and record of processing activities.

Run the contract test before signing and after material configuration changes. A synthetic subject is safer than a real customer's data and catches a common edge case: deletion succeeds in the primary search index while a saved alert, cold archive, or failed export job retains a copy.

Test the whole path.

Operate reconstruction as a product feature

A retention policy that exists only in a compliance document will drift. Express classes in configuration, review changes, deploy them through the same controlled path as application code, and alert on events with an unknown class. Rejecting an unknown class is appropriate for durable audit events; for ordinary diagnostics, routing to a short default window may preserve availability with limited exposure. Make that choice explicit.

Test three paths. First, replay a synthetic marketplace dispute from the evidence ledger and verify actor, order, policy version, notification attempts, and state transitions. Second, run access and erasure workflows against the same subject and inspect every destination. Third, export the incident into a clean environment and check that another engineer can understand it without tribal knowledge. Include throttling, partial failures, duplicate requests, and a provider timeout. Idempotent job identifiers prevent retries from producing inconsistent manifests.

Deploy schema changes additively. Readers should accept the current and previous schema versions while old events remain retained. A field rename that lands in writers before exporters is updated can turn a valid access request into a partial export. That failure is quiet. Schema-version metrics and fixture-based reconstruction tests expose it.

Operational access deserves the same care. Use least-privilege roles, time-bound elevated access, audit searches and exports, and review broad queries. Encryption in transit and at rest is baseline protection under a risk-based security program; it does not cure excessive collection.

The evidence we deliberately stop keeping

The final policy should name exclusions: raw credentials, OTP values, bearer tokens, full message bodies, unbounded request bodies, and direct contact identifiers in general-purpose logs. It should also define short retention for verbose diagnostics and longer, purpose-bound retention only for the compact incident ledger where justified.

This choice has a cost. After diagnostic data expires, engineers may be unable to reproduce a rare parser failure or inspect the exact content that triggered a delivery rejection. The response is better structured outcomes, counters, traces, and targeted temporary logging under change control, not indefinite collection. For a marketplace incident, reconstruction remains possible because the durable ledger preserves who acted, what changed, when it happened, and which policy decided the outcome.

The trade-off is explicit: less forensic texture, less personal data, and a bill that can be attributed to the marketplace activity that caused it.

Buy the service whose verified controls fit that evidence model and contract. Region labels and feature counts come later.

DEV Community