Tracking AI usage (AI Cost Analytics)

AI Cost Analytics shows what your AI features really cost — by customer, feature, model and plan — without storing prompts or generated content.

The MVP is event-based: your application keeps calling AI providers (OpenAI, Anthropic, Azure, etc.) the same way as today, and emits a standardized SaaS Tracker event after each AI call or value signal. There is no gateway, no proxy, no prompt logging, no model routing in this version.

Privacy by design. SaaS Tracker tracks AI usage, cost and value metadata — not your prompts or generated content by default. Do not send prompt, messages, completion, output, response_text or any chat transcripts. They are not part of the schema and will be ignored.

Why send AI usage events

• See AI cost over time and by customer organization.
• Identify which AI features and customers consume your AI margin.
• Spot trial accounts that generate disproportionate AI cost.
• Compare providers and models for cost-per-call and error rate.
• Connect AI cost to plan and MRR (AI cost % of MRR, margin-risk accounts).

AI Cost Analytics is included on the AI Cost plan (€49/mo) and on Starter, Growth and Enterprise.

Supported AI events (MVP)

Send these as normal events to the existing Ingest API. The event name goes in the standard event field; AI metadata goes in a properties object.

• ai_call_completed — an AI call returned successfully
• ai_call_failed — an AI call failed (provider error, timeout, etc.)
• ai_output_accepted — a user accepted/used the AI output
• ai_output_rejected — a user explicitly rejected the AI output
• ai_output_edited — a user kept the AI output but edited it

ai_call_completed is the primary event. ai_call_failed enables error rate tracking. The three output events feed value signals (cost-per-accepted-output, etc.) on Growth and above.

Example: ai_call_completed

{
  "event": "ai_call_completed",
  "user_hash": "a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd",
  "customer_org_id": "acme-corp",
  "properties": {
    "feature": "support_reply_generator",
    "provider": "openai",
    "model": "gpt-5.4-mini",
    "request_type": "generate",
    "input_tokens": 1200,
    "output_tokens": 350,
    "total_tokens": 1550,
    "estimated_cost_eur": 0.014,
    "latency_ms": 1420,
    "success": true,
    "workflow_id": "support-ticket-789",
    "ai_call_id": "call_01HV..."
  }
}

Example: ai_call_failed

{
  "event": "ai_call_failed",
  "user_hash": "a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd",
  "customer_org_id": "acme-corp",
  "properties": {
    "feature": "support_reply_generator",
    "provider": "openai",
    "model": "gpt-5.4-mini",
    "latency_ms": 8500,
    "error_code": "rate_limited",
    "workflow_id": "support-ticket-789",
    "ai_call_id": "call_01HV..."
  }
}

Do not put raw provider error responses or stack traces in error_message. Use a stable, short error category in error_code instead.

Example: ai_output_accepted / rejected / edited

Always include feature, provider, model and ai_call_id on output events — the dashboard uses them to compute per-feature acceptance rate, value-signal-by-provider/model and accurate cost-per-accepted-output (which joins on ai_call_id).

{
  "event": "ai_output_accepted",
  "user_hash": "a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd",
  "customer_org_id": "acme-corp",
  "properties": {
    "feature": "support_reply_generator",
    "provider": "openai",
    "model": "gpt-5.4-mini",
    "workflow_id": "support-ticket-789",
    "ai_call_id": "call_01HV..."
  }
}

{
  "event": "ai_output_edited",
  "user_hash": "a1b2c3d4e5f6789012345678901234567890123456789012345678901234abcd",
  "customer_org_id": "acme-corp",
  "properties": {
    "feature": "support_reply_generator",
    "provider": "openai",
    "model": "gpt-5.4-mini",
    "workflow_id": "support-ticket-789",
    "ai_call_id": "call_01HV...",
    "edit_ratio": 0.12
  }
}

Field reference

All AI fields go under properties. They are all optional unless noted. Token and latency fields default to 0 in ClickHouse when omitted; cost stays unknown (not 0) unless you send an explicit amount or ingest resolves it from the catalog.

ai_call_completed

ai_call_failed

ai_output_accepted / rejected / edited

Why we ask for provider, model and ai_call_id on output events too. ClickHouse doesn’t automatically know which ai_call_completed an output event belongs to. To compute per-feature acceptance rate, edit-ratio distributions or true cost-per-accepted-output, we either group by the fields you send on the output event itself or join to the matching call via ai_call_id. If you only send feature, you’ll still get acceptance rate per feature, but provider/model breakdowns and the joined cost view will be incomplete.

Privacy and what NOT to send

• Do not send the user’s prompt, system prompt or chat transcript.
• Do not send the model’s generated answer or any document/content the AI processed.
• Do not put personal data into feature, error_message or any other property.
• error_message must be a short, stable category text — never the raw provider response.

If prompt/output logging becomes useful later, it will be a separate, opt-in enterprise capability. The default product never stores them.

Cost calculation

SaaS Tracker resolves cost at ingest time (per event, not recalculated later). Priority:

1. Explicit event cost — cost_amount + cost_currency, or estimated_cost_eur / estimated_cost_usd
2. Customer-specific unit price in the model catalog (Admin)
3. Global catalog unit price (EUR per 1M tokens)
4. Unknown — cost stays empty (NULL in analytics), not confused with zero

Send provider and model strings exactly as your app uses them. They must match either:

• a canonical registry key provider:model (e.g. openai:gpt-5.4-mini), or
• an alias configured in Admin (e.g. Azure deployment name → canonical model)

Nested properties.ai.* is supported (fields are flattened at ingest).

Resolution metadata (stored on the event)

After ingest, these fields may appear on the stored JSON (and in ai_usage_events):

Unknown vs zero: missing cost is not stored as 0. Send explicit 0 only when cost is truly zero.

Operator: unmapped models (Admin portal)

Superusers open Admin portal → Unmapped AI models to see (provider, model) pairs that returned unknown_model, then Create alias to map them to the catalog. Historical events are not backfilled.

Best practices

• Use a small, stable set of feature names. Keep them human-readable, e.g. support_reply_generator, meeting_summary. This is your primary slice in the dashboard.
• Always send customer_org_id. Without it AI Cost cannot show per-customer or per-MRR views.
• Always include feature, provider, model and ai_call_id on the matching ai_output_accepted/rejected/edited events too. They are the join keys for the value-signal views; without them the dashboard cannot compute true cost-per-accepted-output or value signals by provider/model.
• Send workflow_id whenever a single user action triggers both an ai_call_completed and a later ai_output_accepted/rejected/edited. This is the coarser correlation key when you don’t have an ai_call_id.
• You may send estimated_cost_eur from your app, or rely on the catalog when provider/model match registry or aliases.
• Use stable API model ids where possible (gpt-5.4-mini, not an internal nickname), or add an alias in Admin portal.
• Send latency_ms even when cost is unknown. Latency and error rate are useful by themselves.

What you’ll see in the product

Open Analytics → AI Cost in the dashboard. You’ll get:

• AI cost over time, calls, tokens.
• Cost by feature, by provider/model and by customer organization.
• AI cost as % of MRR, AI cost by customer billing status, margin-risk accounts (when revenue/org metadata is imported via the Organization data CSV import).
• AI call explorer: browse, filter and sort individual calls by time, organization, feature, status and workflow/call ID (metadata only — no prompt or content).
• Value signals (Growth and Enterprise): acceptance / rejection / edit rate per feature, edit-ratio distribution, and accurate cost-per-accepted-output joined via ai_call_id.

If you haven’t sent any AI events yet, the page shows an empty state with a copy-paste tracking snippet and this same privacy note.

Next: Plans and limits or Ingest API event schema.