Observability in practice: from internal tools to SaaS

Every system in production has a bad day. The difference is who finds out first: you or your users. In a SaaS, the customer opens a ticket. In an ERP, invoices stop going out and the month-end close stalls. In an internal app, nobody complains: people just go back to the spreadsheet.

Observability is the ability to answer "what is happening, and why?" by looking at the data the system already emits, without shipping a new deploy full of console.log.

This post covers the concepts, what to measure first, the tools (free, paid and self-hosted), what they cost and how the architecture changes as the system grows. At the end there's a checklist you can apply to any project.

Monitoring or observability?

Monitoring answers questions you already knew to ask: "did CPU go over 90%?", "is the site up?".

Observability answers questions you didn't anticipate: "why are only Enterprise customers seeing slow reports since Tuesday?".

They complement each other. Monitoring tells you something broke; observability helps you find out what.

The signals

Rule of thumb: metrics alert, traces locate, logs explain.

What to measure first

Don't start by instrumenting everything. Start with what tells you whether users are being served.

RED, for anything that takes requests

• Rate: requests per second.
• Errors: the percentage that fail.
• Duration: latency in percentiles (p50, p95, p99). Averages hide problems: if 5% of requests take 8 seconds, the average still looks great.

USE, for resources

Utilization, Saturation and Errors for CPU, memory, disk, database connections and queues. Saturation is usually the most telling: a queue that only grows or a connection pool that's always full warns you before CPU does.

Business metrics

These are often the most useful: invoices issued per hour, orders completed, payment slips generated, logins. If invoice issuing drops to zero on a Tuesday at 10 a.m., something broke, even with calm CPU and no errors logged.

SLOs: how much failure is acceptable

An SLI is what you measure (say, % of requests that succeed in under 500 ms). An SLO is the target (99.5% over 30 days). The difference, 0.5%, is the error budget: how much failure you can spend before pausing features to work on stability.

Every extra nine costs a lot more. An internal app used during business hours rarely needs 99.9%.

OpenTelemetry: instrument once

OpenTelemetry (OTel) is the CNCF's open standard for producing and shipping logs, metrics and traces. Almost every tool in this post accepts OTel: you instrument your code once and switch vendors by changing configuration, not code.

This is the most important advice in the post: don't tie your code to a vendor's proprietary SDK. Prices change, the company grows, and switching tools shouldn't mean rewriting your instrumentation.

In Next.js, the framework is already instrumented. All it takes is an instrumentation.ts at the project root (or inside src):

import { registerOTel } from "@vercel/otel";

export function register() {
  registerOTel({ serviceName: "erp-api" });
}

To measure a specific operation, create a span:

import { SpanStatusCode, trace } from "@opentelemetry/api";

const tracer = trace.getTracer("billing");

export async function issueInvoice(orderId: string) {
  return tracer.startActiveSpan("issue-invoice", async (span) => {
    span.setAttribute("order.id", orderId);
    try {
      return await sendToTaxAuthority(orderId);
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Where the data goes comes from standard environment variables such as OTEL_EXPORTER_OTLP_ENDPOINT, not from code. Java, .NET, Python, Go and PHP have equivalent SDKs, and several offer automatic instrumentation with no code changes.

Logs that actually help

• Structured (JSON), not free text. That way you can filter by field:

  tenant_id = 42 AND level = error

• With a trace_id. That's what links the log to the trace: from an error you get to the whole request, and back.

• Levels with intent: error is something someone needs to look at; warn is something odd that resolved itself; info is business events; debug stays off in production.

• No personal data. Tax IDs, emails, tokens, passwords and card data don't go in logs. Under privacy laws like GDPR and Brazil's LGPD, logging is processing personal data too, and logs tend to end up in places with looser access control than the database.

{
  "time": "2026-10-16T13:04:22.118Z",
  "level": "error",
  "service": "erp-api",
  "tenant_id": "42",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "msg": "failed to issue invoice",
  "order_id": "4821",
  "error": "timeout after 30s",
  "attempt": 3
}

The tools

Open source and self-hosted

Free in license, not in operations: you pay in servers, disk and someone's hours looking after it.

The most common combination, and one that covers most cases well, is OTel Collector + Prometheus + Loki + Tempo + Grafana. You can start on a single server with Docker Compose, and for local testing there's the grafana/otel-lgtm image, with everything in one container.

Free SaaS plans

To get started without any servers. Figures as of October 2026; check the official pricing page, since they change often.

For an internal app or an early-stage SaaS, Sentry (errors) + Grafana Cloud (metrics, logs and traces) + an uptime monitor usually fits in the free plans for quite a while.

Paid platforms

For a sense of scale, Datadog at annual list price: about US$15 per host/month for infrastructure and US$31 per host/month for APM. Logs are charged twice: US$0.10 per GB ingested plus a fee per million indexed events. Ten servers with APM already cost over US$450 a month before any logs.

Cost traps

Observability bills rarely blow up because of the plan. They blow up because of volume.

• Cardinality. Every unique combination of labels becomes a series. An http_requests metric with route (50) × status (5) × method (4) is 1,000 series. Add user_id with 20k users and it's 20 million. Never use user, order or request IDs as metric labels; those belong in traces and logs.
• Debug in production. One log.debug inside a loop can produce more GB than the rest of the system combined.
• Retention. Keeping everything for a year rarely makes sense. Errors and traces: 15 to 30 days. Aggregated metrics: months. Audit logs: whatever the law or the contract requires, on cheap storage.
• Health checks. The load balancer hits /health every few seconds, and every hit becomes a log and a trace. Filter them before paying for them.
• Per-host billing with autoscaling. With Kubernetes or autoscaling, host count varies, and some vendors bill on the month's peak.

Self-hosted or SaaS?

The rule I follow: start on SaaS, with OpenTelemetry in the code. Consider self-hosting when the bill exceeds the cost of a server plus a few hours of maintenance a month, or when there's a requirement that data never leaves your infrastructure. Since the code speaks OTel, migrating means changing the destination in the Collector.

A self-hosting gotcha: if the monitoring server goes down along with the system, you're blind exactly when you need it most. Keep at least one uptime monitor outside your infrastructure.

How to scale

Observability grows in phases, along with the system.

Phase 1: one app, one database

• Errors with stack traces (Sentry or GlitchTip).
• An external uptime monitor on the main endpoints.
• Structured logs, centralized in one place.
• One dashboard with the API's RED and the main business metrics.
• Alerts for: system down, high error rate, stalled queue.

That alone handles most incidents for an internal app or a small SaaS.

Phase 2: a few services, queues and integrations

• Distributed tracing with OTel, to follow a request from the API to the worker and the external integration.
• trace_id in every log.
• SLOs for critical flows: login, checkout, invoice issuing.
• An OTel Collector between the apps and the destination, to filter and route without touching the apps.

Phase 3: many services, many teams, lots of volume

• Two-tier Collector: an agent on each machine or node (local collection, light processing) and a central gateway (sampling, filtering, routing).
• Trace sampling. Keeping 100% of traces for a system with millions of requests is expensive and mostly useless. There are two approaches:
  • Head sampling: decides at the start of the request (keep 10%, say). Simple and cheap, but it may drop exactly the trace with the error.
  • Tail sampling: decides at the end, after seeing the whole trace. Keeps every error and slow request and only a sample of the rest. Requires a gateway that holds spans in memory for a few seconds.
• Metrics derived from traces: the Collector computes RED from spans before sampling, so metrics stay exact even if you keep only 5% of traces.
• Tiered storage: recent data on fast disk, older data on object storage (S3, R2, GCS). Loki, Tempo and Mimir already work this way.
• Metrics at scale: Prometheus alone only goes so far; then come Mimir, VictoriaMetrics or Thanos for high availability and long retention.
• Governance: standard naming (OTel's semantic conventions), cardinality limits per team and an owner for every dashboard and alert.

A gateway with a health check filter and tail sampling looks like this (processors from the Collector's contrib distribution):

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  # Drop health checks before paying for them
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["url.path"] == "/health"'
  # Keep every error, everything over 1 s and 10% of the rest
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: sample
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }
  batch:

exporters:
  otlphttp:
    endpoint: https://otlp.example.com

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/health, tail_sampling, batch]
      exporters: [otlphttp]

With more than one gateway instance, all spans of a trace must land on the same instance. For that, the agent tier uses the loadbalancing exporter, which routes by trace_id.

Alerts that don't become noise

• Alert on symptoms, not causes. "Checkout error rate above 2% for 5 minutes" is actionable. "CPU at 85%" often isn't: if users are fine, nobody needs to wake up.
• Alert on error budget burn rate: it fires when, at the current pace, the month's SLO will be blown. It catches both sudden outages and slow degradation.
• Every alert has an owner and a runbook: what to check first, how to mitigate, who to call.
• Two levels: what wakes someone up (system down, money stuck) and what becomes a task for the next day.
• Review monthly. An alert that fires and nobody acts on should be tuned or deleted.

On-call tools like PagerDuty, or the incident management in Grafana and Better Stack, organize rotations and escalation. For small teams, a well-filtered Slack or Telegram channel does the job.

By type of system

SaaS

• tenant_id on every log and trace. The most common support question is "what happened to customer X?".
• SLOs per plan or per large customer. The system average can look great while one customer is having a terrible time.
• Noisy neighbors: one tenant importing 1 million rows can degrade everyone. Per-tenant usage metrics reveal this (watch the cardinality if you have thousands of tenants).
• Front-end errors and RUM: much of what customers feel happens in the browser, not the API.
• A public status page: cuts down tickets during an incident.

ERP

• Scheduled jobs are the heart of it: month-end close, reconciliation, payroll, bank statement imports. The most dangerous failure is the job that doesn't run: it produces no error and no log. Use a heartbeat (dead man's switch): the job checks in when it finishes, and the alert fires if the check-in doesn't arrive. HHealthchecks.io and Uptime Kuma (push monitors) do this.
• External integrations: tax authorities, banks, payment gateways and carriers go down and slow down. Measure latency and errors per integration, with timeouts and retries visible.
• Queues: queue size, age of the oldest message and what landed in the dead letter queue.
• Predictable peaks: month-end, tax closing, payroll. Compare with the same period last month, not with last week.
• Database: slow queries (pg_stat_statements on Postgres, Query Store on SQL Server), locks and connections. In an ERP, the bottleneck is almost always the database.
• Auditing isn't observability. Who changed what belongs in the database, with its own retention and access control, not in a log that expires in 15 days.

Internal apps

• Small budget, big impact: if the order system stops, operations stop.
• The minimum is enough: errors, uptime and centralized logs. It fits in a free plan or on a small server.
• Usage is a signal: if active users drop, it may be a bug, slowness or a process that changed. Nobody is going to open a ticket.
• Business hours: after hours, only alert on what truly stops the company.

A stack for each size

In every row, OpenTelemetry is the constant: it's what lets you move between rows in this table without rewriting anything.

Checklist

• Errors captured with stack trace and deploy version
• External uptime monitor, outside your infrastructure
• JSON logs, centralized, with trace_id and no personal data
• RED (rate, errors, p95/p99 latency) on the main endpoints
• At least one business metric per critical flow
• A heartbeat on every scheduled job
• Latency and errors per external integration
• Instrumentation with OpenTelemetry, no proprietary SDK in the code
• No user or order IDs as metric labels
• Retention defined per data type
• Health checks filtered out
• SLOs for critical flows
• Alerts on symptoms, with an owner and a runbook
• Monthly review of alerts and costs