Observability — Mnemose

Principle: Observability is always in scope. Any plan must first answer: how are we going to observe this in flight and understand its true state at any given point?

Architecture overview

Browser (console)
  └─ OTLP/HTTP:4318 ──────────────────────────────────────────────┐
                                                                    ▼
Node.js services (gateway, kernel-broker)                   OTEL Collector
  └─ OTLP/gRPC:4317 ──────────────────────────────────────► (docker)
                                                                    │
                                             ┌──────────────────────┤
                                             ▼                      ▼
                                          Jaeger               Prometheus
                                   (traces :16686)         (metrics :9090)

The OTEL Collector is the mandatory routing layer — services never talk directly to Jaeger or Prometheus. This decouples backend choices from service configuration.

Local dev stack

Local observability is optional and collector-based. When enabled, an OTEL Collector routes spans/metrics to Jaeger and Prometheus (Docker Compose files live outside the deploy path — the platform itself is Workers-only):

Service	URL	Purpose
OTEL Collector	gRPC:4317 / HTTP:4318	Span + metric ingestion
Jaeger	http://localhost:16686	Distributed trace visualization
Prometheus	http://localhost:9090	Metrics storage + query

Workers and the console emit OTLP; when no collector endpoint is configured the telemetry layer is a graceful no-op.

Performance budget

Target: < 2 s TTI (Time to Interactive) for the console on a cold page load over a typical developer connection. The dev perf overlay (PerfOverlay component, visible only in import.meta.env.DEV) shows live TTI measurement in the bottom-right corner of the browser.

Node.js service instrumentation

Package: @mnemose/telemetry

Workspace package at packages/telemetry/.

// In service entry point — MUST be the first import/call
import { initializeTelemetry } from "@mnemose/telemetry";
await initializeTelemetry({ serviceName: "mnemose-my-service" });

Environment variables:

• OTEL_EXPORTER_OTLP_ENDPOINT — gRPC endpoint (e.g. http://otel-collector:4317). Unset = graceful no-op.
• OTEL_SERVICE_NAME — overrides the serviceName option if set.

Auto-instrumentation included: HTTP, Pino.

Adding a new service

1. Add "@mnemose/telemetry": "workspace:*" to the service’s package.json.
2. Call await initializeTelemetry(...) as the very first thing in the entry point.
3. Add OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME env vars to the service environment (.env locally; Worker vars/secrets in production).

Browser instrumentation

Entry: apps/console/src/lib/console-telemetry.ts
Imported as first side-effect in apps/console/src/main.tsx.

Auto-instrumented: fetch requests (with W3C traceparent propagation to backend), document load.

Manual spans

import { consoleTelemetry } from "@/lib/console-telemetry";

const tracer = consoleTelemetry.tracer("my-feature");
const span = tracer.startSpan("operation-name");
span.setAttribute("key", value);
span.end();

Use useRef to hold a span that starts on mount and ends when async data arrives. See tenant-switcher.tsx for the canonical pattern.

Console loading state standards

All loading transitions must be:

1. Visually correct — skeleton shape matches the real layout; no content-flash or reflow when data arrives.
2. Instrumented — every loading boundary has an OTEL span so the load sequence is visible in Jaeger.

Loading context	Component	Span
Auth session init	auth-context.tsx	auth-init
GraphQL client ready	graphql.tsx	graphql-client-ready
Tenant query	tenant-switcher.tsx	tenant-query
Nav items (role)	sidebar.tsx → SidebarMenuSkeleton	(via tenant-query)
Route data	Per-view PageSkeleton component	(per query in the view)

Never show a raw UUID as tenant name. Use “No tenant access” or animated Skeleton bars until the name is available.

Production OTEL

Production services should have OTEL_EXPORTER_OTLP_ENDPOINT set (Worker vars / secrets). When unset, @mnemose/telemetry is a graceful no-op with zero overhead.

Recommended production targets — any OTLP-compatible backend:

• Traces — an OTLP-compatible collector or vendor backend.
• Metrics — scrape the collector’s :8889 metrics endpoint from your Prometheus.

The local-dev collector pipeline is the reference implementation; adapt the same pipeline config for production.

Trace correlation in logs

Pino instrumentation (in @mnemose/telemetry) automatically injects trace_id and span_id into log lines when a span is active. Log lines can be correlated directly with traces via the trace ID.