Observability

Precept ships with optional OpenTelemetry (OTel) instrumentation. Setting a single environment variable causes the server and Workflow worker to begin emitting distributed traces and structured logs to an OTLP-compatible collector. With the variable unset, the OTel SDK is loaded but no exporters are registered — the process produces no telemetry traffic and incurs no overhead from network attempts.

What's instrumented

Out of the box, the following are auto-instrumented and produce spans:

Layer	Coverage
Incoming HTTP requests	Every request reaching the Express server gets a root span with the route, method, status code, and timing.
Outgoing HTTP requests via Node http / https	Calls made through the built-in http/https modules (e.g. third-party SDKs that use them internally).
Outgoing HTTP requests via fetch	Node's native fetch (built on undici) is instrumented separately so SSO proxy, Anthropic, MCP, and ingestion adapter calls all produce client spans.
Express middleware	Each middleware in the request chain emits a span, so you can see where time was spent in routing, body parsing, auth, etc. These spans need the OpenTelemetry ESM loader hook. The server entrypoint (index.mts) registers this hook before OpenTelemetry starts. Without the hook, Express has no middleware spans, and server span names have no route.
oRPC handlers	Each oRPC procedure call gets its own span nested under the request span.
Precept module actions	Each module action gets a module.action span, with a child span for each step. See Module action spans.
Workflow engine stages	Each engine stage gets an engine.stage span on the Workflow worker, with a child span for each timed block of work. See Workflow engine spans.
PostgreSQL queries	Every pg query executed by the server or worker produces a span with the SQL text and parameter count.
pino log records	Application log lines are forwarded to the OTLP logs endpoint alongside traces.

Trace context is propagated automatically between layers, so a single incoming request shows up as a tree: HTTP request → middleware → handler → outgoing HTTP/DB calls.

Enabling OpenTelemetry

To turn instrumentation on, set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable on both the server and Workflow worker processes. Point it at the base URL of an OTLP/HTTP-compatible collector:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc.cluster.local:4318

When this variable is set:

• Traces are exported to <endpoint>/v1/traces
• Logs are exported to <endpoint>/v1/logs

A trailing slash on the endpoint is tolerated and normalized internally.

When this variable is unset, the OTel SDK starts but registers no exporters. No exports are attempted; no connection errors appear in logs.

Configuration reference

Configuration is primarily via standard OTel environment variables, plus one Precept-specific variable that controls database query redaction (see Database query redaction below). Set these on the same processes (server, worker) where they take effect.

Variable	Effect
OTEL_EXPORTER_OTLP_ENDPOINT	Base URL of the OTLP/HTTP collector. Unset → no exporters registered.
OTEL_SERVICE_NAME	Service name attached to all telemetry. Defaults to precept-server for the API server and precept-worker for the Workflow worker.
OTEL_RESOURCE_ATTRIBUTES	Comma-separated key=value pairs attached as resource attributes to every span and log. Use this for per-deployment tags such as tenant labels, environment, or deployment ID. Example: tenant_label=acme,environment=prod,deployment.id=42
OTEL_EXPORTER_OTLP_PROTOCOL	Wire protocol. Unset → traces and the in-process log processor use OTLP/HTTP JSON, but the pino log forwarder defaults to OTLP/HTTP Protobuf. Set to http/json to make every exporter use JSON, or http/protobuf for uniform Protobuf.
OTEL_LOG_LEVEL	Internal log level of the OTel SDK itself (e.g. debug). Useful for diagnosing exporter connectivity issues.
PRECEPT_OTEL_CAPTURE_DB_STATEMENT	Set to true to disable database query redaction and emit the full SQL text on db.statement attributes. See Database query redaction.

TIP

Most collectors accept both JSON and Protobuf, so leaving OTEL_EXPORTER_OTLP_PROTOCOL unset is fine. Set it explicitly only if you've constrained your collector to a single protocol.

Database query redaction

By default, the db.statement attribute on every database span is replaced with the literal string [redacted]. This protects against incidental leakage of business or user data into your traces — the database client renders parameter values into the SQL text before the query is sent, so without redaction every row of every insert, update, and lookup would appear in your trace storage.

Other database attributes (db.system, db.name, db.operation, query timing, status) are unaffected, so you still see which operation ran against which table and how long it took.

To temporarily disable redaction for debugging — for example, when diagnosing a slow query in a non-production environment — set:

PRECEPT_OTEL_CAPTURE_DB_STATEMENT=true

WARNING

Enable full db.statement capture only in environments where the trace data is appropriate to expose. The captured SQL will contain whatever your application wrote — including user IDs, email addresses, encrypted credential blobs, workflow definitions, and any other column data that flowed through the query.

Module action spans

Precept adds its own spans for each module action. These spans use the instrumentation scope precept.

Span	Wraps	Attributes
module.action	The full run of one action, from configuration resolution to includes	precept.module.vendor, precept.module.name, precept.model, precept.resource, precept.action, precept.pagination (true or false), precept.tenant (only when the identity has a tenant)
module.action.config	The resolution of the action configuration	—
module.action.page	One call to the action handler. A paginated action has one span for each page.	precept.page
module.action.validate	Response validation. This span exists only when the validation mode is not disabled.	precept.validate.mode
module.action.map	The mapping to the target model. This span exists only when a mapping applies.	precept.mapping.target
module.action.include	The resolution of @include relationships. This span exists only when the request has @include and the validation mode is disabled.	—

The outgoing HTTP calls of a handler are children of module.action.page. Their client spans show the real downstream URL and status.

An included relationship runs as a separate module action. Its module.action span is a child of module.action.include.

precept.page counts the handler calls of one action run, from 1. It is not the page number of the downstream API. For example, a request with @page: 5 has one page span, with precept.page set to 1.

Workflow engine spans

The Workflow worker adds a span for each engine stage and for each timed block of work in it. These spans use the instrumentation scope precept.

Span	Wraps	Attributes
engine.stage	One stage of a Workflow run	precept.run.id, precept.run.attempt, precept.stage.index, precept.stage.kind, precept.generation, precept.stage.cancelled (only when the stage was stopped or its worker was lost)
sql.table, sql.sink	One SQL query of the stage	The block attributes below
api.calls	One batch of API calls	The block attributes below
js.rowFn, js.routeFn	A JavaScript step's body	The block attributes below
trailer.check	The trailer check of a file read	The block attributes below
manifest.rejects	The export of one node's rejected rows for a Manifest	The block attributes below

A block's span can carry these attributes, and no others:

Attribute	Meaning
precept.run.id	The run's ID.
precept.run.attempt	The run's attempt number, from 1.
precept.stage.index	The stage's index in the run's plan. The reject export of a Manifest has -1.
precept.node.id	The Workflow node that the block works for.
precept.run.reads_decrypted	true when the run reads a decrypted file. Only then is it set.
opId	The plan operation that the block computes.
nodeId	The Workflow node, when a block sets it as a detail. Most blocks set precept.node.id instead.
stageIndex	The stage's index, when a block sets it as a detail.
failed	true when the block failed. Only then is it set.

A block's span is a child of its engine.stage span. Spans that start inside a block are children of the block. For example, the module.action spans of an API call batch are children of api.calls.

Each stage runs as a separate Temporal activity, and Precept installs no Temporal tracing interceptors. So each engine.stage span is the root of its own trace. To find all the traces of a run, search for its precept.run.id.

File paths and other block details are not exported. When a block or a stage fails, its span gets ERROR status and the error attributes below. A stage that a user stopped, or whose worker was lost, is not a failure: its span has precept.stage.cancelled and no ERROR status. When a run reads a decrypted file, the span holds only the class of the error, never its message, because a message can quote a value from the file.

The worker has no Performance page store, so these spans go only to your OTLP backend.

Error attributes

When a Precept span fails, it gets ERROR status, an exception event and the precept.error.* attributes. The exception event holds exception.type, exception.message and exception.stacktrace.

Attribute	Value
precept.error.name	The name of the error, for example DownstreamHTTPResponseError.
precept.error.status	The HTTP status of the error. On the HTTP server span, this is the response status.
precept.error.code	The code of the error. Set only when the error has a string or number code.
precept.error.hint	The hint of the error. Set only when the error has a string or number hint.
precept.error.fields	JSON of the other properties of the error, for example the data and headers of a downstream response. Not set when the error has no other properties.
precept.error.cause	JSON array of { name, message } for each error in the cause chain, to a depth of 5.

Precept cuts precept.error.fields and precept.error.cause at 8192 characters (UTF-16 code units). A cut value is not valid JSON. The cut controls the span size. It is not redaction.

The final error handler of the server records each error on the HTTP server span, for every status. The server span gets ERROR status only when the response status is 500 or more.

Trace IDs in responses

Each server response has an X-Trace-Id header with the trace ID of the request. The CORS configuration exposes this header, so browser code on another origin can read it.

The error body from the final error handler also contains traceId:

{
  "error": {
    "status": 404,
    "name": "TraceNotFoundError",
    "message": "Trace '0af7651916cd43dd8448eb211c80319c' is not in memory",
    "traceId": "4bf92f3577b34da6a3ce929d0e0e4736"
  }
}

A validation error (status 400) has name, message and traceId, but no status. Some responses do not go through the final error handler, for example the response validation error (status 502). These responses have the header, but no traceId in the body. If the server has no valid span for a request, the response has no header and no traceId.

If a user reports an error, ask for the trace ID. Then find the trace in your OTLP backend or on the Performance page. The Performance page holds the span tree only for a failed request.

Performance page and error traces

The Performance page (/performance) and the /rest/telemetry API read an in-process store of spans. The store runs with or without OTEL_EXPORTER_OTLP_ENDPOINT. The page and the API need the telemetry.read permission. See Available Permissions.

The store keeps two kinds of data:

• A call summary for each module.action span: the module, the action, the duration, the status, the page count, the downstream calls and the error.
• The full span tree of each failed request. The store discards the span tree of a request that did not fail.

A request is failed when one of these conditions is true:

• A Precept span ends with ERROR status. Thus a module action that throws always keeps its trace.
• The local root span ends with ERROR status. For an incoming HTTP request, this occurs when the response status is 500 or more.

ERROR status on Express middleware spans and on outgoing HTTP client spans does not count. As a result, a request that returns 400 to 499 keeps no span tree, unless a module action in it failed.

To share a failed request, open it on the Performance page and click "Download trace". The file is trace-<traceId>.json. To view a shared file, click "Open trace file" on the Performance page. The page shows the trace from the file and does not ask the server for it.

An included relationship that fails puts its error in the response as @precept/error. The request returns status 200, but the store keeps its span tree, because the nested module.action span has ERROR status.

WARNING

Precept does not redact error traces. They can contain downstream response bodies and headers, URLs with query strings, and stack traces. They go to the Performance page and, if OTEL_EXPORTER_OTLP_ENDPOINT is set, to your OTLP backend. The db.statement redaction of database spans still applies.

Limits

The store keeps its data in memory:

• Each server replica has its own store. With two replicas, each page load shows the data of one replica.
• A restart clears all data.
• The store keeps at most 10,000 call summaries and 200 error traces. It discards the oldest entry first.
• All error traces together hold at most 20,000 spans.
• One trace keeps at most 2,000 spans plus its local root span. The store marks a trace with more spans as truncated.
• All open requests together keep at most 50,000 spans. At this limit, the store keeps no new spans and marks the affected traces as truncated.
• The store holds at most 1,000 open requests. At this limit, it discards the span tree of the oldest open request.
• Spans that end after the local root span of their request are not in the error trace.

If the store has no error trace for a trace ID, GET /rest/telemetry/traces/:traceId responds with status 404 and TraceNotFoundError.

Sampling

The store sees only the spans that the SDK records. Precept does not configure a sampler, so the SDK reads OTEL_TRACES_SAMPLER. The default sampler records each request without a parent, and follows the parent decision for the other requests.

The page shows all requests only when both of these conditions are true:

• OTEL_TRACES_SAMPLER is unset, or set to always_on or parentbased_always_on.
• Callers do not send a traceparent header with the sampled flag off.

If one of these conditions is false, the page shows only a sample of the requests.

Worker logs

The Workflow worker writes its pino logs to stdout. If OTEL_EXPORTER_OTLP_ENDPOINT is set, the worker also sends these logs to the OTLP logs endpoint. Temporal SDK logs, activity and workflow logs, and native Temporal Core logs go through the same logger.

Variable	Values	Default	Effect
PRECEPT_LOG_LEVEL	trace, debug, info, warn, error, fatal	info	The lowest level that the worker writes.
PRECEPT_LOG_FORMAT	pretty, json, silent	json	The format on stdout. With silent, the worker writes no logs to stdout or OTLP.
TEMPORAL_CORE_LOG_LEVEL	TRACE, DEBUG, INFO, WARN, ERROR	WARN	The filter for native logs of Temporal Core (temporal_sdk_core, temporal_client, temporal_sdk).
TEMPORAL_OTHER_LOG_LEVEL	TRACE, DEBUG, INFO, WARN, ERROR	ERROR	The filter for native logs of other libraries.

An empty value uses the default. If a value is not in the list, the worker stops at start with an error. A native log must pass its Temporal filter and PRECEPT_LOG_LEVEL.

Example: pointing at the OpenTelemetry Collector

The reference path is to run the OpenTelemetry Collector inside your cluster and forward from there to your eventual backend (Honeycomb, Grafana, Datadog, an on-prem store, etc.). A minimal Collector config that accepts OTLP and re-exports it is:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  # ...your downstream exporter here

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [/* your exporter */]
    logs:
      receivers: [otlp]
      exporters: [/* your exporter */]

Then set the standard OTEL_* environment variables on the server and worker processes. With the Helm chart, use server.env and worker.env (see Helm Chart Values):

server:
  env:
    OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc.cluster.local:4318
    OTEL_RESOURCE_ATTRIBUTES: deployment.environment=production
worker:
  env:
    OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.observability.svc.cluster.local:4318
    OTEL_RESOURCE_ATTRIBUTES: deployment.environment=production

OTEL_SERVICE_NAME defaults to precept-server and precept-worker when unset.

Example: pointing directly at a SaaS

Most observability SaaS vendors accept OTLP/HTTP directly and can substitute for the Collector. Consult your vendor's docs for the endpoint URL and the auth header they require, then set:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.eu.example-vendor.com
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20your-token-here

OTEL_EXPORTER_OTLP_HEADERS follows the standard OTel format: comma-separated key=value pairs, URL-encoded.

Verifying it's working

With OTEL_EXPORTER_OTLP_ENDPOINT set, hit any endpoint on the server and the corresponding span should appear in your collector or backend within a few seconds (spans are batched).

For local debugging, set OTEL_LOG_LEVEL=debug on the server process — the OTel SDK will log every export attempt to stderr, including retries when the collector is unreachable.

Disabling instrumentation

Unset OTEL_EXPORTER_OTLP_ENDPOINT (or remove it from your deployment manifest) and restart the affected pods. The OTel SDK will still load, but no exporters will be registered and no telemetry traffic will be produced.