Add Observability

You need to log what a service does. You also need to trace operations across service boundaries. But you do not want to configure a logging framework, pick a format, or wire up an export pipeline. @forwardimpact/libtelemetry provides three tools that work without setup. A Logger produces RFC 5424-formatted lines. A Tracer records spans to a span service. An Observer unifies both for gRPC operations. This page covers one bounded task. It shows you how to add observability to a service. For the full lifecycle setup, see Service Lifecycle.

Prerequisites

• Node.js 22+
• Install the library:

npm install @forwardimpact/libtelemetry

Add a log line

Create a logger with a domain name and call info, error, or debug:

import { createLogger } from "@forwardimpact/libtelemetry";
import { createDefaultRuntime } from "@forwardimpact/libutil/runtime";

const logger = createLogger("my-service", createDefaultRuntime());

logger.info("startup", "Server listening", { port: "3000" });

Expected output on stderr:

INFO 2026-05-04T10:00:00.000Z my-service startup 42001 MSG001 [port="3000"] Server listening

The format follows RFC 5424:

LEVEL TIMESTAMP DOMAIN APP_ID PROC_ID MSG_ID [ATTRIBUTES] MESSAGE

A space separates each field, so you can grep the lines. Attributes appear as key-value pairs inside square brackets. When you provide no attributes, the field is a single dash (-).

Log levels

Control which methods print with the LOG_LEVEL environment variable:

LOG_LEVEL	Methods that print
error	error, exception
info	error, exception, info (default)
debug	all methods

Domain-scoped debug output

Enable debug output for specific domains without a change to the global level:

DEBUG=my-service node server.js

Use comma-separated patterns and wildcards:

DEBUG=my-service,grpc:* node server.js

Use DEBUG=* to enable debug output for all domains.

Log errors

Use logger.exception for caught errors. It logs the message at all levels. It appends the stack trace when you enable debug output:

logger.exception("db", err, { host: "localhost" });

Add a span

The Tracer requires a span service client and a gRPC metadata constructor. After you configure it, one call creates a span:

import { Tracer } from "@forwardimpact/libtelemetry/tracer.js";

const tracer = new Tracer({
  serviceName: "my-service",
  spanClient,      // gRPC client for the span service
  grpcMetadata,     // gRPC Metadata constructor
});

const span = tracer.startSpan("processRequest", {
  kind: "SERVER",
  attributes: { endpoint: "/api/data" },
});

try {
  const result = await handleRequest();
  span.addEvent("processing_complete", { items: String(result.count) });
  span.setOk();
} catch (err) {
  span.setError(err);
  throw err;
} finally {
  await span.end();
}

Trace context propagation

When one service calls another, use startClientSpan for outgoing calls. It returns both the span and the populated metadata:

const { span, metadata } = tracer.startClientSpan("Vector", "QueryItems", {
  resource_id: "doc-123",
});

try {
  const response = await vectorClient.queryItems(request, metadata);
  span.setOk();
} catch (err) {
  span.setError(err);
  throw err;
} finally {
  await span.end();
}

For incoming calls, startServerSpan extracts trace context from the request metadata:

const span = tracer.startServerSpan(
  "Agent",
  "ProcessStream",
  call.request,
  call.metadata,
);

Observe gRPC operations

The Observer class unifies logging and tracing for gRPC handlers:

import { createObserver, createLogger } from "@forwardimpact/libtelemetry";
import { createDefaultRuntime } from "@forwardimpact/libutil/runtime";

const logger = createLogger("agent", createDefaultRuntime());
const observer = createObserver("Agent", logger, tracer);

Observe a server-side unary call:

const response = await observer.observeServerUnaryCall(
  "ProcessRequest",
  call,
  async (call) => {
    // Your business logic here
    return { result: "done" };
  },
);

The observer:

1. Logs the incoming request at debug level.
2. Starts a SERVER span with trace context from gRPC metadata.
3. Runs your handler in the span context so the parent propagates automatically.
4. Logs the response and sets the span status to OK.
5. On error, logs the exception, sets the span status to ERROR, and enriches the error object with trace_id and span_id for correlation.

The same pattern works for streaming calls (observeServerStreamingCall), outgoing unary calls (observeClientUnaryCall), and outgoing streaming calls (observeClientStreamingCall).

When you do not configure a tracer, the observer falls back to logging only. It creates no spans, and the gRPC calls proceed without trace context. So you can add the observer first and wire up tracing later. Your handler code stays the same.

Query and visualize recorded spans

After spans flow into the span index, fit-visualize reads them back. It renders them as Mermaid sequence diagrams. It is a filter-and-query tool. Pipe a JMESPath expression on stdin to select spans. The tool then emits a diagram of the service interactions in those traces.

echo "[?name=='ProcessStream']" | npx fit-visualize

Pass an empty list expression to select every span. Then narrow the result with a filter flag:

echo "[]" | npx fit-visualize --trace 0f53069dbc62d

Flag	Effect
--trace	Restrict to spans whose trace ID matches.
--resource	Restrict to spans whose resource ID matches.

The JMESPath expression and the flags compose. The expression filters span fields (name, kind, attributes). The flags scope the query to a single trace or resource. For example, select only client spans for one resource:

echo "[?kind==\`2\`]" | npx fit-visualize --resource common.Conversation.abc123

When you set --resource, the tool combines every trace that matches into one diagram and titles it by resource ID. A note marks each trace boundary. This helps you follow one conversation across several requests. Without --resource, each trace renders as its own diagram titled by trace ID.

The output is a fenced Mermaid block ready to paste into any Markdown renderer:

sequenceDiagram
    title Trace: 0f53069dbc62d
    participant cli
    participant agent
    cli->>+agent: ProcessStream (time=2026-05-04T10:00:00.000Z)
    agent-->>-cli: OK

When no spans match the filter, the command prints No spans found matching the filter criteria. instead of a diagram. An empty result is then unambiguous.

What's next