wizardshit.ai
Navigation
Lesson

Observability

Structured logging so the claw can see itself. Every interaction, cost, error, and restart in queryable JSONL. The claw reads its own logs and self-reports.

Something broke and you don't know what. The claw stopped responding for an hour. Was it a crash? A stale session? Did the memory analyzer fail? You tail the log file and see walls of unstructured text. Good luck.

This lesson gives the claw eyes on itself. Every interaction, gather cycle, memory analysis, error, and restart becomes a structured JSON line with enough dimensions to answer any question about the system's behavior.

Two consumers read these logs. The claw reads its own activity summary before every response and proactively flags problems. The operator queries with jq when something feels off.

The logger

One file. Three functions.

import { appendFileSync, readFileSync, existsSync, mkdirSync } from "node:fs";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";

const __dirname = dirname(fileURLToPath(import.meta.url));
const LOG_DIR = join(__dirname, ".claw", "logs");
const LOG_FILE = join(LOG_DIR, "claw.jsonl");

mkdirSync(LOG_DIR, { recursive: true });

export function log(type, data = {}) {
  const entry = {
    ts: Date.now(),
    t: new Date().toISOString(),
    type,
    ...data,
  };
  appendFileSync(LOG_FILE, JSON.stringify(entry) + "\n");
}

log() appends one JSON line. Every entry gets a millisecond timestamp (ts for querying) and an ISO string (t for reading). The type field is the event category. Everything else is high-cardinality data specific to that event.

export function query({ hours = 24, type, chatId, limit = 100 } = {}) {
  if (!existsSync(LOG_FILE)) return [];
  const cutoff = Date.now() - hours * 60 * 60 * 1000;
  const lines = readFileSync(LOG_FILE, "utf-8").trim().split("\n");
  const results = [];

  for (let i = lines.length - 1; i >= 0 && results.length < limit; i--) {
    try {
      const entry = JSON.parse(lines[i]);
      if (entry.ts < cutoff) break;
      if (type && entry.type !== type) continue;
      if (chatId && entry.chatId !== chatId) continue;
      results.push(entry);
    } catch { continue; }
  }
  return results.reverse();
}

query() reads from the end of the file for recency. Filters by time window, event type, and chat ID. Returns chronological order.

export function summarize(hours = 24) {
  const entries = query({ hours, limit: 1000 });
  if (entries.length === 0) return "No activity in the last " + hours + " hours.";

  const interactions = entries.filter(e => e.type === "interaction");
  const errors = entries.filter(e => e.type === "error");
  const gathers = entries.filter(e => e.type === "gather");
  const memoryRuns = entries.filter(e => e.type === "memory");
  const restarts = entries.filter(e => e.type === "restart");
  const totalCost = entries.reduce((sum, e) => sum + (e.cost || 0), 0);

  const lines = [`Activity summary (last ${hours}h):`];

  if (interactions.length > 0) {
    const avgDuration = interactions.reduce((s, e) => s + (e.duration || 0), 0) / interactions.length;
    lines.push(`- ${interactions.length} interactions, avg ${(avgDuration / 1000).toFixed(1)}s`);
  }
  if (gathers.length > 0) lines.push(`- ${gathers.length} gather cycles`);
  if (memoryRuns.length > 0) lines.push(`- ${memoryRuns.length} memory analyses`);
  if (restarts.length > 0) lines.push(`- ${restarts.length} restarts`);
  if (errors.length > 0) lines.push(`- ${errors.length} ERRORS: ${errors.map(e => e.message).join("; ")}`);
  lines.push(`- Total cost: $${totalCost.toFixed(4)}`);

  return lines.join("\n");
}

summarize() produces a concise markdown summary. Example output:

Activity summary (last 24h):
- 12 interactions, avg 4.2s
- 4 gather cycles
- 2 memory analyses
- Total cost: $0.8400

The claw reads this before every response. If there are errors, they show up first so the claw can flag them proactively.

High cardinality fields

Every event type logs different dimensions. This is what makes the logs useful.

Interactions:

{"ts":1712345678,"t":"2026-04-05T15:00:00Z","type":"interaction","chatId":"telegram:123","prompt":"what's my week","cost":0.0425,"duration":5200,"turns":3,"model":"claude-opus-4-6","sessionResumed":true}

Gather cycles:

{"ts":1712345678,"t":"2026-04-05T15:00:00Z","type":"gather","source":"live","emails":25,"events":3,"cost":0.02,"duration":4500,"model":"claude-haiku-4-5","turns":2}

Memory analysis:

{"ts":1712345678,"t":"2026-04-05T15:00:00Z","type":"memory","analyzed":6,"updated":true,"cost":0.03}

Errors:

{"ts":1712345678,"t":"2026-04-05T15:00:00Z","type":"error","component":"claw","message":"stale session, retrying","chatId":"telegram:123"}

Boot/restart:

{"ts":1712345678,"t":"2026-04-05T15:00:00Z","type":"boot","event":"restart","adapter":"telegram"}

With these dimensions you can answer: "How much did Haiku cost me this week?" Filter by type=gather, sum cost. "Why was the claw slow yesterday?" Filter by type=interaction, sort by duration. "Which chat generates the most traffic?" Group by chatId.

Wiring it in

Every component imports log and writes structured entries at key moments:

claw.mjs logs every interaction with cost, duration, model, and turn count. Logs errors on failure. Logs stale session recovery.

channel.mjs logs boot events (first start vs restart), owner chat detection, and restart notifications.

gather.mjs logs every gather cycle with source (fixture/live), data counts, cost, duration, and model.

memory.mjs logs every analysis cycle with interaction count analyzed, whether memories were updated, and cost.

The claw reads its own logs

The activity summary gets injected as context before every response:

import { summarize } from "./log.mjs";

// In the ask() function, after memory and digest:
const activitySummary = summarize(24);
if (activitySummary) {
  context += `[SYSTEM ACTIVITY]\n\n${activitySummary}\n\n`;
}

Now when the operator asks "what happened while I was away?" the claw already has the answer. It doesn't need to read files or run commands. The summary is in its context.

Update CLAUDE.md so the claw knows to use this:

## Observability

You have structured logs prepended as [SYSTEM ACTIVITY]. This is a summary of
your own recent activity: interactions, costs, errors, restarts, gather cycles,
memory analysis runs.

When the operator asks about activity, costs, or errors, answer from this summary.
If you see errors, flag them proactively.

Operator queries

Add three new commands to run.sh for slicing the JSONL from the host:

  logs)
    HOURS="${2:-24}"
    cat .claw/logs/claw.jsonl | jq -s --argjson cutoff "$(( $(date +%s) - HOURS * 3600 ))000" \
      '[.[] | select(.ts > $cutoff)] | sort_by(.ts)' 2>/dev/null || echo "No logs yet"
    ;;
  costs)
    cat .claw/logs/claw.jsonl | jq -s '
      group_by(.type) | map({
        type: .[0].type,
        count: length,
        total_cost: (map(.cost // 0) | add)
      }) | sort_by(-.total_cost)
    ' 2>/dev/null || echo "No logs yet"
    ;;
  errors)
    cat .claw/logs/claw.jsonl | jq -s '[.[] | select(.type == "error")] | sort_by(.ts) | .[-20:]' \
      2>/dev/null || echo "No errors"
    ;;

Usage:

./run.sh logs          # last 24 hours
./run.sh logs 48       # last 48 hours
./run.sh costs         # cost breakdown by type
./run.sh errors        # recent errors

These pipe through jq. The costs command groups by type and sums:

cat .claw/logs/claw.jsonl | jq -s '
  group_by(.type) | map({
    type: .[0].type,
    count: length,
    total_cost: (map(.cost // 0) | add)
  }) | sort_by(-.total_cost)
'

Output:

[
  {"type": "interaction", "count": 42, "total_cost": 1.89},
  {"type": "gather", "count": 12, "total_cost": 0.24},
  {"type": "memory", "count": 4, "total_cost": 0.12}
]

You spent $1.89 on conversations, $0.24 on background gathering, $0.12 on memory analysis. That's $2.25 for a day of having a personal familiar. The breakdown tells you if any component is unexpectedly expensive.

The logs skill

Same skill pattern. The claw should know how to query its own observability data:

# Observability

You have structured logs in `.claw/logs/claw.jsonl`. Every interaction,
gather cycle, memory analysis, error, and restart is recorded as a JSON
line with high-cardinality fields.

## Commands

- `./run.sh logs [hours]` — recent activity (default: 24h)
- `./run.sh costs` — cost breakdown by event type
- `./run.sh errors` — recent errors

## Context

You also receive a [SYSTEM ACTIVITY] summary in your context before every
response. Answer from that first. Use the commands above for deeper queries.

## When to suggest

- If the operator asks "what happened while I was away?"
- If the operator asks about costs or spending
- If you see errors in your activity summary, flag them proactively
- If the operator asks about system health

What you have

A claw that can see itself. Structured JSONL with high-cardinality fields on every event. The claw reads a 24-hour summary before each response and proactively flags problems. The operator queries with jq for deeper analysis.

Silent failures are bugs. If the gatherer fails, the log captures it. If a session goes stale, the log captures it. If costs spike, the log shows which component and which model.

What's missing

The claw runs, remembers, has a soul, survives restarts, and can see itself. But it only acts when you ask. You have to remember to check for important emails. You have to remember to ask for a weekly summary. Next: make the claw act on its own schedule.

Previous lesson

Durability

Next lesson

Scheduling