BranchTrigger: Screenshot OCR

Trigger: Screenshot OCR

Watch for new screenshots, extract text, create searchable notes.

The Schedule

macOS saves screenshots to the Desktop by default (check System Settings → Keyboard → Screenshots):

{"type": "on-change", "paths": ["/Users/YOU/Desktop"]}

The Run Script

Finds screenshots, extracts text with an LLM (or just catalogs without one), writes searchable notes:

#!/usr/bin/env bun
import { readdir, rename, mkdir, stat } from "node:fs/promises";
import { join, resolve } from "node:path";
import { homedir } from "node:os";

const SCREENSHOT_DIR = join(homedir(), "Desktop");
const PROJECT_ROOT = resolve(import.meta.dir, "../..");
const NOTES_DIR = join(PROJECT_ROOT, "workdir", "screenshot-notes");
const PROCESSED_DIR = join(PROJECT_ROOT, "workdir", "screenshots-processed");

function isScreenshot(name: string): boolean {
  return name.startsWith("Screenshot") && /\.(png|jpg)$/i.test(name);
}

const files = (await readdir(SCREENSHOT_DIR)).filter(isScreenshot);
if (files.length === 0) { console.log("[screenshot] No new screenshots"); process.exit(0); }

await mkdir(NOTES_DIR, { recursive: true });
await mkdir(PROCESSED_DIR, { recursive: true });

for (const file of files) {
  const srcPath = join(SCREENSHOT_DIR, file);
  const s = await stat(srcPath);
  if (Date.now() - s.mtimeMs < 5_000) continue; // still writing

  // Try coding agent for image description (claude can read images)
  let description = "";
  const proc = Bun.spawnSync(
    ["claude", "-p", `Describe this screenshot. Extract any visible text (OCR). Be concise.`, srcPath],
    { timeout: 30_000 },
  );
  if (proc.exitCode === 0) {
    description = new TextDecoder().decode(proc.stdout).trim();
  }

  if (!description) {
    description = `Screenshot captured. Size: ${Math.round(s.size / 1024)}KB. No OCR available — install a coding agent that supports images.`;
  }

  const date = file.match(/\d{4}-\d{2}-\d{2}/)?.[0] || new Date().toISOString().split("T")[0];
  const noteName = `${date}-${file.replace(/\.[^.]+$/, "")}.md`;
  await Bun.write(join(NOTES_DIR, noteName), `# ${file}\n\n${description}\n`);
  await rename(srcPath, join(PROCESSED_DIR, file));
  console.log(`[screenshot] ${file} → ${noteName}`);
}

What This Creates

Over time you build a searchable text archive of your screenshots:

workdir/screenshot-notes/
  2026-03-13-Screenshot 2026-03-13 at 14.23.45.md
  2026-03-13-Screenshot 2026-03-13 at 15.01.22.md

Each note contains the LLM's description and any extracted text. Pair this with the daily-digest job and your screenshots become part of your daily summary.

Test It

# Take a screenshot (⌘+Shift+4)
# Wait a few seconds for WatchPaths to fire
bun run src/cli.ts logs screenshot-ocr
ls workdir/screenshot-notes/

Companion Notes

Branch: Screenshot → OCR/Description

Watch your screenshot directory. When a new screenshot appears, extract text (OCR) or generate a description, save as a searchable note.

Schedule

{"type": "on-change", "paths": ["/Users/YOU/Desktop"]}

Or wherever your screenshots land. Check System Settings → Keyboard → Screenshots for the path.

Run Script

#!/usr/bin/env bun
import { readdir, rename, mkdir, stat, readFile } from "node:fs/promises";
import { join, extname, resolve } from "node:path";
import { existsSync } from "node:fs";
import { homedir } from "node:os";

const PROJECT_ROOT = resolve(import.meta.dir, "../..");
const SCREENSHOT_DIR = join(homedir(), "Desktop"); // or wherever screenshots go
const NOTES_DIR = join(PROJECT_ROOT, "workdir", "screenshot-notes");
const PROCESSED_DIR = join(PROJECT_ROOT, "workdir", "screenshots-processed");

function isScreenshot(name: string): boolean {
  // macOS names screenshots like "Screenshot 2026-03-13 at 14.23.45.png"
  return name.startsWith("Screenshot") && (name.endsWith(".png") || name.endsWith(".jpg"));
}

async function describeImage(imagePath: string): Promise<string> {
  // Claude can read images via the coding agent CLI
  const proc = Bun.spawnSync(
    ["claude", "-p", "Describe this screenshot concisely. Extract any visible text (OCR). Format as a searchable note.", imagePath],
    { timeout: 30_000 },
  );

  if (proc.exitCode !== 0) {
    // Fallback: just note the file metadata
    const s = await stat(imagePath);
    return `Screenshot taken. Size: ${Math.round(s.size / 1024)}KB. No description available.`;
  }

  return new TextDecoder().decode(proc.stdout).trim();
}

async function main() {
  const files = await readdir(SCREENSHOT_DIR);
  const screenshots = files.filter(isScreenshot);

  if (screenshots.length === 0) {
    console.log("[screenshot] No new screenshots.");
    return;
  }

  await mkdir(NOTES_DIR, { recursive: true });
  await mkdir(PROCESSED_DIR, { recursive: true });

  for (const file of screenshots) {
    const srcPath = join(SCREENSHOT_DIR, file);
    const s = await stat(srcPath);

    // Skip very recent (still being written)
    if (Date.now() - s.mtimeMs < 5_000) continue;

    console.log(`[screenshot] Processing: ${file}`);

    const description = await describeImage(srcPath);

    // Write searchable note
    const timestamp = file.match(/\d{4}-\d{2}-\d{2}/)?.[0] || new Date().toISOString().split("T")[0];
    const notePath = join(NOTES_DIR, `${timestamp}-${file.replace(/\.[^.]+$/, "")}.md`);
    await Bun.write(notePath, `# ${file}\n\n${description}\n\n---\n*Source: ${file}*\n`);

    // Move screenshot to processed
    await rename(srcPath, join(PROCESSED_DIR, file));

    console.log(`[screenshot] → ${notePath}`);
  }
}

main().catch(e => { console.error("[screenshot]", e); process.exit(1); });

What Makes This Interesting

  • Screenshots become searchable text notes automatically
  • The coding agent CLI can often read images directly
  • Over time you build a visual memory with text search
  • Pairs with the daily-digest job — screenshot notes become part of your daily summary

Without a Coding Agent

If your LLM provider doesn't support images, skip the AI and just organize:

// Move screenshots to dated folders, no AI needed
const date = file.match(/\d{4}-\d{2}-\d{2}/)?.[0] || "undated";
const destDir = join(PROCESSED_DIR, date);
await mkdir(destDir, { recursive: true });
await rename(srcPath, join(destDir, file));

Add OCR later when you set up a provider that handles images.