Bhavana AI

AI/ML insights

Dev Log: January 20, 2026

Podcast Summarizer v2

Continued building out the observability system today, focusing on getting real metrics instrumented throughout the codebase. Revamped the TDD plan after a review exposed gaps: placeholder tests were replaced with full test code, file paths were pinned for every task, and KQL verification steps were added to the workbook tasks. Also added a db_storage_used_mb event for Azure SQL capacity tracking that had been missing from the original design.

The core instrumentation follows a structured logging pattern where all metrics flow through a single emit_metric() function. This function handles JSON serialization, timestamps, and PII scrubbing, then prefixes each line with [METRIC] so KQL can filter metrics from mixed log streams with a simple startswith check. The schema version field (schema_v) baked into each event enables future format changes without breaking existing dashboards.

Instrumented the orchestrator with a try/finally pattern to guarantee that _completed metrics are always emitted, even when errors occur. Counters are initialized to zero before the try block so they hold valid values in every code path. Per-delivery metrics (like summarization_completed and email_sent) would need to live inside the delivery_service.deliver_one() call chain, so for now the job-level start/complete events provide the high-level picture while per-delivery metrics are scoped for a later pass.

Wrapped up with lint cleanup, combining nested with statements into single parenthesized context managers to satisfy the SIM117 rule. Also reflected on the inherent limitation of observability instrumentation: it only captures what happens after deployment, so historical analysis requires either backfilling from existing logs or accepting a fresh baseline.

What was fixed:

  1. Placeholder tests replaced with full TDD - All tasks now have complete test code with assertions, expected failure modes, and verification steps
  2. File paths specified - Every task now identifies exactly which files to create/modify
  3. KQL verification added - Workbook tasks now include sample queries and verification steps to ensure dashboards work correctly
  4. Missing event added - db_storage_used_mb was added to track Azure SQL storage capacity for growth monitoring

TDD pattern used throughout:

  • Step 1: Write failing test with clear assertions
  • Step 2: Run test to verify failure (with expected error)
  • Step 3: Write minimal implementation
  • Step 4: Run test to verify pass
  • Step 5: Run full test suite for regressions
  • Step 6: Commit with descriptive message

emit_metric pattern: The [METRIC] prefix is crucial for KQL filtering in Azure Log Analytics. Without a reliable prefix, extracting metrics from mixed log streams becomes error-prone. The schema version (schema_v) enables future changes without breaking existing dashboards.

Orchestrator metrics pattern: The orchestrator uses a try/finally pattern to guarantee the _completed metric is always emitted, even if errors occur. Counters are initialized to 0 before the try block so they have valid values in all paths.

Per-delivery metrics vs job metrics: The CPU processor’s per-delivery metrics (summarization_completed, email_sent) would need to be added inside the delivery_service.deliver_one() call chain, where the actual LLM summarization and email sending occurs. The job-level metrics (started/completed) provide high-level visibility, while per-delivery metrics would require modifying the delivery service.

Observability pattern summary:

  • All metrics flow through emit_metric() which handles JSON serialization, timestamps, and PII scrubbing
  • The [METRIC] prefix enables simple KQL filtering: where Log_s startswith "[METRIC]"
  • Each job emits _started and _completed events with consistent run_id for correlation
  • Per-item metrics (like transcription_completed) provide granular debugging capability

The SIM117 lint rule enforces combining nested with statements into a single parenthesized context manager. This improves readability and reduces nesting. The pattern with (ctx1, ctx2): is cleaner than with ctx1: with ctx2: and avoids the visual indentation creep.

Observability instrumentation is inherently forward-looking - you can only capture what happens after the code is instrumented. For historical analysis, you’d need to backfill from existing logs (if they contain the relevant data) or accept that the dashboard starts fresh from deployment.

Claude Code Transcripts

Contributed a large PR (#47) to Simon Willison’s claude-code-transcripts tool, adding 10 major UX features to the HTML viewer. The project uses pyproject.toml for packaging and follows a claude/<feature-description>-<random-id> branch naming convention. The PR adds collapsible sections, copy buttons, syntax highlighting via Pygments, tool-specific icons for 14 different tools, ANSI escape code sanitization, a markdown/JSON toggle, message metadata, and tool call pairing.

After the main PR landed, iterated on the collapsible behavior. The original implementation used static defaults (thinking always closed, response always open), but long responses created walls of expanded text. Added an auto-collapse threshold: cells with content over 200 characters start collapsed regardless of type, while shorter content respects the original defaults. This uses the existing open_by_default parameter in the Jinja cell macro, now calculated dynamically based on content length.

Then tackled rendering performance for large transcripts. Added content-visibility: auto to cells so the browser skips rendering offscreen elements entirely, with contain-intrinsic-size providing height estimates to prevent scroll jumps. Built a separate viewer layer that serves the original HTML unchanged but injects CSS for collapsing, 3-line previews, and JS for infinite scroll pagination, all without modifying the upstream codebase.

  • This is Simon Willison’s tool for parsing and working with Claude Code transcript files
  • The naming convention for branches appears to be claude/<feature-description>-<random-id> based on existing remote branches
  • The project uses pyproject.toml for Python packaging (modern standard)

PR #47 adds 10 major UX features:

  1. Collapsible sections - Thinking (collapsed), Response (expanded), Tools (collapsed)
  2. Copy buttons - Per code block and per cell
  3. Syntax highlighting - Pygments integration with 500+ languages
  4. Tool icons - 14 tool-specific icons (Read, Write, Bash, etc.)
  5. ANSI sanitization - Strips terminal escape codes
  6. Markdown/JSON toggle - Tab-style view switcher
  7. Message metadata - Character count, token estimates
  8. Tool call pairing - Groups requests with responses

PR #47 features now active in your HTML:

  • Collapsible thinking/response/tools cells
  • Copy buttons on code blocks and cells
  • Syntax highlighting via Pygments
  • Tool icons (Read, Write, Bash, etc.)
  • Markdown/JSON toggle tabs
  • Message metadata (char count, token estimates)

The cell macro (line 168) already has an open_by_default parameter. Currently it’s set statically (thinking=closed, response=open). We need to make it dynamic based on content length - calculated during HTML generation.

Current behavior:

  • User messages: always open
  • Thinking: always closed
  • Response: always open
  • Tools: always closed

New behavior:

  • If content > 200 chars → always collapsed
  • If content ≤ 200 chars → use original default

What changed:

  • Added AUTO_COLLAPSE_THRESHOLD = 200 constant
  • Added should_cell_be_open() helper function
  • All cell types (user, thinking, response, tools) now auto-collapse when content > 200 chars
  • Short content respects original defaults (response open, thinking/tools closed)

The UX improvement:

  • Before: Scrolling through walls of expanded text
  • After: Collapsed cells show the structure, click to expand details

What content-visibility: auto does:

  • Browser skips rendering offscreen elements entirely
  • Only visible cells are laid out and painted
  • Scrolling triggers rendering just-in-time
  • contain-intrinsic-size provides height estimate to prevent scroll jumps

What the viewer does:

  • Serves original HTML from Python tool unchanged
  • Injects CSS for: collapsing all cells, 3-line previews, content-visibility performance
  • Injects JS for: extracting preview text, infinite scroll (loads next page on scroll)
  • No modifications to original claude-code-transcripts codebase

Courses: Stanford CS244C (Gavel)

Dug into the Philly cluster traces to understand how Gavel transforms real-world data into simulation inputs. The key discovery: Gavel’s “Philly” traces are synthetic, not raw Microsoft data. The official traces are JSON job logs with timing and GPU allocations but no workload type information, while Gavel’s traces are TSV files with synthetic DL workloads (ResNet, Transformer, etc.) mapped to profiled throughputs. Gavel preserves the organizational structure (15 virtual clusters) but generates entirely synthetic jobs.

Analyzed the raw Philly trace statistics: 117K total jobs, 72% successful, 82% single-GPU. The duration distribution is heavily skewed with a median of 18 minutes but a mean of 3.2 hours and a maximum of 53 days. These distributions inform the heuristics for mapping real jobs to synthetic workloads.

Worked through the conversion pipeline architecture: Philly JSON provides job timing and GPU counts, mapping heuristics assign Gavel workloads based on duration (short jobs map to small batches, long jobs to larger models), and throughput profiles from simulation_throughputs.json convert duration to iteration counts. The resulting 12% cluster utilization under FIFO is expected since FIFO does not optimize packing and jobs vary widely in size.

Key Discovery: Gavel’s “Philly” traces are synthetic, not raw Microsoft data.

  • Real Philly traces: JSON logs with job status/timing but no workload types
  • Gavel traces: TSV with synthetic DL workloads (ResNet, Transformer, etc.) mapped to profiled throughputs
  • Gavel preserves the organizational structure (15 virtual clusters) but generates synthetic jobs

Philly Trace Statistics:

  • 117K jobs total, 72% successful (Pass)
  • 82% single-GPU, then 8-GPU (5%), 4-GPU (5%), 2-GPU (2%)
  • Median duration: 18 min, Mean: 3.2 hours, Max: 53 days
  • These distributions inform our workload mapping heuristics

Conversion Pipeline Architecture:

  1. Philly JSON has job timing (submitted, start, end) and GPU counts, but no workload types
  2. Mapping heuristics assign Gavel workloads based on duration (short jobs → small batches, long jobs → larger models)
  3. Throughput profiles from simulation_throughputs.json convert duration → iterations
  4. The low utilization (12%) is expected since FIFO doesn’t optimize packing and jobs have varying sizes