cavendish
Wet labOurs in the planKeeping it runningType 4Pressure

What the assistant is worth

A Nightingale measurement run on coding-assistant ROI, with the counterfactual actually measured

ROI
  1. proposed
  2. voting
  3. running · 122d
  4. measuring
  5. concluded

Preregistration · v2 · 20 Apr 2026 · immutable after start

An extension creates a new version rather than editing the old.

Hypothesis

Telemetry-derived cycle-time and rework measures on 84 licensed developers show a return on the coding-assistant licence of at least 2× annual cost, and self-reported time saved does not predict which developers' measures moved.

Kill condition

Kill the survey arm if its correlation with any telemetry series is under 0.3 after ten weeks. Kill the run if telemetry attribution covers fewer than 70% of developer-weeks by week six.

MethodTwo arms on the same 84 developers: fortnightly survey (existing) and telemetry (PR cycle time, review turnaround, rework rate, commits per developer-week) against a baseline quarter before licences. Twenty weeks. Commercial rationale: the licence renewal decision in Q4 and three client requests for the method.
Expected cost$3,000 in telemetry infra, three people part-time for five months
Expected duration5 months
Nothing is written up before a run concludes with a result. This one is running.

Running notes · fed from harness sessions and by the pair

AW
manual · Adam Witanowski · 3 Sep 2026· ⌘↩ to post
  1. harness25 Aug 2026MM Michael Menache

    week 16: median cycle time -9%; top quartile +4% (more agent-code review); rework -6%

  2. manual14 Jul 2026AT Andrew Tran

    Survey arm killed on the kill condition: correlation 0.11 with cycle-time change. Median over-statement 3.4×. Logged as g-roi-survey-method; run continues on telemetry.

  3. harness18 May 2026AT Andrew Tran

    telemetry pipeline: 84 devs, 4 series, baseline quarter loaded; attribution 91% dev-weeks

  4. manual20 Apr 2026AT Andrew Tran

    Type 4, signed by Adam with the renewal as the commercial rationale. Predicted: telemetry shows 1.5× not 2×, survey arm dies, confidence 0.6.

Harness notes are auto-captured from Claude Code sessions: model, date, commit, session reference. Never the transcript, code or paths.

Artifacts

reponightingale-telemetryrepo
measFour telemetry series × 84 developers × 16 weeksmeasurement
noteSurvey vs telemetry correlation (killed arm)notebook