Project Automation Verification
Run — 2026-07-22
Success Criteria
| # | Criterion | Measured by | Status |
|---|---|---|---|
| V1 | Every project skill, agent prompt, template, and lifecycle doc is audited and all gaps documented | Complete gap list with file references | 🟢 Success |
| V2 | The lifecycle is formalized as Create → Requirements → Requirements_Eval → Research → Spec → Spec_Eval → Plan → Plan_Eval → Execute → Eval → Verify → Extract → Extract_Eval → Complete | Lifecycle string matches across canonical files and routing follows it | 🟢 Success |
| V3 | Every review/evaluation phase writes to a durable file artifact | Erik_Todo links and project docs reference saved report files | 🟢 Success |
| V4 | Every skill procedure has all required steps with no missing build/commit/push/sidebar/report updates | Skill-by-skill verification checklist passes | 🟢 Success |
| V5 | A project can move through the full lifecycle via agent handoffs without manual intervention between phases except at explicit human gates | End-to-end dry run succeeds | 🟢 Success |
| V6 | Human gates exist at all required points and the system stops at each one | Gate checklist matches workflow | 🟢 Success |
| V7 | Pending human items are surfaced clearly and the human can review and approve/reject using linked project artifacts | Erik_Todo entries link to stable files | 🟢 Success |
| V8 | User Guide is complete, accurate, and usable by a human unfamiliar with the system | Independent readability review | 🟢 Success |
| V9 | Forge Optimizer can be invoked on project agent executions and findings lead to implemented improvements | At least one Optimizer run on a project agent produces actionable, implemented changes | 🟢 Success |
| V10 | Deferred model-selection work is explicitly documented while required data/reporting foundations are preserved | Requirements, Research, and Spec all state the deferment and the preserved foundations | 🟢 Success |
| V11 | Agent Run Summary can be generated for any project, listing all conversations with model, message, duration, tokens per turn and total, cost per conversation, and total project cost | Summary output matches schema; totals are accurate | 🟢 Success |
| V12 | Project_Create blocks duplicate names across all departments, including Archived projects | Conflict test fails cleanly before any files are created | 🟢 Success |
Verification Details
V1 — Project_Automation_Audit_Results.md documents a complete audit covering lifecycle docs, project skills, agent prompts, snippets, templates, routing, report persistence, and human-gate behavior. Five gaps (A1–A5) were found and all were fixed. No remaining open issues.
V2 — The 14-phase lifecycle string matches identically across: Project_Flow/SKILL.md, Project_Snippet.md, Forge_Chat_Prompt.md, Project_Runner_Prompt.md, Forge_Project_Flow.md, and Project_Automation_Guide.md. Routing tables in both Runner prompts follow the lifecycle.
V3 — All 8 required persisted artifact types have corresponding skills that write to durable files: Requirements_Eval → {Project}_Requirements_Eval.md, Spec_Eval → {Project}_Spec_Eval.md, Plan_Eval → {Project}_Plan_Eval.md, Eval → {Project}_Eval_{N}.md, Verify → {Project}_Verification.md, Extract_Eval → {Project}_Extract_Eval.md, Audit → {Project}_Audit_Results.md, Runs → {Project}_Runs.md. Templates exist for all report types. Each eval skill includes explicit file write and sidebar steps.
V4 — All project skills include: phase update, build/commit/push, sidebar entry via Manage_Pages, and report persistence where applicable. Verified across: Project_Create, Project_Requirements, Project_Requirements_Eval, Project_Spec_Eval, Project_Plan_Eval, Project_Eval, Project_Verify, Project_Extract, Project_Extract_Eval, Project_Execute, Project_Complete.
V5 — Phase routing tables in Project_Runner_Prompt.md and Task_Runner_Prompt.md are consistent with the lifecycle. The Phase file is the single state signal. Project Runner reads it and dispatches the correct agent. No manual intervention needed between phases except at human gates.
V6 — Human gates are present at: after Requirements (before Requirements_Eval), after Requirements_Eval, after each Research decision, after Spec (before Spec_Eval), after Spec_Eval, after Plan (before Plan_Eval), after Plan_Eval, after each Eval, and after each Verify run. The system stops at all these gates.
V7 — Erik_Todo format with 🚫 items, project name, description, and link to the relevant report file. Project_Snippet and Env_Snippet both reinforce the Report-File Rule. Blocked phases are escalated with links to the specific evaluation report.
V8 — Forge/Project_Automation_Guide.md covers all 7 required sections: overview, getting started, detailed phase descriptions, agent roles, human gates, document and report reference, and monitoring and steering. All 14 phases are described including the 4 eval phases.
V9 — Forge_Optimizer supports project mode via --project flag. generate-runs.ts script generates {Project}_Runs.md. Project_Eval appends an Optimizer reminder to Erik_Todo after each Eval passes. The Optimizer uses persisted report files as context.
V10 — Requirements R10 explicitly defers model selection. Spec Section 10 states the deferral and lists the preserved foundations (durable reports, project-level run tracking, stable review links, project/phase metadata). All foundations are implemented.
V11 — Fixed. generate-runs.ts now outputs per-turn token breakdown (Turn/Role/Tokens table per conversation) plus split Input/Output/Total columns in the summary table. Template updated to match.
V12 — Project_Create step 1 runs a find command checking both */Projects/* and */Archived/* paths across the entire repository. The command correctly uses -o to check both locations. Conflicts stop creation with a clear message before any files are created.
Items
| # | Type | Description | Status | Disposition |
|---|---|---|---|---|
| E1 | Gap | Agent Run Summary (generate-runs.ts) outputs total tokens per conversation but does not include per-turn token breakdown as required by R11 (“token usage per turn and in total”). The output table has a single Tokens column (input + output combined). Requirements and Spec both specify per-turn token usage. | 🟢 Resolved | Fix |
Resolution Log
| # | Disposition | Action taken | Resolved |
|---|---|---|---|
| E1 | Fix | Added TurnDetail interface, per-turn breakdown section to output, split Tokens into Input/Output/Total columns in summary table. Updated Runs_Template.md to match. | 🟢 |
Run — 2026-07-25
Success Criteria
| # | Criterion | Measured by | Status |
|---|---|---|---|
| V1 | Every project skill, agent prompt, template, and lifecycle doc is audited and all gaps documented | Complete gap list with file references | 🟢 Success |
| V2 | The lifecycle is formalized as Create → Vision → Vision_Eval → Research → Spec → Spec_Eval → Plan → Plan_Eval → Execute → Execute_Eval → Verify → Extract → Extract_Eval → Complete | Lifecycle string matches across canonical files and routing follows it | 🟢 Success |
| V3 | Every review/evaluation phase writes to a durable file artifact | Erik_Todo links and project docs reference saved report files | 🟢 Success |
| V4 | Every skill procedure has all required steps with no missing build/commit/push/sidebar/report updates | Skill-by-skill verification checklist passes | 🟢 Success |
| V5 | A project can move through the full lifecycle via agent handoffs without manual intervention between phases except at explicit human gates | End-to-end dry run succeeds | 🟢 Success |
| V6 | Human gates exist at all required points and the system stops at each one | Gate checklist matches workflow | 🟢 Success |
| V7 | Pending human items are surfaced clearly and the human can review and approve/reject using linked project artifacts | Erik_Todo entries link to stable files | 🟢 Success |
| V8 | User Guide is complete, accurate, and usable by a human unfamiliar with the system | Independent readability review | 🟢 Success |
| V9 | Forge Optimizer can be invoked on project agent executions and findings lead to implemented improvements | At least one Optimizer run on a project agent produces actionable, implemented changes | 🟢 Success |
| V10 | Deferred model-selection work is explicitly documented while required data/reporting foundations are preserved | Vision, Research, and Spec all state the deferment and the preserved foundations | 🟢 Success |
| V11 | Agent Run Summary can be generated for any project, listing all conversations with model, message, duration, tokens per turn and total, cost per conversation, and total project cost | Summary output matches schema; totals are accurate | 🟢 Success |
| V12 | Project_Create blocks duplicate names across all departments, including Archived projects | Conflict test fails cleanly before any files are created | 🟢 Success |
Verification Details
V1 — Project_Automation_Audit_Results.md documents a complete audit with 5 gaps (A1–A5), all fixed and verified by build. No remaining open issues.
V2 — The 14-phase lifecycle string matches identically across: Project_Flow/SKILL.md, Project_Snippet.md, Forge_Chat_Prompt.md, Project_Runner_Prompt.md, and Forge_Project_User_Guide.md. The Vision document had “Execute → Eval” instead of “Execute → Execute_Eval” — fixed (see E1). Routing tables in both Runner prompts follow the lifecycle.
V3 — All 8 required persisted artifact types have corresponding skills that write to durable files: Vision_Eval → {Project}_Vision_Eval.md, Spec_Eval → {Project}_Spec_Eval.md, Plan_Eval → {Project}_Plan_Eval.md, Execute_Eval → {Project}_Execute_Eval_{N}.md, Verify → {Project}_Verification.md, Extract_Eval → {Project}_Extract_Eval.md, Audit → {Project}_Audit_Results.md, Runs → {Project}_Runs.md. Templates exist for all report types. Each eval skill includes explicit file write and sidebar steps.
V4 — All 14 project skills include: phase update, build/commit/push, sidebar entry via Manage_Pages, and report persistence where applicable. Verified across: Project_Create, Project_Vision, Project_Vision_Eval, Project_Research, Project_Spec, Project_Spec_Eval, Project_Plan, Project_Plan_Eval, Project_Execute, Project_Execute_Eval, Project_Verify, Project_Extract, Project_Extract_Eval, Project_Complete.
V5 — Phase routing tables in Project_Runner_Prompt.md and Task_Runner_Prompt.md are consistent with the lifecycle. The Phase file is the single state signal. Project Runner reads it and dispatches the correct agent. No manual intervention needed between phases except at human gates.
V6 — Human gates are present at: after Vision_Eval convergence, after each Research decision, after Spec_Eval convergence, after Plan_Eval convergence (approval required before Execute), after Execute_Eval if auto-fix doesn’t converge, after Verify, and after Extract_Eval if issues found. The routing table in Project_Runner_Prompt.md correctly identifies all human gates.
V7 — Erik_Todo format with 🚫 items, project name, description, and link to the relevant report file. Project_Snippet and Env_Snippet both reinforce the Report-File Rule. Blocked phases are escalated with links to the specific evaluation report. Preference-based disposition is documented in Project_Snippet and Project_Runner_Prompt.md.
V8 — Forge/Forge_Project_User_Guide.md covers all 7 required sections: overview, getting started, detailed phase descriptions, agent roles, human gates, document and report reference, and monitoring and steering. All 14 phases are described including the eval phases. A companion Forge/Forge_Agents_User_Guide.md covers the agents system. Spec Section 6 referenced Forge/Project_Automation_Guide.md — updated to match actual location (see E2).
V9 — Forge_Optimizer supports project mode via --project flag. generate-runs.ts script generates {Project}_Runs.md. Project_Execute_Eval appends an Optimizer reminder (💡) to Erik_Todo after each Eval passes. The Optimizer uses persisted report files as context.
V10 — Vision R10 explicitly defers model selection. Spec Section 10 states the deferral and lists the preserved foundations (durable reports, project-level run tracking, stable review links, project/phase metadata). All foundations are implemented.
V11 — generate-runs.ts outputs per-conversation summary with agent, model, task, duration, Input/Output/Total token columns, and cost. Per-turn breakdown section includes Turn/Role/Tokens per conversation. Total project cost is computed. Verified with live output for this project (6 conversations, $0.22 total).
V12 — Project_Create step 1 runs a find command checking both */Projects/* and */Archived/* paths across the entire repository. The command correctly uses -o to check both locations. Conflicts stop creation with a clear message before any files are created.
Items
| # | Type | Description | Status | Disposition |
|---|---|---|---|---|
| E1 | Defect | Vision document R2 lifecycle string uses “Execute → Eval” instead of “Execute → Execute_Eval”. The Spec and all implementation files (Project_Flow, Project_Snippet, Forge_Chat_Prompt, Project_Runner_Prompt) correctly use “Execute_Eval”. | 🟢 Resolved | Fix |
| E2 | Variance | Spec Section 6 says the User Guide should be at Forge/Project_Automation_Guide.md, but the actual guide is at Forge/Forge_Project_User_Guide.md (with a companion Forge/Forge_Agents_User_Guide.md). The actual name is more appropriate as a permanent document since it’s not project-specific. | 🟢 Resolved | Update |
Resolution Log
| # | Disposition | Action taken | Resolved |
|---|---|---|---|
| E1 | Fix | Updated Vision R2 lifecycle string from “Execute → Eval” to “Execute → Execute_Eval” to match the Spec and all implementation files. | 🟢 |
| E2 | Update | Updated Spec Section 6 to reference Forge/Forge_Project_User_Guide.md instead of Forge/Project_Automation_Guide.md, matching the actual extraction location. | 🟢 |
Run — 2026-07-26
Re-verification including Plan 12 contributions (stale Inngest dist fix, Agent Handoff edges, Forge Chat routing rules).
Success Criteria
| # | Criterion | Measured by | Status |
|---|---|---|---|
| V1 | Every project skill, agent prompt, template, and lifecycle doc is audited and all gaps documented | Complete gap list with file references | 🟢 Success |
| V2 | The lifecycle is formalized as Create → Vision → Vision_Eval → Research → Spec → Spec_Eval → Plan → Plan_Eval → Execute → Execute_Eval → Verify → Extract → Extract_Eval → Complete | Lifecycle string matches across canonical files and routing follows it | 🟢 Success |
| V3 | Every review/evaluation phase writes to a durable file artifact | Erik_Todo links and project docs reference saved report files | 🟢 Success |
| V4 | Every skill procedure has all required steps with no missing build/commit/push/sidebar/report updates | Skill-by-skill verification checklist passes | 🟢 Success |
| V5 | A project can move through the full lifecycle via agent handoffs without manual intervention between phases except at explicit human gates | End-to-end dry run succeeds | 🟢 Success |
| V6 | Human gates exist at all required points and the system stops at each one | Gate checklist matches workflow | 🟢 Success |
| V7 | Pending human items are surfaced clearly and the human can review and approve/reject using linked project artifacts | Erik_Todo entries link to stable files | 🟢 Success |
| V8 | User Guide is complete, accurate, and usable by a human unfamiliar with the system | Independent readability review | 🟢 Success |
| V9 | Forge Optimizer can be invoked on project agent executions and findings lead to implemented improvements | At least one Optimizer run on a project agent produces actionable, implemented changes | 🟢 Success |
| V10 | Deferred model-selection work is explicitly documented while required data/reporting foundations are preserved | Vision, Research, and Spec all state the deferment and the preserved foundations | 🟢 Success |
| V11 | Agent Run Summary can be generated for any project, listing all conversations with model, message, duration, tokens per turn and total, cost per conversation, and total project cost | Summary output matches schema; totals are accurate | 🟢 Success |
| V12 | Project_Create blocks duplicate names across all departments, including Archived projects | Conflict test fails cleanly before any files are created | 🟢 Success |
Verification Details
V1 — Project_Automation_Audit_Results.md documents a complete audit with 5 gaps (A1–A5), all fixed. Plan 12 did not introduce new gaps. Audit findings are consistent with the current system state.
V2 — The 14-phase lifecycle string Create → Vision → Vision_Eval → Research → Spec → Spec_Eval → Plan → Plan_Eval → Execute → Execute_Eval → Verify → Extract → Extract_Eval → Complete matches identically across all 5 canonical files: Project_Flow/SKILL.md, Project_Snippet.md, Forge_Chat_Prompt.md, Project_Runner_Prompt.md, Forge_Project_User_Guide.md. The Vision V2 criterion text previously had “Execute → Eval” — fixed during this run (see E1). Routing tables in both Runner prompts follow the lifecycle.
V3 — All 8 required persisted artifact types have corresponding skills that write to durable files: Vision_Eval, Spec_Eval, Plan_Eval, Execute_Eval (per-plan), Verification, Extract_Eval, Audit_Results, Runs. Templates exist for all report types. Each eval skill includes explicit file-write and sidebar steps.
V4 — All 14 project skills include: phase update, build/commit/push, sidebar entry via Manage_Pages, and report persistence where applicable. Verified across: Project_Create, Project_Vision, Project_Vision_Eval, Project_Research, Project_Spec, Project_Spec_Eval, Project_Plan, Project_Plan_Eval, Project_Execute, Project_Execute_Eval, Project_Verify, Project_Extract, Project_Extract_Eval, Project_Complete.
V5 — Phase routing tables in Project_Runner_Prompt.md and Task_Runner_Prompt.md are consistent with the lifecycle. Plan 12 contributions verified: (a) agent-sync.ts defines Agent Handoff edges from Forge Chat family → Project Runner (agent_yG6v4v0wfi5EEHQ8htTXH) with Interactive Mode passthrough message; (b) Forge_Chat_Prompt.md includes ## PROJECT ROUTING section specifying when to hand off (project creation, phase approvals, research/eval decisions, project status checks) and when not to (one-off tasks, general questions, non-project file edits); (c) Project_Runner_Prompt.md defines Interactive Mode (from handoffs) and Autonomous Mode (from cron/spawn). The Phase file is the single state signal. No manual intervention needed between phases except at human gates.
V6 — Human gates are present at: after Vision_Eval convergence, after each Research decision, after Spec_Eval convergence, after Plan_Eval convergence (approval required before Execute), after Execute_Eval if auto-fix doesn’t converge, after Verify, and after Extract_Eval if issues found. The routing table in Project_Runner_Prompt.md correctly identifies all human gates and distinguishes Interactive vs Autonomous handling.
V7 — Erik_Todo format with 🚫 items, project name, description, and link to the relevant report file. Project_Snippet and Env_Snippet both reinforce the Report-File Rule. Blocked phases are escalated with links to the specific evaluation report. Preference-based disposition is documented in Project_Snippet and Project_Runner_Prompt.md.
V8 — Forge/Forge_Project_User_Guide.md covers all 7 required sections: overview, getting started, detailed phase descriptions, agent roles, human gates, document and report reference, and monitoring and steering. All 14 phases are described including the eval phases. A companion Forge/Forge_Agents_User_Guide.md covers the agents system. Spec Section 6 references Forge/Forge_Project_User_Guide.md (updated in prior run).
V9 — Forge_Optimizer supports project mode via --project flag. generate-runs.ts script generates {Project}_Runs.md. Project_Execute_Eval appends an Optimizer reminder (💡) to Erik_Todo after each Eval passes. The Optimizer uses persisted report files as context.
V10 — Vision R10 explicitly defers model selection. Spec Section 10 states the deferral and lists the preserved foundations (durable reports, project-level run tracking, stable review links, project/phase metadata). All foundations are implemented.
V11 — generate-runs.ts outputs per-conversation summary with agent, model, task, duration, Input/Output/Total token columns, and cost. Per-turn breakdown section includes Turn/Role/Tokens per conversation. Total project cost is computed. Verified with live output for this project (12 conversations, $0.84 total).
V12 — Project_Create step 1 runs a find command checking both */Projects/* and */Archived/* paths across the entire repository. The command correctly uses -o to check both locations. Conflicts stop creation with a clear message before any files are created.
Spec Compliance (Plan 12 additions)
- Spec §4 (Agent Handoffs): Plan 12 Task 2 configured Agent Handoff edges in
agent-sync.tsfor Forge Chat → Project Runner with passthrough message directing Interactive Mode. Plan 12 Task 3 added## PROJECT ROUTINGsection toForge_Chat_Prompt.mdwith handoff triggers, passthrough data, and exclusions. Both match Spec requirements. - Spec §10 (Cron Scheduling):
schedules.tsreferences current agent IDs (Project Runner:agent_yG6v4v0wfi5EEHQ8htTXH, Task Runner:agent_-uZUR-rTgMhfMZRbwmFAd).inngest.tsusesSCHEDULE_CONFIGcorrectly. Plan 12 Task 1 fixed stale schedule references inForge_Agent_Orchestration.md— the document now accurately describes the current scheduling architecture. - Forge_Agent_Orchestration.md: Comprehensive, references current system components correctly. Includes Inngest schedule config, forge-discovery, forge-spawn, Ralph Wiggum monitor, AgentJob schema with
projectandphasecolumns, and Agent Handoff edges.
Items
| # | Type | Description | Status | Disposition |
|---|---|---|---|---|
| E1 | Defect | Vision V2 criterion text still uses “Execute → Eval” instead of “Execute → Execute_Eval” — the prior 2026-07-25 run claimed this was fixed but only R2 requirement text was updated, not the V2 criterion row. All other canonical files correctly use “Execute_Eval”. | 🟢 Resolved | Fix |
Resolution Log
| # | Disposition | Action taken | Resolved |
|---|---|---|---|
| E1 | Fix | Updated Vision V2 criterion from “Execute → Eval” to “Execute → Execute_Eval” to match the Spec and all implementation files. | 🟢 |