Project Extract Eval

When to Use

Knowledge extraction is complete and needs review before the project proceeds to Complete.

Inputs

• Project path: {Dept}/Projects/{Project}/

Context Boundary

Read only: Vision, Spec, Changelog, and the extracted work products. Never read Research, Plans, Learnings, or Execute_State — the Extract_Eval reviews the extraction quality, not the journey.

Prerequisite

Must be a fresh agent session — new conversation with clean context, no carryover from extraction.

Procedure

1. Read: Vision, Spec, Changelog
2. Update {Project}_Phase.md: set Phase: Extract_Eval_Started
3. Build, commit, push
4. Review the extraction:
   • Use the Changes table in Changelog to identify all extracted content
   • For each extraction, verify the content reached the correct permanent location
   • Verify extracted content is self-contained (no dangling references to the project)
   • Verify no useful information remains trapped inside the project folder
   • Verify no duplication across permanent documents
   • Verify system-wide skills are listed in the Persistent Knowledge prompt (Bot database)
   • Verify department-level knowledge is linked from {Dept}/AGENTS.md
5. Write findings to the project-level extraction review file {Project}_Extract_Eval.md:
   • Use Extract_Eval_Template.md from Forge/Forge_Project/Skills/Project_Create/templates/
   • No issues found: Use the ✅ Clean single-row format — one row with ✅ Clean type and 🟢 Resolved status. Do NOT list empty categories like “No defects”, “No gaps”, etc.
   • Issues found: Use the multi-row table — only include rows for types that have issues. Omit empty categories entirely.
   • Each issue item is one of:
     • Defect — extraction doesn’t match the intent (wrong location, missing content, incorrect content)
     • Gap — something extractable wasn’t extracted
     • Variance — extraction differs from the original for a reason
     • Learning — something from the extraction process that needs permanent storage
   • Issue items start as 🔴 Open
6. Auto-fix loop: If findings exist, attempt to resolve them automatically before escalating to the user:
   • For straightforward fixes (obvious gaps, clear defects), apply the fix directly
   • Re-evaluate after fixes; iterate up to 3 rounds
   • Only escalate to the user when the Thinker and Evaluator cannot converge
   • Log auto-dispositions in the Resolution Log with note “Auto-dispositioned”
7. If auto-fix loop converged (no remaining open findings): set Verdict to Approved. Skip to step 14. Otherwise: Present remaining findings to the user. For each item, the user decides: fix, update, reject, or defer
8. Process user decisions and update item statuses in the Extract_Eval report:
   • fix: Item remains 🔴 Open with Disposition Fix → returns to Project_Extract
   • update: Don’t fix the implementation; update the Spec to match instead. Update the extraction to match reality → item 🟡 In progress → 🟢 Resolved when done
   • reject: I disagree with the finding; do nothing. Item → 🟢 Resolved (no action)
   • defer: Item → 🟢 Resolved (deferred to future work)
9. If any items have Disposition Fix: set Verdict to Rejected. The project returns to Project_Extract to address the issues.
10. If no items have Disposition Fix: set Verdict to Approved (all remaining items resolved or deferred).
11. Update the Extract_Eval report after each iteration (statuses and Resolution Log)
12. Build, commit, push
13. If Verdict is Approved:
    • Update the Extract_Eval report’s frontmatter status to approved
    • Update {Project}_Phase.md: set Phase: Extract_Eval_Completed
    • Build, commit, push
    • Run the Forge Optimizer automatically to analyze project conversations and save the report:
      1. Export env vars: eval $(cat /proc/1/environ | tr '\0' '\n' | grep -E '^(OPENAI_API_KEY|FORGE_DB_URL)=' | sed 's/^/export /')
      2. Run: npx tsx Forge/Skills/Forge_Optimizer/scripts/optimize.ts --project {Project}
      3. The script saves the report to {Project}_Optimizer_Report.md in the project folder (creates or appends a ## Run — {date} section)
      4. Spawn the Page Manager bot via spawnAgent with { botGroup: 'forge', botHandle: 'page-manager' } to add the sidebar entry (if new), build, commit, push
14. If Verdict is Rejected:
    • Update {Project}_Phase.md: set Phase: Extract_Started (returns to Extract for fixes)
    • Build, commit, push
    • The next agent session invokes Project_Extract to address the issues

Report File Conventions

• One extraction review per project: {Project}_Extract_Eval.md in the project folder
• First review creates the file from the template
• Subsequent reviews (after re-extraction) append as a new ## Run — {date} section
• The report file is a durable artifact — it persists across the entire project lifecycle
• This file provides a stable target for todo item review links

Rules

• Failed Extract_Eval returns directly to Extract — there is no separate Fix phase for extraction
• Context boundary is defined above — do not read implementation details, only the extraction outcome
• Extraction evaluation must be a separate agent session from the extraction
• The auto-fix loop should resolve most findings without user involvement. Only genuine ambiguities or creative decisions require human input