Project Automation Research
1. Workflow Audit Methodology
R1: Full Workflow Audit; R4: Complete Work Products
1.1 Finding
The workflow still needs a systematic post-implementation audit, but the canonical target is now the evaluation-based lifecycle rather than the earlier audit/fix vocabulary. The audit must cover lifecycle docs, project skills, agent prompts, shared snippets, and templates, and it must explicitly verify that report files are persisted wherever review outputs are produced.
1.2 Options
Option A: Transition-only audit — verify only phase boundaries and routing.
Option B: Artifact-only audit — verify only files, templates, and report outputs.
Option C: Combined audit — verify transitions first, then artifact completeness and report persistence.
1.3 Recommendation
Option C. The workflow succeeds only if both transitions and work products are correct.
1.4 Decision
Option C. Run a final workflow audit after all structural changes are implemented, and record the results in a persisted project audit file.
Decided by: erik@uvilo.com — 2026-06-01
2. Formal Evaluation Phase Lifecycle
R2: Formal Evaluation Phase Lifecycle
2.1 Finding
The earlier Spec_Audit / Plan_Audit concept evolved into a broader evaluation-phase model. The correct target lifecycle is:
Create → Requirements → Requirements_Eval → Research → Spec → Spec_Eval → Plan → Plan_Eval → Execute → Eval → Verify → Extract → Extract_Eval → Complete
This lifecycle is more consistent than mixing Audit, Evaluate, Fix, and Implement terminology. It also gives Requirements and Extract the same quality-gate treatment as Spec and Plan.
2.2 Options
Option A: Keep mixed Audit/Evaluate/Fix terminology
Option B: Normalize everything around evaluation phases
2.3 Recommendation
Option B. One consistent model is easier to route, easier to document, and easier for humans to follow.
2.4 Decision
Option B. Use the evaluation-phase lifecycle everywhere and retire the old mixed terminology.
Decided by: erik@uvilo.com — 2026-06-01
3. Persisted Review and Evaluation Reports
R3: Persisted Review and Evaluation Reports; R7: Easy Human Review and Feedback; R9: Forge Optimizer Integration; R10: Deferred foundations
3.1 Finding
Erik_Todo, Forge Optimizer, and future model-selection work all need durable review artifacts. Chat-only findings are not sufficient. Every review-producing phase must write to a stable markdown file so the human can review it later and tools can link to it.
3.2 Options
Option A: Inline findings only — keep findings in chat.
Option B: Append everything to one giant project report file.
Option C: Use stable per-phase report files, appending repeated runs within the same file when that phase naturally accumulates history.
3.3 Recommendation
Option C. It preserves stable URLs while keeping reports scoped and reviewable.
3.4 Decision
Option C. The workflow will persist at least these report artifacts:
{Project}_Requirements_Eval.md{Project}_Spec_Eval.md{Project}_Plan_Eval.md{Project}_Eval_{N}.md(one file per plan, append reruns){Project}_Verification.md(append reruns){Project}_Extract_Eval.md{Project}_Audit_Results.md{Project}_Runs.md
Each report should include phase name, run context, checklist/findings, outcome, and next-step disposition.
Decided by: erik@uvilo.com — 2026-06-14
4. Agent Handoff Design
R5: Smooth Agent Handoffs
4.1 Finding
The Phase file remains the simplest orchestration signal, but the handoff model only works if phase names are consistent and evaluation outcomes are written to files that the next agent and the human can inspect.
4.2 Options
Option A: Phase-file orchestration with consistent naming and persisted reports
Option B: Add a separate handoff payload file
4.3 Recommendation
Option A. It preserves the current orchestration model with minimal extra machinery.
4.4 Decision
Option A. Use the Phase file as the canonical state signal, and use persisted report files as the canonical review artifacts.
Decided by: erik@uvilo.com — 2026-06-01
5. Human-in-the-Loop Gate Design
R6: Human-in-the-Loop at Correct Points; R7: Easy Human Review and Feedback
5.1 Finding
Human review is easier when Erik_Todo links directly to the exact file that needs attention. This depends on persisted report files existing for evaluation phases.
5.2 Options
Option A: Keep minimal Erik_Todo entries
Option B: Use enhanced Erik_Todo entries that link directly to the relevant document or report file
5.3 Recommendation
Option B. The review item should always say what needs attention and where to review it.
5.4 Decision
Option B. Erik_Todo entries will link directly to the Requirements draft, Spec draft, Plan draft, evaluation report, verification report, or other persisted artifact that requires review.
Decided by: erik@uvilo.com — 2026-06-01
6. User Guide Scope and Format
R8: User Guide
6.1 Finding
The guide must explain not only phases and agents, but also the report files the system now produces and how humans use those files during review.
6.2 Options
Option A: Keep the guide in the project only
Option B: Publish a permanent Forge guide
Option C: Draft in-project, extract permanently
6.3 Recommendation
Option C. It fits the project lifecycle and still yields permanent documentation.
6.4 Decision
Option C. The guide is a project deliverable extracted to Forge/Project_Automation_Guide.md.
Decided by: erik@uvilo.com — 2026-06-01
7. Forge Optimizer Integration
R9: Forge Optimizer Integration
7.1 Finding
The Optimizer should remain manually invoked, but it needs project-awareness and should be able to reference the project’s persisted review files and run summaries.
7.2 Options
Option A: Automatic Optimizer runs after every phase
Option B: Manual project-aware Optimizer with a workflow reminder
7.3 Recommendation
Option B. It preserves user control over cost while making the workflow discoverable.
7.4 Decision
Option B. Add project-aware Optimizer support and a post-Eval reminder that points the user to the command.
Decided by: erik@uvilo.com — 2026-06-01
8. Model Selection System
R10: Model Selection System [Deferred / Not required in this iteration]
8.1 Finding
The full model-selection system is not part of this iteration, but this project must still preserve the prerequisites that future model-selection work will need: durable evaluation reports, project-level run tracking, stable project/phase metadata, and linkable human-review artifacts.
8.2 Options
Option A: Implement the full system now
Option B: Defer implementation but preserve the required foundations now
8.3 Recommendation
Option B. The workflow project should not broaden into a full experimentation platform in this iteration.
8.4 Decision
Option B. Defer model-selection implementation. Preserve the reporting and run-tracking foundation in this project.
Decided by: erik@uvilo.com — 2026-06-14
9. Agent Run Summary
R11: Agent Run Summary
9.1 Finding
Project-level run tracking still requires explicit project/phase data and a durable summary artifact. The summary should complement, not replace, the evaluation reports.
9.2 Options
Option A: Infer project from prompts
Option B: Add first-class project and phase fields and generate a durable runs document
9.3 Recommendation
Option B. It is cleaner and queryable.
9.4 Decision
Option B. Add project and phase to AgentJob, pass them from Project Runner, and generate {Project}_Runs.md from AgentJob + MongoDB + LangFuse data.
Decided by: erik@uvilo.com — 2026-06-01
10. Global Project Name Uniqueness
R12: Global Project Name Uniqueness
10.1 Finding
Project names currently risk colliding across departments or with archived work. Because project folder names also drive document prefixes, collisions are operationally expensive and hard to unwind later.
10.2 Options
Option A: Enforce uniqueness only within a department
Option B: Enforce global uniqueness across every department’s Projects/ and Archived/ directories before creation
10.3 Recommendation
Option B. It prevents collisions before they enter the system.
10.4 Decision
Option B. Project_Create must search all departments, including Archived projects, and stop with a clear conflict message before creating any files when a duplicate name is found.
Decided by: erik@uvilo.com — 2026-06-14