Orchestration Spec

A multi-agent orchestration layer that schedules, dispatches, monitors, and recovers AI agents via LibreChat’s UI Chat API — enabling cron-driven project management, supervised sub-agent delegation, and self-healing execution within the Forge stack.

Requirements: Orchestration Requirements

Research: Orchestration Research · Orchestration Research Gemini

Learnings: Orchestration Learnings

---

1. Execution Layer — LibreChat UI Chat API

All orchestrator-driven agent execution uses the LibreChat UI Chat API (POST /api/agents/chat/agents), not the OpenAI-compatible Agents API. The UI Chat API uses the ResumableAgentController — generation runs independently in the background, disconnected from the HTTP connection that started it. This is the single most consequential architectural decision, confirmed by empirical investigation (Research §11, Learnings §Task 6).

1.1 Agent Invocation Protocol

The orchestrator starts an agent by sending:

{
  "text": "<task prompt>",
  "endpoint": "agents",
  "agent_id": "<agent_id>",
  "conversationId": "new" | "<existing-uuid>",
  "parentMessageId": "<previous-assistant-message-id>"
}

The API returns immediately: {streamId, conversationId, status: "started"}.

streamId === conversationId (always). Both are UUIDs.

1.2 Support Endpoints

Endpoint	Method	Purpose
/api/agents/chat/stream/:streamId	GET (SSE)	Real-time progressive events (not burst)
/api/agents/chat/stream/:streamId?resume=true	GET (SSE)	Reconnect with sync + replay of missed events
/api/agents/chat/status/:conversationId	GET	Poll generation status
/api/agents/chat/active	GET	List active job IDs for current user
/api/agents/chat/abort	POST	Cancel a running generation

1.3 Persistence Guarantees

• User and assistant messages (including tool-call details) are persisted to MongoDB automatically.
• Conversation titles are auto-generated from content.
• Conversations appear in Chat History UI and are queryable via /api/convos.
• Conversation continuation: pass same conversationId + parentMessageId from previous assistant response; controller loads history from MongoDB.
• Partial response saved if all SSE subscribers disconnect while generation is running.

1.4 SSE Event Types (Real-Time)

Event	Content
created	User message details (messageId, conversationId)
on_run_step	Step started (tool call, message creation)
on_run_step_delta	Incremental step updates, tool call argument streaming
on_run_step_completed	Step finished
on_message_delta	Token-by-token content: delta.content[{type, text}]
final	Full conversation object + response message

Events arrive progressively as they happen — not replayed in a burst (unlike the OpenAI-compatible API).

1.5 Auth Lifecycle

The UI Chat API requires JWT session authentication (API key auth returns 401).

Item	Value
JWT validity	15 minutes
Refresh token validity	7 days
Login	POST /api/auth/login with {email, password}
Refresh	POST /api/auth/refresh with refreshToken cookie (no Authorization header)
Refresh returns	New JWT + new refreshToken cookie (rotation)

The orchestrator must implement: login → store JWT + refreshToken cookie → refresh before 15-min expiry → re-login if refresh fails (7-day expiry exceeded).

Critical: The /api/auth/refresh endpoint does NOT require the Authorization header. Sending both JWT + refresh cookie can cause 401 errors.

User-Agent: Non-browser User-Agents get flagged and eventually banned (2-hour non_browser ban). The orchestrator must send a browser User-Agent header on all requests.

Role requirement: The service account must have ADMIN role. USER role gets 403 “Insufficient permissions to access this agent.”

1.6 Parameter Control

The orchestrator has minimal control over agent behavior through API parameters. Unknown/extra parameters are silently ignored. Agent’s own model_parameters override per-request params. Behavior must be encoded in the agent’s configuration and the task prompt — not in API parameters.

1.7 Why Not the OpenAI-Compatible API

Feature	OpenAI-Compatible (/v1/)	UI Chat (/chat/agents)
Execution model	Synchronous — client must hold connection	Resumable — generation runs independently
Conversation ID	❌ Never returned	✅ Returned immediately
Message persistence	❌ Nothing saved to MongoDB	✅ Full persistence
Streaming	Silent during tool execution, burst at end	Real-time progressive streaming
Disconnect resilience	❌ Agent cancelled on disconnect	✅ Agent continues independently
Reconnection	❌ Not possible	✅ ?resume=true with sync/replay
Abort	Client disconnect = cancel	✅ POST /chat/abort endpoint
Auth	API key	JWT session

---

2. Orchestrator Service — Dispatcher

A Railway web service (TypeScript) that serves as the thin orchestration layer between scheduling and LibreChat agent execution. It is a dispatcher, not a replacement for LibreChat’s agent runtime. Custom scripts (not Mastra or any framework) — the orchestration logic is simple: receive trigger → invoke agent → monitor → handle failure.

2.1 Deployment

Railway web service. Stays running, exposes an HTTP endpoint. Scheduling is handled by Inngest, which POSTs to the orchestrator on schedule. Isolated from LibreChat, independently deployable, direct network access to LibreChat’s API within Railway’s private network.

2.2 Core Loop

1. Receive trigger from Inngest (or on-demand via POST /orchestrator/run)
2. Authenticate with LibreChat (JWT lifecycle)
3. Invoke the appropriate agent via UI Chat API with task prompt + context
4. Optionally subscribe to SSE for real-time monitoring (stall detection)
5. On completion: parse final event, update orchestration state
6. On stall/timeout: abort → Ralph Loop restart (§5)

2.3 JWT Auth Manager

A self-contained module within the orchestrator that manages the auth lifecycle:

login(email, password) → {jwt, refreshToken}
refresh(refreshToken) → {jwt, refreshToken}
getValidToken() → jwt  // auto-refreshes if <2min remaining

Tokens stored in memory (not filesystem). On orchestrator restart, full re-login.

2.4 Hybrid Migration Path

The orchestrator is designed with clean abstraction: when LibreChat ships native background agents + scheduling (on their 2026 roadmap), the execution layer can be swapped without rewriting orchestration logic. The orchestrator should be thin.

---

3. Discovery MCP Server

A custom MCP server that gives agents structured access to Forge data without filesystem grepping. Configured in librechat.yaml and available to agents via the standard MCP tool interface.

3.1 forge-discovery MCP (Read-Only)

Backed by filesystem reads. Provides structured, relational queries that Typesense isn’t optimized for, plus fuzzy search for when the agent doesn’t know the exact name or department.

Tools:

Tool	Returns
list_departments	All department folders with README summaries
list_projects	Active projects in a department (name, phase, files)
find_project	Project match given a partial or misspelled name and unknown department
get_project_phase	Current phase of a specific project
get_project_files	All files in a project folder with metadata
list_skills	All available skills with descriptions
find_skill	Skill match given a partial or misspelled name, description, and unknown department
get_skill_details	Full SKILL.md content for a named skill

Projects live in {DEPT}/Projects/. The Discover MCP reads the filesystem structure and returns structured data — it is a read-only index.

---

4. Sub-Agent Spawning

4.1 Spawn MCP Tool (R5, R11)

An MCP tool that an LLM can call to spawn a sub-agent via the UI Chat API. Two modes:

Mode 1 — Disconnect (fire-and-forget): Spawns the sub-agent, records the conversationId, then disconnects. The orchestrator or calling agent looks for the finished conversation later (via /api/convos or /api/agents/chat/status/:conversationId). Use when the caller doesn’t need to monitor progress in real-time.

Mode 2 — Monitor (Ralph Wiggum): Spawns the sub-agent, subscribes to the SSE stream, and monitors tool-calls and reasoning in real-time. Can interrupt under certain conditions (§5). On completion, timeout, or error, the monitor checks if the job is done and continues it in the same conversation or a fresh one.

Spawn tool parameters:

Parameter	Type	Description
agent_id	string	Target agent ID
task_prompt	string	The task to execute
conversation_id	string	"new" or existing UUID for continuation
mode	enum	disconnect | monitor
required_tools	string[]	Tools the sub-agent must call (for Ralph Wiggum validation)
max_duration_seconds	number	Stall timeout (for Ralph Wiggum)

Spawn tool returns:

Field	Type	Description
conversation_id	UUID	The LibreChat conversation ID
stream_id	UUID	Same as conversation_id (for SSE subscription)
status	string	"started"

4.2 Transcript Review (R7)

Sub-agent transcripts appear in LibreChat’s Chat History automatically (the UI Chat API persists everything). No special mechanism needed — the conversationId links to a full, reviewable transcript with tool-call details and auto-generated titles. This works out of the box but must be tested to confirm consistent behavior across all agent configurations.

---

5. Ralph Wiggum Monitor (R6, R12)

A monitoring wrapper around sub-agent execution. Named after the OpenClaw Playbook’s “Ralph Loop” pattern (see Forge/Skills/Orchestration/references/OpenClaw_Playbook/06_Coding_Agents.md), but adapted for our architecture: instead of re-running a CLI command with fresh context, the Ralph Wiggum monitor subscribes to a running agent’s SSE stream, watches its behavior in real-time, and intervenes when needed.

The key distinction from the OpenClaw Ralph Loop: this is TypeScript code monitoring a conversation via SSE events, not an LLM monitoring another LLM (that’s R12’s future extension — see §9).

5.1 Monitoring Behavior

1. After spawning a sub-agent (via Spawn MCP tool in monitor mode), subscribe to the SSE stream at /api/agents/chat/stream/:streamId
2. Parse incoming events to track:
   • Tool calls made (via on_run_step events with stepDetails.type: "tool_calls")
   • Token output progress (via on_message_delta events)
   • Completion (via final event)
3. Detect stall: no events for max_duration_seconds (configurable per invocation)
4. Detect missing required tools: after completion, check if all tools in required_tools list were called

5.2 Intervention Conditions

Condition	Action
No events for max_duration_seconds	Abort → restart in fresh conversation with additional context: “Previous attempt stalled. Resume from: [state summary]“
Agent completes but didn’t call a required tool	Continue in same conversation: “You must call {tool_name} before finishing. Current state: [state]“
Agent errors out	Check if partial work was saved (MongoDB). If yes, continue in same conversation. If no, restart fresh
Agent times out (generation-level)	Abort → restart fresh with context from filesystem state
Maximum retries exceeded	Append 🚫 item to Forge/Output/Erik_Todo.md, notify user

5.3 Continuation Strategy

When restarting, the Ralph Wiggum monitor must provide fresh context to the new agent invocation:

• Same conversation continuation (preferred when partial work exists): pass the existing conversationId + parentMessageId from the last assistant message. The controller loads history from MongoDB automatically.
• Fresh conversation (when the previous attempt was corrupted or confusing): start with conversationId: "new" and include a summary of the task + current state in the task prompt.

5.4 Context Is a Cache, Not State

Borrowed from the OpenClaw Playbook (§06): the agent must be able to reconstruct its situation from the filesystem and database state alone. If the agent can’t recover from a restart by reading the current state of files and todo items, the architecture has a single point of failure. The Ralph Wiggum monitor assumes this principle — it doesn’t try to pass accumulated context; it relies on the filesystem as the source of truth.

---

6. Project Runner Agent

A runner agent that operates in two modes — Interactive and Autonomous — serving as the single project-aware entry point for all project work. It reconciles stalled projects hourly and handles human gate decisions interactively.

6.1 Two-Mode Architecture

Interactive Mode: Invoked via LibreChat Agent Handoffs from Forge Chat, under the user’s account. The user is present and can respond to questions and approvals in real-time. Used when the user initiates project work, a phase completion requires human approval, or the user asks to check project status.

Autonomous Mode: Invoked by the hourly cron schedule or via spawn_agent, under forge@uvilo.com. No human is present. Used for hourly reconciliation — catching stalled agents, advancing orphaned completions, and confirming blocked projects have Erik_Todo entries.

6.2 Agent-to-Agent Handoff (Happy Path)

When an SRP agent completes a phase and no human gate is required, it calls spawn_agent directly to invoke the next agent in the routing table — instead of writing _Completed and waiting for Project Runner. This eliminates idle latency between phases.

The completing agent:

1. Sets its own phase to _Completed
2. Looks up the next phase in the routing table
3. Calls spawn_agent with the appropriate agent_id and task prompt

Project Runner never handles happy-path dispatch — it only reconciles failures and handles human gates.

6.3 Phase Routing Table

Phase	Agent	Mode	Human Gate?
Vision	Project Thinker	Autonomous	No
Vision_Eval	Project Evaluator (fresh session)	Autonomous	Yes — on block
Research	Project Thinker	Autonomous	No
Spec	Project Thinker	Autonomous	No
Spec_Eval	Project Evaluator (fresh session)	Autonomous	Yes — on block
Plan	Project Thinker	Autonomous	No
Plan_Eval	Project Evaluator (fresh session)	Autonomous	Yes — on approval
Execute	Project Worker	Autonomous	No
Execute_Eval	Project Evaluator (fresh session)	Autonomous	Yes — on block
Verify	Project Evaluator	Autonomous	Yes — on issues
Extract	Project Thinker	Autonomous	No
Extract_Eval	Project Evaluator (fresh session)	Autonomous	Yes — on block

6.4 Hourly Reconciliation

Project Runner runs hourly to catch failures the happy path missed:

1. Scan all active projects via forge-discovery
2. For each project with Phase _Started and a stale AgentJob (no activity for 60 minutes): abort and re-dispatch or escalate
3. For each project with Phase _Completed where no next phase has started: spawn the next agent (happy-path handoff missed)
4. For each project with Phase _Blocked: confirm Erik_Todo entry exists

6.5 Agent Configuration

Field	Value
Agent ID	agent_<id> (created in LibreChat Agent Builder)
Tools	forge-discovery MCP, uvilo-filesystem, forge-spawn
Instructions	Project Runner system prompt (dual-mode)
Model	Primary: OpenRouter / z-ai/glm-5.1 / Clone: anthropic / claude-sonnet-4-6

---

7. Task Runner Agent

Same pattern as the Project Runner (§6). One master agent and one or more clones using different models. The Task Runner checks for incomplete items on the agent todo list and dispatches the correct SRP agent to execute each task.

7.1 Task Dispatch Workflow

1. Scheduled by the orchestrator (Inngest)
2. Checks Forge/Output/Agent_Todo.md for incomplete items
3. For each incomplete task, determine which SRP agent handles it (based on task type or title pattern matching)
4. Spawn the SRP agent via the Spawn MCP tool (disconnect mode for simple tasks, monitor mode for complex ones)
5. Update todo item state to [ ] 🟨 (convoId: "{conversationId}") Existing todo description here
6. On SRP agent completion (checked via status endpoint), update todo item to completed [x] ✅ or blocked [ ] 🚫 and add a new item to Forge/Output/Erik_Todo.md

7.2 Agent Configuration

Same tool set and clone pattern as Project Runner. Different instructions focused on task dispatch rather than project phase management.

---

8. SRP Agents

Single-Responsibility Principle agents — small, focused agents that do one thing well. Benefits: smaller system prompts, more predictable behavior, composability, easier testing and debugging. These specialized agents with smaller contexts can be called by agents running complex jobs with large contexts, potentially reducing the size of every agent’s system prompt.

8.1 Initial SRP Agent Set

Agent	Single Responsibility
Page Manager	Create, rename, move, delete pages; update front matter; update sidebar to match
Project Worker	Knows basic project flow by default; loads the necessary project skill as needed
Project Evaluator	Same as Project Worker, but using a model more suitable for evaluation
Project Thinker	Same as Project Worker, but using a model more suitable for complex planning

8.2 SRP Agent Design Principles

• Each agent has one job and a minimal system prompt
• Each agent has access to only the MCP tools it needs (principle of least privilege)
• Complex agents call SRP agents via the Spawn MCP tool rather than doing everything themselves
• SRP agents return structured results that the calling agent can use without re-reading the filesystem

8.3 Agent Configuration Management

Agent definitions start as manual creation in LibreChat Agent Builder (Option A from Research §8). When the orchestrator is built, migrate to programmatic deployment via LibreChat’s Agent CRUD API (/api/agents/v1/ with JWT auth) — agent configs as TypeScript objects, deployed to LibreChat on orchestrator startup. This keeps agent definitions version-controlled in git.

We also need to update agent-sync.yaml and update-agents.ts to handle multiple agent families — members that share all settings except name, description, model, and model parameters (like the existing Forge Agent family). Each agent family needs its own system prompt file: PROJECT_RUNNER.md, TASK_RUNNER.md, PAGE_MANAGER.md, PROJECT_WORKER.md, PROJECT_EVALUATOR.md, PROJECT_THINKER.md (analogous to how the Forge Agent uses FORGE.md).

8.4 Model Selection

After implementation we will create a process to determine the best language model to use for the various use cases. Best here is defined as the optimal balance between output quality, error rate, speed, and cost.

---

9. Future: LLM-Based Monitoring (R12 Extension)

The Ralph Wiggum pattern described in §5 is TypeScript code monitoring the SSE stream — it checks for tool-call presence and stall conditions programmatically. R12 envisions a deeper version: an LLM monitoring another LLM’s reasoning and tool-calls, understanding the semantic intent, and making intelligent intervention decisions.

This is the same pattern as R8/R9 (master + clone) applied to monitoring: instead of code rules (“no event for N seconds = stall”), an LLM monitor can detect subtler problems (“the agent has been reading the same file three times in a row — it’s stuck in a loop” or “the agent called the right tool but with wrong arguments based on the task description”).

This is out of scope for the initial implementation but the architecture supports it: the SSE stream provides all the information (tool calls, reasoning text, step deltas) that an LLM monitor would need. The Spawn MCP tool’s monitor mode would gain a third option: llm_monitor, where a second LLM session receives the sub-agent’s SSE events as input and can issue abort/continue decisions.

---

10. Cron Scheduling (R1)

Agents must be triggered on a schedule. LibreChat does not have built-in cron scheduling. Inngest handles scheduling — it POSTs to the orchestrator’s HTTP endpoint on each schedule, which then invokes the appropriate agent.

10.1 Schedule Configuration (Inngest)

Inngest schedules are configured in the Inngest project (separate from the orchestrator). The orchestrator does not need its own cron engine or a separate /orchestrator/trigger endpoint — it reuses POST /orchestrator/run (§10.2).

Job	Schedule	Description
Project Runner	Hourly	Reconcile stalled/orphaned projects
Task Runner	Hourly	Check for new agent todo items
Memory Compaction	Once daily	Run memory compaction agent
Infrastructure Check	Monthly	Check infrastructure configuration

10.2 On-Demand Triggering

The orchestrator exposes a lightweight HTTP endpoint for agent invocation (used by both Inngest schedules and on-demand requests):

POST /orchestrator/run
{
  "agent_id": "agent_<id>",
  "task_prompt": "...",
  "mode": "disconnect" | "monitor"
}

This allows manual triggering from other agents or ad-hoc requests.

---

11. Data Storage — Hybrid Architecture

11.1 Filesystem (Source of Truth for Documents)

Project documents remain as markdown files in the git repo: Requirements, Spec, Plans, State, Learnings, WIP. These are human-reviewed artifacts. The filesystem is the source of truth for project structure and document content.

11.2 Postgres — forge Database (Structured State)

The Railway VectorDB Postgres instance with a separate forge database for orchestration state (Research §12). This replaces the original MongoDB approach — 8-9× lower latency via Railway private network, schema enforcement, ACID transactions, and naming conventions aligned with the Neon uvilo database.

Connection: FORGE_DB_URL env var → postgresql://postgres:{password}@vectordb.railway.internal:5432/forge

Orchestration state — AgentJob table tracking which agents are running, job status, retry counts:

Column	Type	Description
id	TEXT (Cuid2, agt_ prefix)	Primary key
conversationId	TEXT	LibreChat conversation ID
agentId	TEXT	Agent that was invoked
taskPrompt	TEXT	The task given to the agent
status	TEXT CHECK	running | completed | failed | aborted
startedAt	TIMESTAMPTZ	When the agent was invoked
completedAt	TIMESTAMPTZ	When the agent finished
retryCount	INT	Number of Ralph Wiggum restarts
parentJobId	TEXT FK	For sub-agent tracking
metadata	JSONB	Arbitrary key-value
createdAt	TIMESTAMPTZ	Record creation time
updatedAt	TIMESTAMPTZ	Record update time

Project index — Project table, mirrored from Phase files for fast queries without filesystem reads:

Column	Type	Description
id	TEXT (Cuid2, prj_ prefix)	Primary key
projectName	TEXT UNIQUE	Project folder name
department	TEXT	Department folder name
phase	TEXT	Current phase
status	TEXT CHECK	draft | review | approved | published | archived
updatedAt	TIMESTAMPTZ	Last sync time
createdAt	TIMESTAMPTZ	Record creation time

Naming conventions (aligned with Neon uvilo database): PascalCase table names, camelCase columns, Cuid2 IDs with 3-letter prefix, Table_column_idx index names. See Research §12.5 for full convention rules and reserved prefix registry.

11.3 MongoDB (Conversation History — LibreChat)

LibreChat stores conversation content in its existing MongoDB instance — user messages, assistant messages, tool call details, conversation metadata. The conversationId in AgentJob is a reference key to MongoDB conversation records. Orchestration state and conversation data are inherently separated by storage system.

11.4 Typesense (Semantic Search)

The existing Typesense index already provides semantic search across all repo content. The forge-discovery MCP provides structured, relational queries that complement (not replace) Typesense search.

---

Requirement Coverage

Req	Description	Spec Section	Status
R1	Cron scheduling of agents	§10	✅ Covered (Inngest)
R2	Project discovery tool	§3.1	✅ Covered
R3	Skill discovery tool	§3.1	✅ Covered
R4	Todo/checklist MCP	—	⏭️ Skipped (per user)
R5	Spawning sub-agents	§4	✅ Covered
R6	Ralph Wiggum pattern	§5	✅ Covered (TS monitoring)
R7	Sub-agent transcript review	§4.2	✅ Covered (UI Chat API — works out of box, test needed)
R8	Project Runner agent	§6	✅ Covered (master + clones)
R9	Task Runner agent	§7	✅ Covered (same pattern as R8)
R10	SRP agents	§8	✅ Covered
R11	Spawn MCP tool (disconnect/monitor)	§4.1	✅ Covered
R12	Ralph Wiggum MCP tool (LLM monitoring)	§9	✅ Covered (future extension, architecture supports it)