archived Visibility internal Owner erik@uvilo.com Approver _ Created 2026-04-26 Updated 2026-06-27

Orchestration Research

1. LibreChat Current State & Roadmap

R1, R5, R7: Related Requirements

1.1 Finding

LibreChat ships with two multi-agent mechanisms:

Agent Handoffs — agents delegate to specialized agents (recursive handoffs supported). An orchestrator agent can call a specialist, which can in turn call another specialist.
Mixture-of-Agents (MoA) chains — up to 10 agents in sequence, each receiving prior agents’ output. Configured in the Agent Builder UI under Advanced Settings.

LibreChat exposes three agent execution APIs:

API	Endpoint	Auth	Execution Model	Persistence
OpenAI Chat Completions	`POST /api/agents/v1/chat/completions`	API key	Synchronous	❌ None
Open Responses	`POST /api/agents/v1/responses`	API key	Synchronous	⚠️ Broken (store: true doesn’t save messages)
UI Chat (Resumable)	`POST /api/agents/chat/agents`	JWT session	Resumable (background)	✅ Full (user + assistant + tool calls)

The UI Chat API is the recommended execution path for orchestration — it solves every deal-breaker identified in the OpenAI-compatible API investigation (see Section 11).

Agent ID format: agent_<alphanumeric> (e.g., agent_69d69e3198cb9ee870cd0373). Currently 11 agents configured across OpenRouter (GLM, Gemini), OpenAI (GPT variants), and Anthropic (Claude variants).

What’s NOT yet available:

No background/scheduled agents. The 2026 roadmap lists “Background Agents and Subagents” under Workflows & Interactivity, but it’s not implemented yet.
No human-in-the-loop approval gates (roadmap item, not shipped).
No built-in cron scheduling for agents.
Multi-agent orchestration is listed as a “future update” with no concrete timeline.
The OpenAI Agents SDK integration request (#9989) was closed as “not planned” — LibreChat is building its own agent orchestration.

1.2 Options

Approach	Pros	Cons
A: Wait for LibreChat native scheduling	No custom code to maintain; deep integration	No timeline; could be months; blocks all orchestration work
B: External orchestrator scripts calling Agents API	Works today; full control over scheduling, retries, Ralph loop; TypeScript-native	Must build/maintain scripts; looser coupling; need to handle streaming + error recovery ourselves
C: Hybrid — scripts now, migrate to native when available	Ship immediately; path to reduce custom code later	Migration effort when LibreChat ships the feature

1.3 Recommendation

Option C (Hybrid). Build orchestrator scripts that call the Agents API today. Design the scripts with clean abstraction so that when LibreChat adds native background agents + scheduling, we can swap the execution layer without rewriting the orchestration logic. The orchestrator should be thin — it’s a scheduler + dispatcher, not a replacement for LibreChat’s agent runtime.

1.4 Decision

Supported by investigation. The Agents API is functional and usable. No native scheduling exists. Hybrid is the only viable path forward.

2. Orchestration Architecture Pattern

R5, R6, R8, R9, R10: Related Requirements

2.1 Finding

Three dominant multi-agent orchestration patterns:

Supervisor (Hub-and-Spoke) — A central orchestrator decomposes tasks, delegates to specialists, monitors progress, and synthesizes results.
Mesh/Peer-to-Peer — Agents communicate directly without a central coordinator.
Hierarchical — Director agents manage agent pods (clusters of specialists).

For our use case, the Supervisor pattern is the clear fit — we need traceability, quality gates, and centralized state. The OpenClaw Playbook’s Ralph Loop pattern addresses R6 — agents that stop prematurely are detected and restarted with fresh context.

Investigation-constrained refinements to the Supervisor pattern:

The OpenAI-compatible Agents API (/v1/chat/completions) is fundamentally synchronous — an orchestrator must hold the HTTP connection for the entire sub-agent execution. However, the UI Chat API (/chat/agents) uses the ResumableAgentController which runs generation independently in the background. This changes the constraints significantly:

Using the UI Chat API: The orchestrator does NOT need to hold a connection. Generation runs independently. The orchestrator can disconnect and reconnect later to check status or retrieve results.
Using the OpenAI-compatible API: The orchestrator must hold the connection for the entire execution, with a ~4-minute timeout budget.

Recommended: Use the UI Chat API for orchestration. The resumable execution model eliminates the need for the orchestrator to be a long-running process holding connections. Instead:

The orchestrator can be a fire-and-forget dispatcher — start generation, record the conversationId/streamId, move on
Results are checked asynchronously via GET /api/agents/chat/status/:conversationId
The SSE stream can be subscribed to for real-time monitoring, but it’s optional
Partial results are saved automatically if the orchestrator disconnects
The Ralph Loop is still needed for stall detection, but reconnection is supported

Remaining constraint: JWT auth (15-minute expiry) requires the orchestrator to implement token refresh logic. The API key auth only works with the synchronous /v1/ endpoints.

2.2 Options

Pattern	Fit	Trade-off
Supervisor	High — matches requirements for Project Runner / Task Runner / SRP agents	Single point of failure at supervisor; potential bottleneck
Mesh	Low — no need for peer-to-peer agent communication	More resilient but no centralized control or audit trail
Hierarchical	Medium — useful if we scale to multiple departments	Over-engineered for current single-user, single-department setup

2.3 Recommendation

Supervisor pattern with the following hierarchy:

Scheduler (cron) → triggers Project Runner or Task Runner
Project Runner — checks active projects, detects phase completions, initiates next phases
Task Runner — reads agent todo list, dispatches SRP agents
SRP Agents — single-responsibility agents (Create Page, Update Sidebar, etc.)

The Ralph Loop pattern wraps each SRP agent execution: if it stalls/crashes, the orchestrator kills and restarts it with a fresh session.

Operational constraints (using UI Chat API): No strict time budget — agents run independently in the background. The orchestrator polls status or subscribes to SSE. The Ralph Loop is still needed for stall detection, but disconnection no longer means data loss.

Operational constraints (using OpenAI-compatible API, fallback): Each SRP agent call must complete in ≤4 minutes. Tasks requiring longer must be split into multiple calls with explicit context passing between them.

2.4 Decision

Supported by investigation — revised. The UI Chat API’s resumable execution model significantly relaxes the constraints. The Supervisor pattern is confirmed, but the orchestrator no longer needs to hold connections for the entire sub-agent execution. The Ralph Loop remains essential for stall detection, but reconnection is supported and disconnection no longer causes data loss. JWT auth lifecycle management is a new requirement.

3. Cron Scheduling Implementation

R1: Related Requirement

3.1 Finding

LibreChat does not have built-in cron scheduling for agents. We need an external scheduler. Options:

Railway cron jobs — Railway supports cron services natively. A lightweight TypeScript service with node-cron or cron package, running as a Railway service, can trigger orchestrator scripts on schedule.

System-level cron — Simple but less portable, harder to monitor.

Mastra workflows — Mastra has a workflow engine with suspend/resume but no built-in cron. Would need external triggering anyway.

The orchestrator script needs to:

Wake up on schedule
Determine what needs doing (check project phases, todo lists)
Call the LibreChat Agents API with the right agent + prompt + context
Stream the response, monitoring for completion/stall
On stall/timeout: kill session, restart (Ralph loop)
On completion: update state files, check for next actions

3.2 Options

Approach	Pros	Cons
A: Railway cron service (TypeScript)	Runs where we already deploy; native cron support; can use the same env vars and network	Another service to maintain; need to handle cold starts
B: Standalone cron script on existing Railway service	No new service; runs alongside LibreChat	Tightly coupled; cron failures affect the main app; harder to scale
C: GitHub Actions scheduled workflow	Free; no infra to manage	Can’t easily call LibreChat API internally; 10-min max runtime; limited to public-network endpoints

3.3 Recommendation

Option A: Dedicated Railway cron service. A small TypeScript app with node-cron that runs the orchestrator logic. It’s isolated from LibreChat, can be independently deployed/scaled, and has direct network access to the LibreChat Agents API within Railway’s private network.

3.4 Decision

Supported by investigation. No changes from original recommendation — the investigation confirms no native scheduling exists and Railway is the right deployment target.

4. MCP Tools for Discovery & Task Tracking

R2, R3, R4: Related Requirements

4.1 Finding

These three requirements (Project Discovery, Skill Discovery, Todo/Checklist) are all about building custom MCP servers that give agents structured access to Uvilo OS data without filesystem grepping.

MCP Server Architecture:

LibreChat supports MCP servers via librechat.yaml config (both SSE and Streamable HTTP transports)
MCP tools are automatically available to agents that have the server configured
Deferred tools (lazy loading) are supported — tools load on demand, saving context window

For R2 (Project Discovery): An MCP server that reads the filesystem structure and returns structured project data (name, phase, files, current state). This is essentially a read-only index of {DEPT}/Projects/.

For R3 (Skill Discovery): An MCP server that reads */Skills/*/SKILL.md files and returns skill metadata (name, description, when to use, prerequisites). Already partially served by the Typesense MCP — could extend it or build a dedicated skill MCP.

For R4 (Todo/Checklist MCP): This is more complex — it needs read AND write. Agents need to create, update, and query todo items with states (empty, started, blocked, completed). This is essentially a lightweight task tracker.

4.2 Options

Approach	Pros	Cons
A: Three separate MCP servers	Single responsibility; independent deployment; easy to debug	More services to maintain; potential duplication
B: One unified “Uvilo OS” MCP server	Single deployment; shared auth; simpler config in librechat.yaml	Larger codebase; one failure affects all tools; harder to iterate on individual features
C: Two servers — Read-only (Projects + Skills) and Read-write (Todo)	Balances SRP with practicality	Moderate complexity

4.3 Recommendation

Option C: Two MCP servers.

Uvilo-Discover MCP — read-only tools: list_projects, get_project_phase, get_project_files, list_skills, get_skill_details. Backed by filesystem reads (consistent with current architecture).
Uvilo-Todo MCP — read-write tools: create_item, update_item, list_items, get_item. Backed by a simple data store (see Section 6).

The existing Typesense MCP already provides semantic search across all repo content. The Discover MCP provides structured, relational queries (list projects in a department, get current phase) that Typesense isn’t optimized for.

4.4 Decision

Supported by investigation. No changes — investigation of the Agents API doesn’t affect MCP server architecture decisions.

5. Mastra vs. Custom Scripts vs. TanStack Start

R1, R5, R8, R9, R10: Related Requirements (also Additional Requirements for GUI)

5.1 Finding

Mastra (v1.0, Jan 2026, 22.8k GitHub stars) is a TypeScript agent framework built on Vercel AI SDK. It provides:

Agent creation with createAgent
Workflow engine with createWorkflow / createStep — supports branching, parallel, suspend/resume, human-in-the-loop
Supervisor agents that coordinate specialized sub-agends
Built-in observability (Mastra Studio)
Memory system with thread/resource patterns
Exposes agents as REST APIs

Mastra’s overlap with LibreChat:

Both provide agent runtimes (Mastra uses AI SDK directly; LibreChat uses its own agent framework)
Both support tool use and MCP
Mastra has workflows; LibreChat has MoA chains and handoffs
Mastra has no chat UI; LibreChat is primarily a chat UI
Mastra has no user management, no multi-user chat, no conversation history

Key tension: Mastra and LibreChat solve overlapping problems. Running both means maintaining two agent runtimes. However, they could be complementary:

LibreChat = chat UI + conversation history + human interaction
Mastra = workflow engine + scheduling + orchestration logic

TanStack Start (RC status) is a full-stack React framework. Relevant for the GUI (Additional Requirements) but not for the core orchestration engine.

5.2 Options

Approach	Pros	Cons
A: Custom TypeScript orchestrator scripts	Full control; no framework lock-in; lightweight; directly calls LibreChat API	Must build workflow logic from scratch; no built-in observability; more boilerplate
B: Mastra for orchestration + LibreChat for chat	Built-in workflow engine with branching, suspend/resume, human-in-the-loop; observability; supervisor pattern built-in; TypeScript-native	Two agent runtimes to maintain; Mastra’s agents can’t use LibreChat’s conversation history or MCP tools directly; integration complexity; Mastra is still maturing
C: Mastra for orchestration only (no Mastra agents)	Use Mastra’s workflow engine as a scheduler/dispatcher; agents still run in LibreChat	Awkward integration — Mastra workflows call LibreChat API, but Mastra’s own agent primitives go unused; two systems for the price of one
D: TanStack Start for GUI only	Modern, type-safe React framework; good for the admin dashboard	Doesn’t solve orchestration; additional framework to learn and maintain; RC status means potential breaking changes

5.3 Recommendation

Option A for orchestration (custom scripts). Option D for the GUI (TanStack Start), when the GUI phase begins.

Mastra’s workflow engine is compelling but introduces a second agent runtime. The orchestration we need is relatively simple: schedule → discover task → invoke agent → monitor → handle failure. This doesn’t require a full workflow engine. Custom scripts calling the LibreChat Agents API give us exactly what we need with minimal overhead.

When LibreChat ships native background agents and scheduling, we can retire the custom scripts and use LibreChat’s built-in features. Mastra would add a migration path that’s harder to reverse.

For the GUI, TanStack Start is the right call — it’s TypeScript-native, Vite-based, and gives us SSR + server functions for a dashboard that manages cron jobs, projects, skills, and todo lists.

5.4 Decision

Supported by investigation — revised. The UI Chat API’s resumable model simplifies the orchestration logic further. Custom scripts are even more appropriate now — the pattern is simply: dispatch (POST) → poll status (GET) → subscribe for events (optional SSE) → handle completion. No need for complex connection management. The main new requirement is JWT lifecycle management (login → refresh → re-login), which is straightforward to implement.

6. Filesystem vs. Database for State

R4, R8, R9: Related Requirements

6.1 Finding

Current Uvilo OS architecture is entirely filesystem-based: markdown files in a git repo, read and written by agents. This works well for:

Single-user access (Erik)
Human readability (markdown files)
Version control (git history)
Simplicity (no database to manage)

The orchestration requirements introduce new access patterns:

Concurrent access — multiple agents may read/write state simultaneously
Structured queries — “list all projects in Research phase”, “find blocked todo items”
Fast lookups — “what’s the current phase of project X?” without reading a file
Atomic updates — updating a todo item state without race conditions

Research consensus: filesystem works for single-agent, single-user scenarios. Database becomes essential with concurrent access, structured queries, and shared state. A hybrid approach (filesystem interface for agents, database underneath) is increasingly popular.

For Uvilo OS specifically:

Project/skill discovery (R2, R3) is read-only — filesystem works fine, and the Typesense index already provides structured search.
Todo/checklist (R4) needs concurrent read-write — this is where a database adds real value.
Project state (R8) is currently markdown files updated by one agent at a time — filesystem works, but orchestration may have multiple agents updating state.

6.2 Options

Approach	Pros	Cons
A: Stay filesystem-only	No new infrastructure; consistent with current architecture; human-readable	Race conditions with concurrent agents; poor query performance; no atomic updates for todo items
B: Move everything to a database	ACID transactions; fast queries; concurrent access; scalable	Lose human-readability; need migration; more infrastructure; breaks current agent workflows that read/write markdown
C: Hybrid — filesystem for documents, database for structured state	Best of both; agents read project docs from filesystem (as today); todo/checklist and orchestration state in a database; incremental migration	Two storage systems to manage; need to keep them in sync

6.3 Recommendation

Option C: Hybrid. Keep markdown files for project documents (Requirements, Spec, Plans, State, Learnings — these are human-reviewed artifacts). Add a lightweight database for:

Todo/checklist items (R4)
Orchestration state (which agent is running, last heartbeat, Ralph loop status)
Project phase index (mirrored from Phase files for fast queries)

Database choice: We already have a Neon Postgres database available. Use it with a simple schema — no need for a separate database service. Drizzle ORM for type-safe TypeScript access.

The Discover MCP (R2, R3) can remain filesystem-backed — it’s read-only and the filesystem is the source of truth for project structure. The Todo MCP (R4) writes to Postgres.

6.4 Decision

Supported by investigation — revised. The UI Chat API handles conversation persistence automatically. The database is still needed for orchestration state (which agents are running, job tracking, retry counts) and todo/checklist items, but NOT for message persistence (the UI API does this natively).

7. Sub-Agent Transcript Review

R7: Related Requirement

7.1 Finding — Revised Based on Investigation (Tasks 2–6)

The original research assumed that LibreChat’s built-in conversation storage would automatically capture sub-agent transcripts. This is partially correct — it depends on which API is used.

The OpenAI-compatible Agents API (/v1/chat/completions and /v1/responses) is effectively stateless. It proxies to the LLM and returns the response, but does NOT persist conversations or messages to MongoDB.

API	Creates conversation doc?	Saves messages?	Title auto-set?
Chat Completions (no conversation_id)	❌ No	❌ No	N/A
Chat Completions (with conversation_id)	Uses existing	❌ No — still 0 messages	Preserves existing
Responses API (no previous_response_id)	✅ Yes — creates doc	❌ No — 0 messages	Agent display name
Responses API (store: true)	✅ Yes — creates doc	❌ No — store is broken	Agent display name
UI Chat API (`/chat/agents`)	✅ Yes	✅ Full persistence	✅ Auto-generated from content

The UI Chat API solves the transcript review problem completely:

conversationId is returned immediately in the JSON response
Both user and assistant messages (including tool call details) are persisted to MongoDB
Titles are auto-generated from conversation content (e.g., “Remembering TestUser Name”, “Forge Skill Summaries”)
Conversations appear in Chat History UI and are queryable via /api/convos endpoint
Conversation continuation works with conversationId + parentMessageId — the controller loads history from MongoDB automatically

Conversation continuity via the UI Chat API: Pass the same conversationId and set parentMessageId to the previous assistant’s messageId. The controller loads the full conversation history from MongoDB automatically — no need to pass full message history in the request.

Neither OpenAI-compatible API returns a conversation ID. The conversationId is generated server-side but never exposed in the response body, headers, or SSE stream events.

7.2 Options

Approach	Pros	Cons
D: Use UI Chat API for orchestration	Full transcript review in Chat History UI; auto-generated titles; conversation continuity built-in; no MongoDB manipulation needed	JWT auth (15 min) requires refresh logic; no API key support
B: Orchestrator maintains its own transcript log (file or DB)	No MongoDB dependency; simpler	Separate from Chat History UI; duplicates storage; harder to review
C: Skip transcript review for now	No extra work	Loses audit trail; can’t review what sub-agents did

7.3 Recommendation

Option D: Use the UI Chat API for orchestration. This eliminates all the complexity of Options A–C:

Start chat via POST /api/agents/chat/agents → get {conversationId, streamId} immediately
Subscribe to SSE at GET /api/agents/chat/stream/:streamId for real-time events (optional)
Check status via GET /api/agents/chat/status/:conversationId (for polling)
Continue conversation by passing conversationId + parentMessageId in subsequent calls
Review transcript in Chat History UI or via /api/convos endpoint

No MongoDB manipulation needed. No manual message persistence. Auto-generated titles. Full disconnect resilience.

The only downside is JWT auth — the orchestrator must implement:

Login with email/password → store JWT + refreshToken
Before JWT expires (15 min), call /api/auth/refresh with refreshToken cookie → get new JWT
If refresh fails (7-day expiry exceeded), re-login

This is straightforward to implement and far less complex than the MongoDB manipulation required by Option A.

7.4 Decision

Supported by investigation — resolved. The UI Chat API (/chat/agents) provides full transcript persistence, auto-generated titles, conversation continuity, and disconnect resilience out of the box. No MongoDB manipulation is needed. The orchestrator only needs to implement JWT lifecycle management (login → refresh → re-login). This is far simpler than the original Option A (pre-create conversations in MongoDB).

8. SRP Agent Design

R10: Related Requirement

8.1 Finding

Single-Responsibility Principle agents are small, focused agents that do one thing well. Benefits:

Smaller system prompts (less context window usage)
More predictable behavior (narrow scope = fewer failure modes)
Composability (complex agents call SRP agents as tools)
Easier to test and debug

Candidate SRP agents based on existing Forge skills:

Create Page — creates a markdown file with proper frontmatter
Update Sidebar — adds/updates sidebar entries in astro.config.mjs
Build & Deploy — runs Astro build and deploys to Vercel
Git Commit — stages changes, commits with conventional message, pushes
Project Phase Check — reads project phase files and reports status
Todo Item Creator — creates a todo item in the database

Investigation constraint (using OpenAI-compatible API): Each SRP agent invocation must complete within the ~4-minute timeout budget (see Section 9). Agents that typically run longer must be designed to operate in smaller units of work, or the orchestrator must decompose the task into multiple sequential API calls.

Investigation constraint (using UI Chat API): No strict time budget — agents run independently in the background. The orchestrator can poll for status or subscribe to SSE events. Long-running tasks are not a concern for the UI Chat API.

8.2 Options

Approach	Pros	Cons
A: Define SRP agents in LibreChat Agent Builder	Uses existing infrastructure; agents appear in chat; transcript review works	Agent definitions are in LibreChat’s MongoDB, not in the git repo; harder to version control
B: Define SRP agents in code (TypeScript config)	Version-controlled; reproducible; can be deployed programmatically	Need to build/configure agents via API rather than UI

8.3 Recommendation

Option B with a migration path. Define agent configurations as TypeScript objects (name, instructions, tools, capabilities) in the orchestrator codebase. On deploy, the orchestrator calls LibreChat’s agent configuration API to create/update agents. This keeps agent definitions in git and makes them reproducible across environments.

However, the Agent CRUD API (Task 7) has not been investigated yet. If LibreChat lacks a functional agent configuration API, start with Option A (manual creation in Agent Builder) and track programmatic deployment as a future improvement.

8.4 Decision

Partially supported — upgraded. The SRP agent concept is validated. The UI Chat API removes the 4-minute time budget constraint, allowing longer-running SRP agents. The deployment method can use LibreChat’s Agent CRUD API (available at /api/agents/v1/ with JWT auth) to create/update agents programmatically. Start with Option A (manual creation in Agent Builder) for initial setup, migrate to Option B (programmatic via CRUD API) when the orchestrator is built.

9. Agents API Operational Profile

R5, R6: Related Requirements

New section — based on empirical investigation (Tasks 1–5).

9.1 Execution Model — Revised

The OpenAI-compatible Agents API (/v1/chat/completions, /v1/responses) is fundamentally synchronous — the client must hold the HTTP connection for the entire agent execution. However, the UI Chat API (/chat/agents) uses the ResumableAgentController which runs generation independently in the background.

For orchestration, the UI Chat API is the recommended execution path (see Section 11). The synchronous constraints below apply only to the OpenAI-compatible API.

OpenAI-compatible API (synchronous):

No background/async execution mode
background: true parameter in Responses API is hardcoded to false in buildResponse()
Client disconnect triggers AbortController.abort(), immediately cancelling the agent run
No fire-and-forget pattern; no way to retrieve results after disconnection

UI Chat API (resumable):

Generation runs independently — client receives {streamId, conversationId, status: "started"} immediately
Client disconnect does NOT cancel the agent run
SSE stream subscription is separate from generation start
Reconnection supported via ?resume=true query parameter
Partial response saved if all SSE subscribers disconnect
Abort supported via POST /api/agents/chat/abort

9.2 Timeout Budget

Scenario	Max Safe Duration	Notes
Non-streaming	~4 minutes (240s)	Tested up to 264s successfully; Railway proxy likely has 300s limit
Streaming (moderate response)	~70s / ~400KB SSE	Works reliably within this range
Streaming (large response)	~180s / >500KB SSE	CDN/proxy buffering may cause failures
LibreChat agent loop	No built-in timeout	Relies on LLM provider timeout (typically 10 min)

Recommendation: Use the UI Chat API for all orchestrator-driven agent calls — no timeout budget concerns. If using the OpenAI-compatible API as fallback, use non-streaming mode with a client-side timeout of 240s. If a task may exceed 4 minutes, decompose it into multiple API calls.

9.3 Streaming Behavior — Revised

OpenAI-compatible API: The SSE stream is COMPLETELY SILENT during tool execution. No events are sent until the agent has finished all tool calls and produced its final response. The stream: true flag only affects how the final LLM response is delivered, not intermediate tool execution steps.

For the Responses API, streaming events are replayed in a burst after completion — the events look like real-time progress but are all emitted in <1 second after the entire execution finishes. This is a significant UX trap.

UI Chat API: Streaming IS REAL-TIME. Events arrive progressively during agent execution:

on_run_step events fire as tool calls begin and complete
on_message_delta events deliver token-by-token content as it’s generated
Tool call arguments stream incrementally via on_run_step_delta
No silence period — the orchestrator can monitor progress in real-time

Implication for orchestration: The UI Chat API’s real-time streaming enables progress monitoring, stall detection, and the Ralph Loop. The orchestrator can detect when an agent is stalled (no events for N seconds) and take action.

9.4 Disconnect Resilience — Revised

OpenAI-compatible API: Client disconnect = immediate agent cancellation with total data loss. No partial results are saved. The orchestrator MUST:

Maintain its HTTP connection for the entire sub-agent execution
Implement retry logic at the orchestrator level (not API level)
Design for idempotency — re-running an agent call should be safe
Use the Ralph Loop pattern: detect stall → kill → restart with fresh context from filesystem state

UI Chat API: Client disconnect does NOT cancel the agent. Generation continues independently. Partial results are saved if all SSE subscribers disconnect. The orchestrator can:

Disconnect after starting generation — the agent continues
Reconnect later via SSE ?resume=true to get sync state + missed events
Poll status via GET /api/agents/chat/status/:conversationId
Abort cleanly via POST /api/agents/chat/abort

This eliminates the single biggest risk of the OpenAI-compatible API — the orchestrator no longer needs to maintain persistent connections for the entire agent execution.

9.5 Parameter Control

The orchestrator has minimal control over agent behavior through API parameters:

Parameter	Chat Completions	Responses API	Actually Honored?
`model` (agent ID)	✅	✅	✅ Yes
`messages` / `input`	✅	✅	✅ Yes
`stream`	✅	✅	✅ Yes
`conversation_id`	✅ (LibreChat ext)	—	✅ Yes (but not returned)
`temperature`	✅ (no error)	✅ (validated)	❌ Likely overridden by agent config
`max_tokens` / `max_output_tokens`	✅ (no error)	✅ (validated)	❌ Likely overridden
`response_format` / `text.format`	✅ (no error)	❌ Hardcoded	❌ Not enforced
`instructions` (override)	—	✅ (reflected)	❌ Agent’s own instructions take precedence
`tools` (override)	—	❌ Hardcoded	❌ Agent uses its own configured tools

Key insight: Unknown/extra parameters are silently ignored — no errors are returned. The agent’s own model_parameters likely override per-request params. The orchestrator should NOT attempt to control agent behavior through API parameters; instead, encode all behavior in the agent’s configuration and the prompt.

9.6 Recommended Orchestrator Call Pattern — Revised

Using the UI Chat API (recommended):

1. Ensure valid JWT (login or refresh)
2. POST /api/agents/chat/agents:
   - text: task prompt
   - endpoint: "agents"
   - agent_id: target agent
   - conversationId: "new" | existing UUID
   - parentMessageId: previous assistant messageId (for continuation)
3. Response: {streamId, conversationId, status: "started"}
4. Optionally subscribe to SSE: GET /api/agents/chat/stream/:streamId
   - Real-time progress events
   - Stall detection via event gap monitoring
5. Or poll status: GET /api/agents/chat/status/:conversationId
6. On completion: parse final event, update orchestration state
7. On stall: abort via POST /api/agents/chat/abort, then restart (Ralph Loop)

Using the OpenAI-compatible API (fallback):

1. Pre-create conversation in MongoDB (UUID + descriptive title)
2. Build messages[] with full context (state, history, instructions)
3. Call Chat Completions API:
   - model: agent_id
   - messages: [context + task]
   - conversation_id: pre-created UUID
   - stream: false
   - client timeout: 240s
4. On success:
   - Save user message + assistant response to MongoDB messages collection
   - Update conversation's messages array
   - Parse response, update orchestration state
5. On timeout/disconnect:
   - Log failure, increment retry count
   - If retries < max: restart with fresh context (Ralph Loop)
   - If retries exhausted: mark task as 🚫, alert user

10. Agent Handoffs — Programmatic Viability

R5: Related Requirement

New section — Task 6 deferred, this section contains preliminary findings.

10.1 What We Know

Agent Handoffs work within the LibreChat UI — agents delegate to specialized agents via discoverConnectedAgents (BFS when primaryConfig.edges?.length > 0)
Handoff configuration is set in the Agent Builder UI (target agents)
The Agents API source code includes handoff discovery logic in the controller

10.2 What We Know (Updated from Task 6)

Agent Handoffs work within the LibreChat UI — agents delegate to specialized agents via discoverConnectedAgents (BFS when primaryConfig.edges?.length > 0)
Handoff configuration is set in the Agent Builder UI (target agents)
The Agents API source code includes handoff discovery logic in the controller
The UI Chat API (/chat/agents) supports handoffs — it uses the same initializeAgent function that includes discoverConnectedAgents
The OpenAI-compatible API also includes handoff logic in its controller
Handoffs are a server-side execution mechanism — when an agent decides to hand off, the target agent runs in the same generation job

10.3 Impact on Architecture

Since the UI Chat API supports handoffs:

Interactive delegation (user → Forge Agent → SRP agent) can use handoffs naturally
Scheduled delegation uses direct API calls to specific agents (no handoff needed)
The orchestrator doesn’t need to implement its own delegation logic — it can either:
- Call a specific SRP agent directly (simpler, recommended)
- Call a router agent that uses handoffs to delegate (more flexible but less predictable)

10.4 Decision

Supported by investigation. Handoffs work through both the UI Chat API and the OpenAI-compatible API. For orchestration, direct API calls to specific agents are recommended (simpler, more predictable). Handoffs are an optimization for interactive sessions where the user’s agent decides which specialist to call.

11. UI Chat API — Resumable Execution Profile

R5, R6, R7: Related Requirements

New section — based on empirical investigation (Task 6).

11.1 Endpoint

POST /api/agents/chat/agents — starts an agent generation that runs independently of the HTTP connection.

Auth: JWT session token (Bearer header) + refreshToken cookie. API key auth does NOT work.

Request body:

{
  "text": "task prompt",
  "endpoint": "agents",
  "agent_id": "agent_<id>",
  "conversationId": "new" | "<uuid>",
  "parentMessageId": "00000000-0000-0000-0000-000000000000" | "<msg-id>"
}

Response (immediate):

{"streamId": "<uuid>", "conversationId": "<uuid>", "status": "started"}

streamId === conversationId always. UUID format.

11.2 Support Endpoints

Endpoint	Method	Purpose
`/api/agents/chat/stream/:streamId`	GET (SSE)	Subscribe to real-time events
`/api/agents/chat/stream/:streamId?resume=true`	GET (SSE)	Reconnect with sync state + replay
`/api/agents/chat/status/:conversationId`	GET	Check if generation is active
`/api/agents/chat/active`	GET	List active job IDs for current user
`/api/agents/chat/abort`	POST	Abort a running generation

11.3 SSE Event Types (Real-Time)

Event	Content
`created`	User message details (messageId, conversationId)
`on_run_step`	Step started (tool call, message creation)
`on_run_step_delta`	Incremental step updates
`on_run_step_completed`	Step finished
`on_message_delta`	Token-by-token content: `delta.content[{type, text}]`
`final`	Full conversation object + response message

Events arrive in real-time as they happen — NOT replayed in a burst.

11.4 Persistence

User messages saved to MongoDB messages collection
Assistant messages saved with full content array (text + tool_call items)
Conversations saved to MongoDB conversations collection with auto-generated title
Chat History UI shows all conversations with full transcripts
/api/convos endpoint returns paginated conversation list

11.5 Conversation Continuation

Pass conversationId + parentMessageId (the assistant’s messageId from previous exchange). The controller loads the conversation history from MongoDB automatically. No need to include full message history in the request.

Tested: Agent remembered “TestUser” across two exchanges via conversation continuation. ✅

11.6 Disconnect Resilience

Generation continues independently after client disconnect. Partial response is saved if all SSE subscribers disconnect. Reconnection via ?resume=true replays missed events and syncs state.

Tested: Disconnected after 3 seconds of active streaming; generation completed with full response saved to MongoDB. ✅

11.7 Abort

POST /api/agents/chat/abort with {conversationId} returns {success: true, aborted: "<id>"}. Generation stops immediately. ✅

11.8 Auth Lifecycle

Item	Value
JWT validity	15 minutes
Refresh token validity	7 days
Login	`POST /api/auth/login` with `{email, password}`
Refresh	`POST /api/auth/refresh` with refreshToken cookie (no Authorization header)
Refresh returns	New JWT + new refreshToken cookie (rotation)

Refresh flow: Login → store JWT + refreshToken cookie → before 15 min expiry, call /api/auth/refresh with cookie → get new JWT + cookie → repeat. If refresh fails (7-day expiry exceeded), re-login.

User-Agent requirement: The uaParser middleware flags non-browser User-Agents (e.g., Node.js fetch, curl). After ~40 violations, the account receives a 2-hour non_browser ban (stored in MongoDB logs collection with keys ban:<userId> and BANS:<ip>). The orchestrator must send a browser User-Agent header on all requests to avoid being banned.

Role requirement: The service account must have ADMIN role. USER role accounts get 403 “Insufficient permissions to access this agent” — the canAccessAgentFromBody middleware checks PermissionBits.VIEW, which USER role lacks. Agent access is also per-agent (ACLs).

11.9 Long-Running Execution

Test with Forge GLM 5.1 [Master] (filesystem tools, 5 file reads):

24 seconds total execution
291 total SSE events
5 tool call steps (visible in real-time)
267 content delta events (token-by-token streaming)
All messages persisted with tool_call content parts ✅

No timeout budget concerns — generation runs independently in the background.

11.10 Implications for Architecture

The UI Chat API replaces the OpenAI-compatible API as the primary orchestration mechanism. It solves every deal-breaker:

✅ conversationId returned immediately
✅ Full message persistence (user + assistant + tool calls)
✅ Auto-generated titles
✅ Real-time streaming (not burst)
✅ Disconnect resilience (generation continues independently)
✅ Reconnection support
✅ Clean abort mechanism
✅ Conversation continuation via conversationId/parentMessageId
✅ No timeout budget concerns

Only downside: JWT auth (15 min) requires refresh logic. This is straightforward to implement and far less complex than the MongoDB manipulation required by the original plan.

12. Database Choice — Postgres on Railway VectorDB

R4, R8, R9, R11: Related Requirements

12.1 Finding

Plan 2 Task 4 originally specified MongoDB (using the existing LibreChat MongoDB instance) for orchestration state. Three database options were evaluated:

Option	Description	Latency (query)	Latency (connect)	Cost
A: Railway VectorDB (Postgres)	Existing Postgres on Railway private network	~2-3ms	~15ms	Already provisioned
B: Neon Serverless Postgres	Existing Neon project (`falling-silence-90558137`) hosting Uvilo app data	~19-20ms	~120ms	Already provisioned
C: Railway MongoDB	Existing MongoDB instance used by LibreChat	~2-5ms (private network)	~10ms	Already provisioned

Empirical latency benchmarks (run from Railway workspace, 50 queries, persistent connection):

Metric	Railway VectorDB	Neon Serverless	Ratio
Connect	15.1ms	120.9ms	8×
SELECT 1 avg	2.27ms	18.92ms	8.3×
SELECT 1 p50	2.03ms	18.89ms	9.3×
INSERT avg	3.22ms	20.07ms	6.2×
INSERT p99	6.58ms	24.20ms	3.7×
SELECT by PK avg	2.51ms	19.14ms	7.6×
SELECT by PK p99	7.26ms	21.15ms	2.9×

Neon’s ~120ms connect time is due to SSL handshake + Neon’s cold-start proxy. The round-trip to us-west-2.aws.neon.tech from Railway adds consistent ~18-19ms per query regardless of operation. Railway VectorDB’s ~2-3ms comes from private-network access — no internet hop, no SSL termination for internal traffic.

MongoDB vs Postgres comparison:

Aspect	MongoDB	Postgres
Schema enforcement	Flexible but no enforcement — bad data silently accepted	Strong typing, CHECK constraints, ENUMs
Relational queries	Requires `$lookup` pipelines, awkward	Native JOINs
Transactions	Multi-document transactions possible but heavyweight	ACID by default
JSON support	Native	`jsonb` — indexed, queryable, same flexibility for metadata
Connection model	One persistent connection per service	Connection pooling (`pg-pool`) — handles bursty loads better
Vector search	Requires Atlas Vector Search (separate)	pgvector already installed on VectorDB

Neon database existing content: The Neon uvilo database hosts ~45 Prisma-managed tables for the Uvilo app (User, Convo, AiMessage, Execution, Taxonomy, ResourceItem, etc.). Mixing orchestration operational data into the app’s production database risks cross-concerns and adds latency.

Railway VectorDB existing content: The VectorDB Postgres instance currently hosts RAG API embeddings (pgvector with halfvec, sparsevec, vector types + HNSW/IVFFlat indexes). The database name is railway (default). Creating a separate forge database on the same instance provides clean isolation from RAG data while keeping the same private-network latency.

12.2 Options

Approach	Pros	Cons
A: Railway VectorDB (separate `forge` database)	8-9× lower latency; already provisioned; private network; pgvector available; clean separation from RAG data	Two databases on same Postgres instance share memory/CPU
B: Neon (new schema in `uvilo` database)	Existing managed service; branching for dev/preview	8-9× higher latency; mixed with app production data; Neon cold-start overhead
C: Railway MongoDB (original Plan 2 spec)	Already used by LibreChat; document model; same private network	No schema enforcement; no relational queries; orchestration data mixed with LibreChat conversation data

12.3 Recommendation

Option A: Railway VectorDB with a separate forge database.

Rationale:

8-9× lower latency — orchestrator queries job status frequently (polling, Ralph Wiggum monitoring). Each Neon query adds ~20ms vs ~2-3ms on VectorDB.
Zero cost increase — VectorDB is already provisioned. Adding a forge database is negligible overhead.
Private network — no egress, no SSL overhead, no cold-start delays.
pgvector already installed — future-proof if orchestration needs semantic matching.
Separate database — DROP DATABASE forge resets orchestration without touching RAG data. Simpler permissions.
Neon is for the app — mixing orchestration operational data into the Uvilo app’s production database risks cross-concerns.
MongoDB is for LibreChat — conversation data lives there. Orchestration state (job tracking, phase index) is operationally distinct and benefits from Postgres’s schema enforcement and relational queries.

12.4 Decision

Supported by investigation. Use Railway VectorDB Postgres with a separate forge database. Adopt Neon naming conventions (see §12.5).

12.5 Naming Conventions (Aligned with Neon `uvilo` Database)

The Neon uvilo database uses Prisma-managed conventions. The forge database must align to maintain consistency across the Uvilo stack.

Convention rules:

Element	Convention	Example
Table names	PascalCase	`AgentJob`, `Project`
Column names	camelCase	`conversationId`, `startedAt`
Index names	`Table_column_idx`	`AgentJob_conversationId_idx`
Primary keys	`TEXT` with Cuid2 ID (3-letter prefix + `_` + 24-char Cuid2 body)	`agt_cmm6650d9001r04la64y4m6cv`
Timestamps	`TIMESTAMPTZ` with `DEFAULT NOW()`	`createdAt`, `updatedAt`
Enums	`TEXT` with `CHECK` constraint	`CHECK (status IN ('running','completed','failed','aborted'))`
JSON columns	`JSONB` with `DEFAULT '{}'`	`metadata`

Existing ID prefix registry (Neon uvilo database — must not be reused):

Prefix	Table	Prefix	Table	Prefix	Table
`acc_`	Account	`apk_`	ApiKey	`aut_`	Author
`boo_`	Book	`bot_`	Bot	`bug_`	Bot (group?)
`chn_`	Channel	`chp_`	ChildPrompt	`cnv_`	Convo
`dom_`	LifeDomain	`epi_`	PodEpisode	`evt_`	Event
`exe_`	Execution	`fac_`	Factoid	`goo_`	GoodreadsList
`hab_`	Habit	`img_`	CloudImage	`itl_`	ItemLabel
`jrn_`	Journey	`lbl_`	Label	`log_`	ExecutionLog
`msg_`	AiMessage	`not_`	Notification	`pod_`	Podcast
`prm_`	Prompt	`qur_`	QuizResult	`qqr_`	QuizQuestionResult
`quq_`	QuizQuestion	`qus_`	QuizSection	`quz_`	Quiz
`rea_`	ResAuthor	`rel_`	ResourceLink	`ret_`	ResTaxonomy
`ses_`	Session	`set_`	Setting	`spc_`	Space
`spu_`	SpaceUser	`tam_`	TaxonomyMap	`tax_`	Taxonomy
`tra_`	Transcript	`tsl_`	TaxonomySearchLog	`usr_`	User
`ver_`	Verification	`vid_`	Video

Reserved prefixes that must NOT be used: acc, apk, aut, boo, bot, bug, chn, chp, cnv, dom, epi, evt, exe, fac, goo, hab, img, itl, jrn, lbl, log, msg, not, pod, prm, qur, qqr, quq, qus, quz, rea, rel, ret, ses, set, spc, spu, tam, tax, tra, tsl, usr, ver, vid

Forge database table and prefix assignments:

Table	Prefix	Example ID
`AgentJob`	`agt_`	`agt_cmm6650d9001r04la64y4m6cv`
`Project`	`prj_`	`prj_cmm6650d9001r04la64y4m6cv`

12.6 Schema Design

-- Connect to VectorDB and create the forge database
CREATE DATABASE forge;

\c forge;

-- Agent execution tracking (replaces Plan 2's orchestration_jobs)
CREATE TABLE "AgentJob" (
  id              TEXT PRIMARY KEY DEFAULT 'agt_' || gen_random_uuid(),
  conversationId  TEXT,
  agentId         TEXT NOT NULL,
  taskPrompt      TEXT,
  status          TEXT NOT NULL DEFAULT 'running'
                  CHECK (status IN ('running','completed','failed','aborted')),
  startedAt       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completedAt     TIMESTAMPTZ,
  retryCount      INT DEFAULT 0,
  parentJobId     TEXT REFERENCES "AgentJob"(id),
  metadata        JSONB DEFAULT '{}',
  createdAt       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updatedAt       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Project phase tracking (replaces Plan 2's orchestration_phase_index)
CREATE TABLE "Project" (
  id              TEXT PRIMARY KEY DEFAULT 'prj_' || gen_random_uuid(),
  projectName     TEXT NOT NULL UNIQUE,
  department      TEXT NOT NULL,
  phase           TEXT NOT NULL,
  status          TEXT NOT NULL DEFAULT 'draft'
                  CHECK (status IN ('draft','review','approved','published','archived')),
  updatedAt       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  createdAt       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Indexes
CREATE INDEX "AgentJob_conversationId_idx" ON "AgentJob" (conversationId);
CREATE INDEX "AgentJob_status_idx" ON "AgentJob" (status);
CREATE INDEX "AgentJob_agentId_idx" ON "AgentJob" (agentId);
CREATE INDEX "Project_department_idx" ON "Project" (department);
CREATE INDEX "Project_status_idx" ON "Project" (status);

Notes on the schema:

id is TEXT (not UUID) — Cuid2 strings generated in the TypeScript application layer using the cuid2 npm package. The SQL DEFAULT uses gen_random_uuid() as a fallback only; the app should always supply the ID.
parentJobId in AgentJob is a self-referencing foreign key for sub-agent tracking (matches Plan 2 spec).
Project.projectName has a UNIQUE constraint — one record per project (matches Plan 2’s unique index on project_name).
Project.status uses the same status values as Uvilo OS document frontmatter: draft, review, approved, published, archived.
Index naming follows the Neon convention: Table_column_idx.
AgentJob replaces orchestration_jobs; Project replaces orchestration_phase_index. No orchestration_ prefix needed since the database itself is named forge.
LibreChat stores conversation content in MongoDB. The conversationId in AgentJob is a reference key — the actual messages are not duplicated. Orchestration state and conversation data are inherently separated by storage system.

12.7 Connection Details

Property	Value
Host	`vectordb.railway.internal`
Port	`5432`
Database	`forge`
User	`postgres`
Password	(from Railway VectorDB env vars: `POSTGRES_PASSWORD`)
Internal URL	`postgresql://postgres:{password}@vectordb.railway.internal:5432/forge`
npm package	`pg` or `postgres` (lightweight alternative)

12.8 Impact on Plan 2

Plan 2 Task 4 changes:

Remove mongodb dependency → Add pg (or postgres) dependency
Remove MONGO_URI env var → Add FORGE_DB_URL env var
Remove orchestration_ prefix on collections → Tables are AgentJob and Project in forge database
Change src/db.ts from MongoDB client to Postgres client
Change helper functions from MongoDB operations to SQL queries
Add Cuid2 ID generation in TypeScript (using cuid2 npm package with 3-letter prefix)
Connection pooling via pg-pool recommended for the orchestrator’s persistent service model

Section	Original Decision	Revised Decision	Status
1. LibreChat Current State	Pending	Supported by investigation — Three execution APIs identified; UI Chat API recommended	✅ Resolved
2. Architecture Pattern	Pending	Supported by investigation — revised — Supervisor + Ralph Loop confirmed; UI Chat API removes synchronous constraints; JWT auth required	✅ Resolved
3. Cron Scheduling	Pending	Supported by investigation — Railway cron service confirmed	✅ Resolved
4. MCP Tools	Pending	Supported by investigation — Two MCP servers confirmed	✅ Resolved
5. Mastra vs Custom	Pending	Supported by investigation — revised — Custom scripts confirmed; UI Chat API simplifies pattern further	✅ Resolved
6. Filesystem vs DB	Pending	Supported by investigation — revised — Hybrid confirmed; DB not needed for message persistence (UI API handles it)	✅ Resolved
7. Sub-Agent Transcripts	Pending	Supported by investigation — resolved — UI Chat API provides full persistence; no MongoDB manipulation needed	✅ Resolved
8. SRP Agent Design	Pending	Partially supported — upgraded — SRP concept validated; UI API removes 4-min constraint; CRUD API available	✅ Resolved
9. API Operational Profile	N/A (new)	Established — revised — Synchronous constraints only for OpenAI API; UI Chat API is resumable with no timeout budget	✅ Resolved
10. Agent Handoffs	N/A (new)	Supported by investigation — Handoffs work through all APIs; direct calls recommended for orchestration	✅ Resolved
11. UI Chat API Profile	N/A (new)	Established — Resumable execution, real-time streaming, full persistence, disconnect resilience	✅ New
12. Database Choice — Postgres on Railway VectorDB	Pending	Supported by investigation — Railway VectorDB chosen over MongoDB and Neon; separate `forge` database; Neon naming conventions adopted	✅ Resolved