Forge Infrastructure

How Forge’s infrastructure works: architecture, deployment, services, environment, and tool-specific operational knowledge.

Terminology

Term	What it is
uvilo-mono	The monorepo. Contains multiple apps and packages. Checked out at prodRepoRoot.
forgentic	An app within the uvilo-mono monorepo. Deployed on Railway. This is the runtime environment the Forge agentic system currently runs in.

Usage is context-dependent: use “uvilo-mono” when referring to the repository or codebase, and “forgentic” when referring to the deployed application or runtime environment on Railway. Where both names appear together, simplify to the context-appropriate single term.

Architecture

Railway Project: chat-uvilo-os
├── forgentic runtime (Bot execution engine)
│   ├── Bot system — agents with prompts, MCP tools, and model configuration
│   ├── Tool model — tools may be eager (always available) or deferred (loaded on demand)
│   │   Use `toolSearch` to find available tools and `loadTool` to activate deferred ones
│   ├── Volume: /workspace (persistent)
│   │   ├── /workspace/erik@uvilo.com/uvilo-os/  (git working copy — any branch)
│   │   └── /workspace/.trash/                    (trashed files)
│   └── MCP Servers (attached to Bots):
│       ├── forge-bash — TypeScript MCP (shell execution)
│       ├── forge-filesystem — TypeScript MCP (file read/write/search)
│       ├── uvilo-trash — TypeScript MCP (safe file deletion)
│       ├── forge-discovery — TypeScript MCP (skill/project discovery, Stdio transport)
│       ├── vercel — npx vercel-mcp
│       ├── github — npx @modelcontextprotocol/server-github
│       ├── context7 — npx @upstash/context7-mcp
│       ├── railway — npx @jasontanswe/railway-mcp
│       ├── neon — npx @neondatabase/mcp-server-neon
│       ├── notion — npx @notionhq/notion-mcp-server
│       ├── linear — npx @touchlab/linear-mcp-integration
│       ├── posthog — npx mcp-remote
│       ├── playwright — SSE transport
│       └── suprsend (dev/staging/prod) — Python3 proxy
├── Playwright MCP (image: mcr.microsoft.com/playwright/mcp)
│   ├── Headless Chromium browser for AI-driven web browsing
│   ├── Serverless: enabled (sleeps after 10 min, wakes on traffic, ~5-15s cold start)
│   ├── Internal only: http://playwright-mcp.railway.internal:8931/sse
│   └── Connected to forgentic runtime via SSE MCP transport (NOT streamable-http)
├── Orchestrator (custom image: Railway web service, TypeScript/Hono)
│   ├── Dispatches agents via Bot Chat API
│   ├── API-key authentication (service account)
│   ├── Inngest scheduling (cron → POST /orchestrator/run)
│   ├── forge-discovery MCP (project/skill discovery, Stdio transport)
│   ├── AgentJob table in forge Postgres database (status tracking)
│   ├── Reaper cron (*/15 min — cleans up stuck jobs)
│   └── Source: orchestrator/ in repo root
├── Typesense (via Railway — persistent volume)
│   └── Search index for repo docs and website DocSearch
├── forge Postgres (via Railway — persistent volume)
│   └── AgentJob table, orchestration state, agent configs, conversation history, user data
└── Inngest (external — inngest.com)
    ├── Scheduled function execution (Project Runner hourly, Task Runner hourly, etc.)
    └── POSTs to orchestrator on schedule

⚠ Always deploy stateful services via Railway “Add Database” (or equivalent), not as bare Docker images. A bare container has no persistent volume — data is lost on every redeploy. Railway’s “Add Database” provisioner attaches a volume, enables backups, and provides a Database tab. See the update-infrastructure skill for the persistence checklist.

Config deployment model: Bot and MCP configuration is managed through the forgentic runtime’s Bot system. Each Bot has its own system prompt, model, and attached MCP tools. Configuration changes are deployed by updating Bot definitions and redeploying the runtime. Environment variables control service connectivity and API keys.

Startup model: The forgentic runtime runs as a Railway web service. On deployment, Railway builds from the connected GitHub repo and starts the service. The runtime loads Bot definitions, attaches MCP tools, and begins serving agent conversations. Redeploys pick up the latest code and configuration from the repo.

Railway config changes require a new deployment. service_update (start command, region, replicas, etc.) changes the service configuration but does not create a new deployment — the change sits pending until one is triggered. A dashboard “Redeploy” won’t pick it up either: it re-deploys the last successful deployment, which was built under the old config. To activate a config change, you must trigger a new deployment. For image-based services (no GitHub repo connected), deployment_trigger requires a commitSha that doesn’t exist — use variable_bulk_set (or variable_set) to set/change any env var instead, which forces a new deployment that picks up the pending config. Alternatively, the operator can click “Deploy” (not “Redeploy”) in the Railway dashboard.

Path Constants

Token	Value
ROOT	/workspace/erik@uvilo.com/uvilo-os
FORGE	{ROOT}/Forge
CONFIGS	{FORGE}/Configs
SKILLS	{FORGE}/Skills
OUTPUT	{FORGE}/Output
ORG	{ROOT}/Uvilo
PRODUCT	{ROOT}/Product

Any {LIKE_THIS} token in instructions must be substituted from these constants verbatim.

Railway IDs

Item	Value
Project	e9a90e20-12bf-41e9-b3de-2bbef30a1e6d
Production env	f7b0fc43-f3e5-4491-9360-92614bebd30c

Environment Variables

Reading Railway env vars

Railway env vars are NOT available via os.environ or echo $VAR in the runtime container. To read them:

Shell: cat /proc/1/environ | tr '\0' '\n' | grep VAR_NAME Python: open('/proc/1/environ').read().split('\0')

Do NOT use: os.environ, echo $VAR, os.getenv() — unreliable in the Railway container.

Key variables

Category	Key variables	Used by
LLM API keys	OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_KEY, OPENROUTER_KEY	Bot runtime, Forge Optimizer
Service tokens	SUPRSEND_SERVICE_TOKEN, RAILWAY_API_TOKEN, NEON_API_KEY, NOTION_TOKEN, LINEAR_ACCESS_TOKEN	MCP servers
Infrastructure	FORGE_DB_URL, TYPESENSE_URL, TYPESENSE_ADMIN_KEY	Postgres, Typesense
Orchestration	FORGE_DB_URL, INNGEST_SIGNING_KEY, INNGEST_EVENT_KEY, INNGEST_DEV	Orchestrator, forge DB, Inngest scheduling
Git credentials	GITHUB_TOKEN, GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL	uvilo-git, uvilo-git-remote
Vercel	VERCEL_API_KEY	vercel MCP
Email	EMAIL_HOST, EMAIL_USERNAME, EMAIL_PASSWORD	Postmark SMTP
Analytics	POSTHOG_API_KEY, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY	PostHog, Langfuse
Search	SERPER_API_KEY, TAVILY_API_KEY, FIRECRAWL_API_KEY, JINA_API_KEY	Web search, scraping

Model-Specific Quirks

Model-specific quirks (required parameters, rejected parameters, pricing) are documented in the Choose_AI_Model skill’s model reference files at Forge/Skills/Choose_AI_Model/Models/. Refer to the relevant model reference file before invoking a model programmatically. For use-case-based model selection (e.g., “the indexing use case”), see the Choose_AI_Model skill’s Use Case Table.

Tool-specific credential access

MCP servers receive credentials through environment variables set in the Bot/MCP configuration. Each MCP server reads its required API keys and tokens from process.env at startup. For Python MCP servers running in the Railway container, env vars are injected from /proc/1/environ at the top of the module before any os.environ or os.getenv() calls.

Manual Operations

Updating the forgentic Runtime

The forgentic runtime is deployed from the GitHub repo connected to the Railway service. To update:

1. Commit and push changes to the dev branch
2. Railway auto-deploys from the connected repo, or trigger a manual deploy via the Railway dashboard
3. Verify the deployment is healthy via the /health endpoint

SSH Access to Railway

To browse the filesystem and execute shell commands in a Railway service, you need to use the Railway CLI.

Authenticate Railway

The first time you try to use a railway command like railway link or railway ssh you will be prompted to go through an OAuth flow to sign in via a browser.

After authenticating you still need to set up an SSH key to use railway ssh. If it returns No SSH keys registered with Railway, do the following:

1. Generate a railway ssh key: ssh-keygen -t ed25519 -f ~/.ssh/railway_ed25519 -C "railway"
2. Open the generated key in ~/.ssh/railway_ed25519.pub
3. Copy the contents of the file, something like ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...long base64 blob... railway
4. In Railway go to Account Settings > SSH Keys: https://railway.com/account/ssh-keys
5. Under Add New Key give the key a name (the device you are using), and paste the content of the .pub file into Public Key
6. Now we need to make sure the Mac environment is aware of the key:

Host ssh.railway.com
    IdentityFile ~/.ssh/railway_ed25519
    IdentitiesOnly yes

You should now be able to use railway ssh, but not before you link it to a service on Railway.

Link to a service

railway link

> Select a workspace Erik Dakoda's Projects
> Select a project chat-uvilo-os
> Select an environment production
> Select a service <esc to skip>

Common Operations

Deploying Config Changes

1. Commit and push Bot/MCP configuration changes to the dev branch
2. Railway auto-deploys from the connected repo, or trigger a manual deploy via the Railway dashboard
3. For environment-variable-only changes, use variable_bulk_set (or variable_set) to set/change any env var, which forces a new deployment that picks up the pending config
4. Verify the deployment is healthy via the /health endpoint

SSH Access

The Railway CLI is not available in the Forge environment. Use forge-bash__run to run commands inside the runtime container instead.

User Management

User and Bot access is managed through the forgentic runtime’s Bot system. Bots are configured with specific tools, prompts, and model parameters. Access control is managed at the Bot level.

Vercel — Documentation Site

Component	Details
Project	uvilo-os (Uvilo team)
Framework	Astro Starlight
Root Directory	.internal/ (with “Include files outside root” enabled)
Preview URL	os.uvilo.com (dev branch)
Access Control	Standard Protection (team members only)
Env vars	PUBLIC_TYPESENSE_URL, PUBLIC_TYPESENSE_SEARCH_KEY — must be scoped to Preview (the site serves the dev branch preview build)

API Key

vercel-mcp reads the API key from a CLI argument (VERCEL_API_KEY=<key>), NOT from process.env. Pass as arg: args: ["-y", "vercel-mcp", "VERCEL_API_KEY=<key>"].

Preview URLs

Always use os.uvilo.com — it always shows the latest preview. Do not use per-deployment URLs.

Deployment Protection Bypass

Vercel Standard Protection requires authentication for preview deployments. Bypass method: query parameter on first navigation:

https://os.uvilo.com?x-vercel-protection-bypass=<secret>

The bypass secret is stored in the VERCEL_PROTECTION_BYPASS environment variable (also mirrored in GitHub repo secrets). The query parameter sets a bypass cookie, so subsequent navigations don’t need it.

DNS

Type	Name	Value
CNAME	os	9a5064d98cd8354e.vercel-dns-016.com.

Typesense

Setting	Value
Host	Read from TYPESENSE_URL env var
Admin key	Read from TYPESENSE_ADMIN_KEY env var
Search-only key	Read from TYPESENSE_SEARCH_KEY env var
Collections	uvilo (repo docs), uvilo_docs (website DocSearch)

⚠ Never hardcode Typesense hosts or API keys in source code. Always read from environment variables:

• In agent sessions: Use forge-typesense__search_knowledge tool (eager tool, always available)
• TypeScript scripts: Use process.env.TYPESENSE_URL
• Shell access: Use forge-bash__run and read from /proc/1/environ: cat /proc/1/environ | tr '\0' '\n' | grep TYPESENSE_URL
• Website build (Astro config): Use process.env.PUBLIC_TYPESENSE_URL in astro.config.mjs — NOT import.meta.env. Config files run in Node.js before Vite starts, so import.meta.env.PUBLIC_* silently resolves to undefined. Only Vite-transformed source files (.astro, .ts, .tsx components) can use import.meta.env. For Vercel builds, set via Vercel env vars (scoped to Preview). For local dev, copy .internal/.env.example to .internal/.env and fill in values (the file is gitignored).
• GitHub Actions: Use ${{ secrets.TYPESENSE_URL }}

These env vars must be set in:

• Railway env vars (for runtime container, read via /proc/1/environ)
• GitHub repo secrets (for CI workflows)
• Vercel project env vars (for website build)
• .internal/.env file (for local Astro dev server; copy from .internal/.env.example)

Indexing scripts: Forge/Typesense/Maintenance/index-department/ (per-department TypeScript CLI), .internal/scrape-site.py (website pages), .internal/backfill-summaries.py (backfill summaries). All read TYPESENSE_URL and TYPESENSE_ADMIN_KEY from /proc/1/environ. Run via forge-bash__run or the reindex-typesense skill.

Website search: Starlight DocSearch powered by Typesense (plugin: starlight-docsearch-typesense@1.0.1).

MCP Server Details

forge-bash

TypeScript MCP server. Source: Forge/Configs/MCP_Servers/forge-bash/. Provides forge-bash__run tool for shell command execution. Injects Railway env vars from /proc/1/environ at startup.

forge-filesystem

TypeScript MCP server. Provides forge-filesystem__read_text_file, forge-filesystem__write_file, forge-filesystem__edit_file, forge-filesystem__list_directory, forge-filesystem__search_files tools.

uvilo-trash

TypeScript MCP server. Source: Forge/Configs/MCP_Servers/uvilo-trash.py. Provides safe file deletion (trash/untrash) instead of rm.

forge-typesense

Search tool for Uvilo OS documentation and repo content. Provides forge-typesense__search_knowledge tool.

forge-discovery

TypeScript MCP server. Source: Forge/Configs/MCP_Servers/forge-discovery/. Provides forge-discovery__list_departments, forge-discovery__list_dept_projects, forge-discovery__get_project, forge-discovery__list_skills, forge-discovery__get_skill, forge-discovery__find for skill/project discovery. Stdio transport.

vercel

Command: npx vercel-mcp. Env: VERCEL_API_KEY via Railway env var.

github

Command: npx -y @modelcontextprotocol/server-github. Env: GITHUB_TOKEN.

context7

Command: npx -y @upstash/context7-mcp.

railway

Command: npx -y @jasontanswe/railway-mcp. Env: RAILWAY_API_TOKEN.

neon

Command: npx -y @neondatabase/mcp-server-neon start ${NEON_API_KEY}.

notion

Command: npx -y @notionhq/notion-mcp-server. Env: NOTION_TOKEN.

linear

Command: npx -y @touchlab/linear-mcp-integration. Env: LINEAR_ACCESS_TOKEN.

posthog

Command: npx -y mcp-remote@latest https://mcp.posthog.com/sse.

Playwright MCP

Separate Railway service. Image: mcr.microsoft.com/playwright/mcp. Internal URL: http://playwright-mcp.railway.internal:8931/sse. Serverless enabled (5-15s cold start). Start command: npx @playwright/mcp@latest --headless --browser chromium --no-sandbox --port 8931 --host 0.0.0.0 (do NOT use node cli.js — the image’s internal path may change between versions).

Transport: Use SSE endpoint (/sse), NOT streamable-http (/mcp). The streamable-http transport has a 5-second heartbeat timeout that kills sessions when the LLM takes >5s between tool calls (navigate → snapshot), causing the browser to reset to about:blank. The SSE transport has no heartbeat timeout and keeps sessions alive indefinitely. See upstream issues: #1293, #1307, #1140.

Allowed hosts: Env var is PLAYWRIGHT_MCP_ALLOWED_HOSTNAMES (not HOSTS — the README is wrong).

surveymonkey

Command: npx -y mcp-remote@latest ${COMPOSIO_SM_MCP_URL}. Env: COMPOSIO_API_KEY.

suprsend-dev/staging/prod

Command: python3 /workspace/erik@uvilo.com/uvilo-os/Forge/Configs/MCP_Servers/suprsend-mcp-proxy.py <workspace>. Injects Railway env vars at startup. Uses SUPRSEND_SERVICE_TOKEN.

MCP Development Notes

Schema Compatibility

Claude’s Anthropic API requires tool inputSchema to conform to JSON Schema draft-2020-12. Common incompatibilities from third-party MCP binaries:

1. Type arrays ["null","X"] — draft-07, not allowed in 2020-12
2. Bare boolean true/false as schemas (from Go interface{})
3. $schema URIs pointing to draft-07 or earlier
4. Boolean-form exclusiveMinimum/exclusiveMaximum (draft-04)

Fix: write a Python stdio proxy that intercepts tools/list responses and rewrites schemas. See suprsend-mcp-proxy.py for the pattern.

forgentic Runtime Behavior

• LangGraph recursion limit: Each tool call counts toward the limit (default 100). Cap polling to 6 calls max.
• Self-restart drops response: Always use a short timeout (2-3s), treat connection error as success.
• Config changes require redeploy, not just restart. Bot and MCP configuration is parsed during deploy.
• Tool loading: Tools may be eager (always available) or deferred (loaded on demand). If a tool is deferred, use toolSearch to find it and loadTool to activate it. Use unloadTool when finished to reduce context.
• Agent spawning: Use spawnAgent to create sub-agents with their own Bot, model, and tools. Use checkAgentJob to poll status. Use abortAgent to cancel a running sub-agent.

Git Operations

When local and remote branches have diverged: git pull --rebase origin <branch> && git push. Prevention: always git pull origin dev immediately before making commits.

Fixing .git Permissions

Repo cloned as root but MCP servers may run as a different user. Fix: chmod -R 777 /workspace/erik@uvilo.com/uvilo-os/.git (startup may do this, but new git objects may still have root ownership).

Dubious Ownership

git config --global --add safe.directory /workspace/erik@uvilo.com/uvilo-os

NPM Notes

• Pin exact versions in package.json (no ^ or ~). The lockfile is a disposable artifact.
• Use overrides field to pin critical transitive dependencies.
• GitHub username hyphens are dropped in npm scopes (e.g., jason-tan-swe → @jasontanswe).

Suprsend Notes

• Service Tokens are account-wide — not scoped to a workspace.
• CLI binary has no -w/--workspace flag. Workspace selection via --config yaml or per-tool parameters.
• CLI v0.2.19 defines “workspace” field twice with Required(), causing duplicate required array entries. The suprsend-mcp-proxy.py deduplicates these.
• Binary download: use Python’s urllib. Linux x86_64 asset: suprsend_Linux_x86_64.tar.gz. Add User-Agent header to GitHub API requests.

Linear Notes

Use @touchlab/linear-mcp-integration (Touchlab fork). Do NOT use linear-mcp-server (jerhadf) — it has an MCP protocol bug returning JavaScript objects where strings are required. Auth via LINEAR_ACCESS_TOKEN env var.

Markdown Linting

Config: .markdownlint.json. Run from .internal/: npm run lint:md or npm run lint:md:fix.

Persistent Volume Structure

The runtime service mounts a persistent volume at /workspace. This volume survives redeployments but is not backed up automatically.

/workspace/
├── erik@uvilo.com/
│   └── uvilo-os/             ← git working copy (any branch)
└── .trash/                   ← trashed files (uvilo-trash MCP)

Volume mount caveat: Railway’s volume mount at /workspace hides the image’s /workspace contents. Any files placed in the image’s /workspace during build are unreachable at runtime. The runtime must bootstrap any required directory structure on first boot or rely on the git working copy.

Environment Variable Locations

Env vars and secrets are distributed across multiple locations:

Location	Contents	How to update
Railway env vars	~60 variables (API keys, tokens, connection strings)	Dashboard: Service > Variables, or variable_bulk_set API
.internal/.env	Typesense URL/key for Astro dev server	Local file (gitignored); copy from .internal/.env.example and fill in values
Vercel project env vars	PUBLIC_TYPESENSE_URL, PUBLIC_TYPESENSE_SEARCH_KEY	Vercel dashboard or MCP
GitHub repo secrets	TYPESENSE_URL, TYPESENSE_ADMIN_KEY, VERCEL_PROTECTION_BYPASS	GitHub repo Settings > Secrets > Actions

Runtime Service Details

The forgentic runtime is the current deployment.

Property	Value
Config folder	Forge/Configs/
MCP server sources	Forge/Configs/MCP_Servers/
Orchestrator source	orchestrator/ (TypeScript/Hono)
Orchestrator start	node dist/index.js
Orchestrator DB	forge Postgres (FORGE_DB_URL)
AgentJob table	AgentJob in forge Postgres

Container Notes

Available tools: bash, curl, wget, nano, git — pre-installed at build time.

Default shell: /bin/sh → dash, /bin/bash → bash.

Package manager: apt-get (but container filesystem is read-only at runtime — install packages in the Dockerfile instead).

MCP server startup strategy

Tools may be eager (always available) or deferred (loaded on demand). Deferred tools are loaded via toolSearch → loadTool and unloaded after use via unloadTool. This reduces conversation startup time.

Railway env var injection for Python MCP servers

Railway env vars are NOT inherited by child processes (MCP stdio servers). Python MCP servers inject env vars at the top of their module by reading /proc/1/environ directly, BEFORE any os.environ or os.getenv() calls.

TypeScript MCP servers receive env vars through the Bot/MCP configuration environment setup.

Lessons Learned

Lessons that have no natural home elsewhere in this document. If a lesson relates to a specific section, it belongs there instead.

• Never trigger a runtime redeploy from an active agent session. Any action that causes a new deployment — variable_set, variable_delete, variable_bulk_set, deployment_trigger, etc. — kills the running container and drops the agent’s session context mid-response. Instead: prepare the change, ask the operator to type “Continue”, end inference, then apply in the next turn when context is preserved. Alternatively, ask the operator to make the change manually via the Railway dashboard.
• Use Hono over Express for esbuild-bundled ESM services. Express is CJS-heavy, requiring complex esbuild banners/shims to handle require("path") etc. The bundle was 1.3MB and still crashed. Hono is ESM-native, bundles cleanly to 290KB (133KB with --external:pg), no CJS shim issues beyond the standard createRequire banner.
• esbuild ESM + CJS dependencies need a createRequire banner. When bundling with --format=esm and dependencies use CJS require(), esbuild generates a __require polyfill that throws “Dynamic require of X is not supported” at runtime. Add --banner:js with import { createRequire } from 'module'; const require = createRequire(import.meta.url); so require is available before any bundled module code runs. Also mark pg as --external since it has native C++ bindings.
• Stdio transport for repo-reading MCPs — no separate Railway service needed. Use Stdio transport (same as forge-bash, forge-discovery). The runtime launches the MCP as a child process on the same container where the repo is mounted. No separate Railway service, no networking, no auth.
• Inngest Cloud account required for production scheduling. The orchestrator’s Inngest scheduled functions won’t fire automatically without an Inngest Cloud account. INNGEST_DEV=1 mode only works with a local Inngest Dev Server. Create an Inngest Cloud account, obtain signing key + event key, and configure the Railway service.
• import.meta.env.PUBLIC_* does NOT work in astro.config.mjs. Config files are evaluated by Node.js before Vite’s transform pipeline runs, so import.meta.env.PUBLIC_* silently resolves to undefined. Use process.env.PUBLIC_* instead. Never provide fallback values for required env vars — throw an error so the build fails visibly.
• Don’t use @cyanheads/git-mcp-server — unresolvable ENOENT issue with the MCP SDK environment filtering.
• Bun-based Docker images default to localhost binding. When deploying Bun servers on Railway, set HOST=0.0.0.0 as an env var (not in the start command — Railway’s Nixpacks builder interprets KEY=VALUE command as trying to run an executable named KEY=VALUE). Without it, the server only listens on localhost inside the container — the Railway healthcheck passes (runs inside the container) but external traffic gets 502.
• Railway internal networking between containers may be refused. Always test internal connectivity before assuming it works. If internal networking fails, use the public URL as a fallback.
• Railway custom domains vs generated domains. If a custom domain returns 502 but the Railway-generated domain works, the CNAME DNS record likely points to the wrong Railway proxy target. The custom domain’s DNS must point to Railway’s CDN, and the CNAME target must match what Railway expects for that domain.