published Visibility internal Owner erik@uvilo.com Approver _ Created _ Updated _

Forge Setup

Step-by-step environment rebuild instructions and disaster recovery plan for the Forge infrastructure. If followed from scratch on a blank Railway account, this document reproduces the entire Forge environment.

See Forge Infrastructure for the declarative description of what exists and how it’s configured.

Execution Contexts

This document distinguishes two execution contexts:

Context	Who	Where	Tools available
Agent	Forge agent	In the current forgentic / uvilo-mono Railway environment	`forge-bash__run`, attached MCP tools (forge-filesystem, forge-discovery, forge-typesense, etc.), Railway MCP (deferred tool), GitHub MCP, Vercel MCP, etc.
Operator	Human (Erik)	Local machine	Docker CLI, Railway dashboard, domain registrar, Vercel dashboard, RoboForm

Steps are tagged [Agent] or [Operator] to make the boundary explicit. In a disaster recovery scenario, some steps can only be performed by the Operator (e.g., configuring secrets, managing DNS).

Part A: Disaster Recovery — What to Back Up

1. Application Data (uvilo-mono / bot-craft)

Contents: Bot configurations, conversation state, user accounts, and runtime data for the uvilo-mono / bot-craft platform.

Backup method: Railway’s built-in backup feature on any database service backing the uvilo-mono runtime.

Enabling backups (agent): Use the Railway MCP deferred tool to enable backups on the database service. Set schedule to Daily + Weekly + Monthly.

Enabling backups (human): Railway Dashboard → database service → Backups tab → Enable. Set schedule to Daily + Weekly + Monthly.

Restore: Railway Dashboard → database service → Backups tab → select backup → Restore.

Frequency: Daily (automatic via Railway). Before any destructive operation, verify a recent backup exists.

2. Typesense Index

Contents: ~814 documents (503 repo + 311 website pages).

Backup method: Railway’s built-in backup feature (Typesense service with attached volume).

Enabling backups (agent): Use the Railway MCP deferred tool to enable backups on the Typesense service. Set schedule to Daily + Weekly + Monthly.

Enabling backups (human): Railway Dashboard → Typesense service → Backups tab → Enable. Set schedule to Daily + Weekly + Monthly.

Restore: Railway Dashboard → Typesense service → Backups tab → select backup → Restore.

Alternative: The index can be fully rebuilt from the repo and website using the indexing scripts (Forge/Typesense/Maintenance/index-department/, .internal/scrape-site.py) run through the current repo tooling or forge-bash__run. This is the recommended approach when the index is corrupted or needs a fresh rebuild — the canonical source is always the repo and website, not a backup snapshot.

3. Railway Persistent Volume (`/workspace`)

Critical files on the volume:

File	Importance	Can be recreated?
`/workspace/erik@uvilo.com/uvilo-os/`	Medium — git working copy	Can be re-cloned from GitHub

Environment variables and service configuration are stored in Railway environment variables on the forgentic container, not on the volume. They must be backed up separately (see §5 below).

4. uvilo-mono / bot-craft Railway Service Build/Deploy

Service: The forgentic / uvilo-mono Railway service, built from the uvilo-os repository.

Can be rebuilt from the repo source. The service is deployed directly from the GitHub repository via Railway’s GitHub integration or from a container image built from the repo.

Rebuild requires (operator or agent):

Access to the ErikDakoda/uvilo-os GitHub repository
Railway project with the forgentic service configured

5. Railway Environment Variables

Variables across the forgentic and supporting Railway services, covering API keys, tokens, connection strings, and configuration.

Backup method: Export via Railway dashboard:

Forgentic Service → Variables → Raw Editor → copy JSON
Store in RoboForm safe notes

Restore: Re-import via dashboard (paste JSON) or Railway MCP variable_bulk_set.

⚠ When any Environment Variable changes, always prompt the Operator to back up the updated contents to RoboForm.

6. DNS Records

Type	Name	Value
CNAME	`os`	`9a5064d98cd8354e.vercel-dns-016.com.`

Additional DNS entries may exist for current service domains (e.g., Typesense public URL, Playwright MCP URL). DNS is managed at the domain registrar. These are standard records and easy to recreate.

7. Vercel Project Config

Setting	Value
Project name	`uvilo-os`
Team	Uvilo
Framework	Astro
Root directory	`.internal/`
Include files outside root	Enabled
Protection bypass secret	Stored in Railway environment variables

8. Railway Backup Summary

Every Railway service with an attached volume should have Railway backups enabled. The expected configuration:

Service	Volume	Backup schedule
Forgentic (uvilo-mono)	`/workspace` (5GB)	Daily + Weekly + Monthly
Typesense	Data volume	Daily + Weekly + Monthly

Part B: Step-by-Step Environment Rebuild

Prerequisites

Before starting, you need:

GitHub PAT with repo scope (for cloning) — stored in RoboForm
Domain access — ability to edit DNS records for uvilo.ai and uvilo.com — [Operator]
Vercel account access (Uvilo team) — [Operator] or [Agent] via Vercel MCP
RoboForm access — contains env var JSON backup — [Operator]

The Railway MCP is available as a deferred tool in the Forge environment and should be used for all Railway operations instead of the CLI.

Step 1: Railway Project Setup [Agent]

Use the Railway MCP deferred tool for all operations:

Create a new Railway project: chat-uvilo-os
Note the project ID for subsequent steps

Step 2: uvilo-mono / bot-craft Service [Agent]

2a. Create the Railway Service [Agent]

Use Railway MCP:

Create a new service from the GitHub repository ErikDakoda/uvilo-os or from the current container image for the forgentic runtime.
Set the start command appropriate for the uvilo-mono / bot-craft service.
Add a persistent volume: mount at /workspace, 5GB
Set the Railway domain for the service
Enable Railway backups: Use Railway MCP or Dashboard → Backups tab → Enable. Set schedule to Daily + Weekly + Monthly.

2b. TypeScript Build (Automatic) [Agent]

The service automatically runs the TypeScript build after cloning the repo. This discovers all package.json files with a build script (MCP servers, etc.) and runs npm install && npm run build in each.

Why: dist/ is gitignored — built output is never committed. It must be rebuilt on every fresh checkout. The auto-build ensures MCP servers and other bundled TypeScript projects are ready before the service starts.

If the auto-build fails, startup continues with a warning. To manually rebuild via forge-bash__run:

npx tsx Forge/Skills/Build_Typescript/rebuild-typescript.ts

2c. Set Environment Variables [Agent]

Critical: Use variable_bulk_set ONCE with all variables. Each individual variable_set call triggers a deployment.

Restore from the RoboForm backup JSON:

Agent: Use Railway MCP variable_bulk_set with the full JSON
Human: Railway Dashboard → Forgentic Service → Variables → Raw Editor → paste JSON → Save

If no backup is available, set these essential variables manually (see Forge Infrastructure for the full list):

Variable	Purpose
`GITHUB_TOKEN`	GitHub PAT (repo scope; used for MCP servers and git operations)
`OPENAI_API_KEY`	OpenAI API key
`ANTHROPIC_API_KEY`	Anthropic API key
`TYPESENSE_URL`	Typesense host URL
`TYPESENSE_ADMIN_KEY`	Typesense admin API key
`TYPESENSE_SEARCH_KEY`	Typesense search-only key

2d. Credentials via Railway Environment Variables [Agent]

Credentials are supplied through Railway environment variables on the forgentic container. There is no separate secrets file to seed — all API keys, tokens, and connection strings are set as Railway env vars in Step 2c.

2e. Deploy Bot/MCP Configuration [Agent]

Bot and MCP configurations are deployed from the repository. After the service is running and the repo is cloned, verify that Bot and MCP tool configurations are loaded correctly by checking the agent’s available tools and Bot definitions.

Step 2f: Set GitHub Repository Secrets [Operator]

The index-search GitHub Actions workflow requires secrets. Set them in GitHub → ErikDakoda/uvilo-os → Settings → Secrets and variables → Actions:

Secret	Source
`TYPESENSE_URL`	Same value as Railway env var `TYPESENSE_URL`
`TYPESENSE_ADMIN_KEY`	Same value as Railway env var `TYPESENSE_ADMIN_KEY`
`VERCEL_PROTECTION_BYPASS`	Same value as the Vercel protection bypass secret in Railway env vars

Rotation rule: If you rotate the Vercel bypass secret, update all three locations: Railway env var, GitHub repo secret VERCEL_PROTECTION_BYPASS, and Vercel project protection settings.

Step 3: Typesense Service [Agent]

Typesense is not available via Railway’s “Add Database” menu. It must be deployed as a Docker image service with a persistent volume.

3a. Create the Service

Use Railway MCP service_create_from_image with image typesense/typesense:latest
Set the start command:
```
sh -c 'mkdir -p /data && /opt/typesense-server --data-dir /data --api-port 8108 --api-key $TYPESENSE_API_KEY --enable-cors'
```
Why sh -c and mkdir -p: The volume mount at /data is empty on first boot. Typesense won’t start without the data directory. The sh -c wrapper creates it before launching the server.
Create a persistent volume: mount at /data (5GB)
Set environment variable TYPESENSE_API_KEY to a randomly generated admin key (openssl rand -hex 24). This is the TYPESENSE_ADMIN_KEY used by all indexing scripts and the MCP server.

3b. Configure Public Networking

Public networking is required. Typesense is accessed by:

Vercel builds (website search queries from browsers)
GitHub Actions CI (DocSearch scraper writes to it)
Indexing scripts run through the current repo tooling or forge-bash__run (repo/website indexing)

Use Railway MCP domain_create to generate a public URL. Railway auto-assigns a domain (e.g., typesense-production-33cf.up.railway.app).
This URL becomes the TYPESENSE_URL env var used everywhere.

3c. Set Railway Environment Variables

On the forgentic service, set these (use variable_bulk_set to avoid multiple deployments):

Variable	Value
`TYPESENSE_URL`	Public URL from Step 3b (e.g., `https://typesense-production-33cf.up.railway.app`)
`TYPESENSE_ADMIN_KEY`	Same value as `TYPESENSE_API_KEY` set on the Typesense service
`TYPESENSE_SEARCH_KEY`	Derived from the admin key (see below)

Deriving the search-only key: The TYPESENSE_SEARCH_KEY is a scoped-down key that allows only search operations (not admin). Generate it using the Typesense API:

curl -H "X-TYPESENSE-API-KEY: $TYPESENSE_ADMIN_KEY" \
  "${TYPESENSE_URL}/keys" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"description": "Search-only key for uvilo and uvilo_docs", "actions": ["documents:search"], "collections": ["uvilo", "uvilo_docs"]}'

The response contains a value field — that’s your TYPESENSE_SEARCH_KEY.

3d. Enable Backups

Enable Railway backups: Use Railway MCP or Dashboard → Typesense service → Backups tab → Enable. Set schedule to Daily + Weekly + Monthly.

3e. Create Collections

Two collections are required before indexing can run:

Collection	Purpose	Populated by
`uvilo`	Repo documents + website pages	`index-department` CLI (repo), `scrape-site.py` (website)
`uvilo_docs`	Website search (DocSearch format)	GitHub Actions `index-search` workflow

Run the collection creation script via forge-bash__run:

cd /workspace/erik@uvilo.com/uvilo-os
npx tsx Forge/Typesense/Maintenance/create-typesense-collections.ts

This creates both collections. The uvilo schema includes fields for frontmatter metadata and embeddings. The uvilo_docs schema is minimal — the DocSearch scraper will overwrite it with its full schema on first run, but the collection must exist beforehand.

3f. Index Data

After the service is running and the collections are created, index the data using forge-bash__run:

cd /workspace/erik@uvilo.com/uvilo-os/.internal
cd Forge/Typesense/Maintenance/index-department && npm run build
node Forge/Typesense/Maintenance/index-department/dist/index.js --all  # repo docs into 'uvilo' collection
python3 scrape-site.py       # website pages into 'uvilo' collection

3g. Verify CI Indexing Workflow

Push a markdown change to dev and confirm the “Index Search” workflow succeeds in the Actions tab. This workflow scrapes the deployed website into the uvilo_docs collection — it runs automatically on every push to dev that touches *.md files. It requires GitHub repo secrets (see Step 2f).

3h. Local Dev Environment

Create .internal/.env for local Astro dev server (needed if running npm run dev locally):

cd .internal
cp .env.example .env
# Edit .env and fill in the Typesense URL and search-only key

The values come from Railway env vars (TYPESENSE_URL and TYPESENSE_SEARCH_KEY). This file is gitignored — it is not committed to source control.

Step 4: Playwright MCP Service [Agent]

Use Railway MCP to create a new service from image: mcr.microsoft.com/playwright/mcp
Set start command: npx @playwright/mcp@latest --headless --browser chromium --no-sandbox --port 8931 --host 0.0.0.0 (do NOT use node cli.js — the image’s internal path may change between versions)
Enable serverless mode (sleeps after 10 min, wakes on traffic, ~5-15s cold start)
Set env var PLAYWRIGHT_MCP_ALLOWED_HOSTNAMES (not HOSTS)
Note internal URL: http://playwright-mcp.railway.internal:8931/sse
Use SSE transport endpoint (/sse), NOT streamable-http (/mcp) — the streamable-http transport has a 5-second heartbeat timeout that kills sessions when the LLM takes >5s between tool calls, causing the browser to reset to about:blank. SSE has no heartbeat timeout.

Step 5: DNS Configuration [Operator]

os.uvilo.com → CNAME to 9a5064d98cd8354e.vercel-dns-016.com.
Add any additional CNAME records for current service domains as needed.

Step 6: Vercel Project (Documentation Site) [Agent] or [Operator]

Connect the ErikDakoda/uvilo-os repo
Framework preset: Astro
Root directory: .internal/
Enable “Include files outside root directory”
Set env vars (scope both to Preview, since the site serves the dev branch preview build):
- PUBLIC_TYPESENSE_URL
- PUBLIC_TYPESENSE_SEARCH_KEY
Deploy

Part C: Operational Gotchas

Hard-won lessons that don’t fit naturally into the declarative infrastructure doc. These are the non-obvious things that waste hours if you don’t know them.

Railway variable_bulk_set Always Bulk

Every variable_set or variable_bulk_set call triggers a new deployment. Setting 20 vars individually = 20 deployments, hitting rate limits fast.

Rule: Always collect ALL variables first, then use variable_bulk_set ONCE. If rate-limited, provide the full JSON to the user to paste in the Railway dashboard instead.

During initial setup: Use the env var backup from RoboForm to restore all vars at once via the dashboard Raw Editor.

Railway Config Changes Require a New Deployment

service_update (start command, region, replicas, etc.) changes the service configuration but does not create a new deployment — the change sits pending. A dashboard “Redeploy” won’t pick it up either: it re-deploys the last successful deployment, which was built under the old config.

To activate a config change you must trigger a new deployment:

Agent: For image-based services (no GitHub repo), deployment_trigger requires a commitSha that doesn’t exist. Use variable_bulk_set (or variable_set) to set/change any env var — this forces a new deployment that picks up the pending config.
Operator: Click “Deploy” (not “Redeploy”) in the Railway dashboard.

Astro Config Files Can’t Use `import.meta.env`

astro.config.mjs is evaluated by Node.js before Vite starts. import.meta.env.PUBLIC_* silently resolves to undefined, causing config values to fall back to defaults without any build error.

Fix: Use process.env.PUBLIC_* in astro.config.mjs. Never provide fallback values for required env vars — throw an error so the build fails visibly:

const typesenseUrl = new URL(process.env.PUBLIC_TYPESENSE_URL ?? (() => { throw new Error('PUBLIC_TYPESENSE_URL is not set'); })());

Railway Domain targetPort — Use “default”

When creating a Railway domain, setting targetPort=3080 causes 502 errors. Railway’s internal routing expects the port to be auto-detected from the container’s EXPOSE or listening port.

Fix: Omit targetPort or use "default". Only set an explicit port if the service listens on a non-standard port and Railway can’t detect it.

Failed Railway Services — Delete and Recreate

When a Railway service fails with “unable to connect to registry” or “Failed to create deployment”, redeploying or restarting doesn’t help. The deployment record is stuck in a bad state.

Fix: Delete the service entirely and recreate it with service_create_from_image. Fresh services deploy cleanly.

Bun Servers Default to localhost Binding

Bun’s Bun.serve() binds to localhost by default (unlike Node.js which often binds to 0.0.0.0). On Railway, the healthcheck passes (it runs inside the container where localhost works) but external traffic from the Railway proxy gets 502.

Fix: Set HOST=0.0.0.0 as an env var (not in the start command). Do NOT put HOST=0.0.0.0 in the start command — Railway’s Nixpacks builder interprets KEY=VALUE command as trying to run an executable named KEY=VALUE.

TCP Proxies Required for External Database Access

Railway services are private by default. HTTP domains only route HTTP traffic — raw TCP ports (like PostgreSQL’s 5432) are NOT exposed externally without a TCP proxy.

Symptom: Connection times out when connecting from an external client (e.g., SQLPro, pgAdmin, DBeaver) to yourdb.up.railway.app:5432.

Fix: Create a TCP proxy for the service:

Use Railway MCP tcp_proxy_list to check if a proxy exists
Use Railway MCP tcp_proxy_create with:
- environmentId: the environment ID (usually production)
- serviceId: the database service ID
- applicationPort: the database port (e.g., 5432 for PostgreSQL)
Railway returns a public domain and proxy port (e.g., switchyard.proxy.rlwy.net:50665)
Use the returned host and port in your external client

Key insight: Each TCP proxy maps a unique public port to an internal service port. The proxy port is auto-assigned and not the same as the application port.

Python 3.12+ vs 3.11 — TarFile.extractall(filter=)

The filter="data" keyword argument to TarFile.extractall() was added in Python 3.12. Some environments ship Python 3.11 (bookworm). Code that uses filter="data" will crash with TypeError.

Fix: Add a version check:

if sys.version_info >= (3, 12):
    tf.extractall(tmpdir, filter="data")
else:
    tf.extractall(tmpdir)

Reading Railway Environment Variables in the Forgentic Container

When scripts running in the forgentic Railway container need to read Railway environment variables, they may not be available via standard os.environ or echo $VAR — they are only in PID 1’s environment.

Shell: cat /proc/1/environ | tr '\0' '\n' | grep VAR_NAME Python: open('/proc/1/environ').read().split('\0')

When using forge-bash__run, environment variables from the Railway container are typically available in the shell session. If a variable appears missing, check PID 1’s environment as shown above.

Use Railway MCP, Not Railway CLI

The Railway CLI (railway command) is not available in the Forge environment. All Railway operations should be performed using the Railway MCP (available as a deferred tool in the Forge environment) or the Railway dashboard (for the operator).

Common Railway MCP operations:

service_create_from_image — create services
variable_bulk_set — set environment variables
variable_set — set a single variable (avoid — triggers redeploy)
service_update — update service configuration

If the Railway MCP is unavailable, direct the operator to use the Railway dashboard.

Part D: Post-Setup Verification Checklist

After setting up or rebuilding the environment run the Verify Infrastructure skill to confirm everything is fully operational.

Railway Services

Forgentic (uvilo-mono) service is running and healthy (check Railway dashboard or Railway MCP)
Typesense service is running (check Railway dashboard or Railway MCP)
Playwright MCP service is running (check Railway dashboard or Railway MCP)
All services show green status in Railway dashboard
Forgentic service Start command is correct

uvilo-mono / bot-craft Runtime Health

The forgentic service responds to health checks
Bot configurations load correctly
Agent conversations work with at least one model
File operations work through forge-filesystem tools

MCP Servers

Git Operations

git pull origin dev works from /workspace/erik@uvilo.com/uvilo-os/
git push works (requires GITHUB_TOKEN env var)
Git identity is set correctly (git config user.name, git config user.email)
No “dubious ownership” errors (safe.directory is configured)

Typesense Search

Typesense index has documents (check collection size)
Typesense indexes correct source
Repo document search works (via forge-typesense MCP)
Website search works (os.uvilo.com search bar)

Documentation Site

https://os.uvilo.com loads (use Playwright if behind Vercel protection bypass)
Protection bypass sets cookie (verify via Playwright)
Sidebar navigation works
Search (Typesense DocSearch) returns results
Search JS bundle contains correct Typesense config (fetch the Search.astro_*.js bundle from the preview deployment and confirm it contains the railway.app host and a non-empty API key)
Latest preview deployment matches dev branch

Environment Variables

All forgentic Railway env vars are set (check via Railway dashboard or cat /proc/1/environ | tr '\0' '\n' | wc -l)
.internal/.env exists with Typesense vars for local dev (copy from .internal/.env.example and fill in values)
Vercel project env vars are set
TYPESENSE_URL secret exists in GitHub repo (Settings → Secrets → Actions)
TYPESENSE_ADMIN_KEY secret exists in GitHub repo
VERCEL_PROTECTION_BYPASS secret exists in GitHub repo
index-search workflow has run successfully at least once (Actions tab → Index Search)

Railway Backups

Typesense — Railway backups enabled with Daily + Weekly + Monthly schedule
Forgentic (uvilo-mono) — Railway backups enabled with Daily + Weekly + Monthly schedule
At least one backup exists for Typesense (verify recent backup timestamp in Dashboard)

Railway Backup Restore Verification

Typesense restore from backup — verified by restoring the latest backup (or confirming the restore flow works, or confirming reindex from scripts as alternative)

RoboForm Backups (with Operator’s help)

RoboForm safe note has current forgentic Railway env var JSON
RoboForm safe note has current GHCR PAT (if applicable)

Railway MCP Functional

Railway MCP is available as a deferred tool and responds to requests (verified by the railway MCP server check above)