Forge Skills 2 Requirements
Problem Statement
Forge agents repeatedly waste tokens, time, and money due to gaps in their knowledge, skills, and operating instructions. These failures are not random — they fall into identifiable categories: not knowing what tools exist, not knowing how the environment works, not knowing when to stop and ask for help, and not having efficient patterns for common operations. This project catalogs the known failure modes and produces actionable fixes.
R1: Deferred Tool Awareness
Agents were completely unaware that MCP tools are deferred (not loaded into context)
and that ToolSearch exists to discover and load them on demand. A mention has since
been added to FORGE.md, but the underlying problem persists: agents lack a mental model
of their own tool architecture. When an agent needs a capability (e.g., browser
automation), it reaches for npm install instead of searching for an existing MCP
server — leading directly to R2.
Requirement: Agents must understand the deferred-tool architecture and follow a discovery path (ToolSearch → check available MCP servers → ask user) before attempting to install anything.
R2: Don’t Install What Already Exists
When asked to test os.uvilo.com using Playwright, the agent tried to npm install playwright instead of searching for the existing Playwright MCP server. This failed
because of incompatibilities with the Railway Alpine container. The agent
didn’t know that Playwright MCP already existed as a Railway service, didn’t think to
use ToolSearch, and didn’t know that installing packages in the Alpine container is
constrained.
Requirement: Before installing any package, agents must check whether an MCP server already provides the needed capability. Agents must also be aware of Alpine container constraints that make many installations impossible.
R3: Vercel Deployment Protection Bypass
The agent couldn’t figure out how to bypass Vercel’s deployment protection to access
os.uvilo.com via Playwright. No knowledge file explains that Vercel Standard Protection
requires authentication via a special bypass token. Knowledge/Tools/Vercel.md
only covers API key handling and preview URLs — not access control.
Requirement: Agents must be able to access Vercel-protected deployments when using browser automation.
R4: Secrets and Environment Variable Access
Agents are unaware of what secrets exist or how to properly access environment variables
on Railway. There is no inventory of available secrets (.secrets file, Railway env
vars). The /proc/1/environ pattern for reading Railway env vars is buried in a
Knowledge file but not surfaced when relevant. Agents try os.environ or echo $VAR
which don’t work reliably on Railway.
Requirement: Agents must know what credentials and environment variables are available and how to access them on Railway.
R5: Knowledge File Naming and Organization Audit
AI_Guide.md contains loosely related items (page generation system, content inclusion
rules, writing conventions, key files reference) under a vague name. The name “AI Guide”
tells an agent nothing about what’s inside or when to load it. Other Knowledge files may
have similar naming problems.
Requirement: Every Knowledge file name must clearly indicate what it contains and when an agent should load it — without reading the file first. Files covering multiple unrelated topics must be split.
R6: Ask-for-Help Escalation Rule
An agent trying to install TypeSense thrashed for several hours, costing ~$40 in
tokens. Erik then installed TypeSense in the Railway dashboard using a template in
under 1 minute. The existing lesson in Lessons.md says “ask for help early” but is
too soft — agents interpret “2–3 tries” as permission to keep going.
Requirement: Agents must stop and ask the user to intervene after 2 failed attempts on infrastructure/deployment tasks (installing services, configuring databases, setting up MCP servers, modifying Railway/Vercel settings). This rule must have system-prompt priority.
R7: Shell Permission Friction
When using the uvilo-shell tool (for example when an agent runs rm -f /tmp/uvilo-shell-session.yaml - required by the end-of-session
protocol), it hits the permission check, gets prompted, and then grants “session” —
requiring multiple tool calls. The user is never involved in the decision. This adds no
safety while wasting context and tokens. The same pattern may exist for other commands
that FORGE.md instructs agents to run but that aren’t in the allowed list.
Requirement: Commands that agents are instructed to run by FORGE.md must not require interactive permission approval. The permission system must not create guaranteed friction points for routine operations.
R8: Systematic Inefficiency Detection
There is no mechanism to systematically review past conversations for inefficiencies.
Problems like R1–R7 are discovered manually, often after significant waste. The
Forge_Optimizer project folder exists with a sample transcript in
Assets/2026-04-13_18-20.json.
Requirement: There must be a repeatable procedure (skill) that analyzes conversation transcripts to find token waste, unnecessary tool calls, failed approaches, missing knowledge, and patterns that should be codified as lessons or skills. It must produce a structured report of findings with recommended changes.
R9: Agents Skip Required Steps from Their Own Operating Rules
FORGE.md contains a SIDEBAR RULE: “Every new file MUST be added to the sidebar in
.internal/astro.config.mjs as a { slug: '...' } entry in the appropriate group, in
the same commit as the file itself.” When creating the Forge_Skills_2 project, the
agent created three new markdown files and committed them — but did not add any of them
to the sidebar. The user had to ask “Where is Forge Skills 2 in the menu?” only for the agent to reply “It’s not — I missed the sidebar rule. Let me add it now.”
This is not an edge case — it’s a core operating rule written in the system prompt. The agent read FORGE.md (it’s inlined), saw the rule, and still skipped it. This is the same class of problem as R1 and R2: the agent has the information but doesn’t act on it when it matters. Rules in FORGE.md are treated as background context, not as checklist items. An agent creating files focuses on the file content and the commit, but doesn’t walk back through its operating rules to verify compliance.
Requirement: Agents must verify compliance with operating rules after performing actions those rules govern. Rules that are frequently skipped must be structured so agents cannot overlook them.
Requirement Agent should summarize major steps taken. For example at the end of the output it should say “Committed and pushed to dev.” or “Created file and matching sidebar menu entry.”
R10: Content Inclusion Pattern Complexity
The glob patterns in .internal/src/content.config.ts are unnecessarily complicated — ~30 lines of per-department includes, per-directory excludes, and .generated/ re-inclusions. The root cause is three compounding decisions: (1) SKILL.md files are excluded from the content collection then re-included as .generated/ read-only wrappers, (2) directories like Scripts/ and Schemas/ are excluded to avoid duplicates with file-view wrappers, and (3) Forge is special-cased with individual file patterns instead of a blanket glob.
The actual requirement is simple: all .md files should be included directly. Non-.md files still need .generated/ wrappers. SKILL.md files should be included as regular pages with combined frontmatter (see R11), not as read-only file-view pages.
Requirement: The content inclusion pattern must be simplified to a blanket **/*.md glob (excluding .internal/) with no per-department or per-directory special-casing. The SKILL.md exclusion and .generated/ SKILL.md wrapper machinery must be removed.
R11: Combined Frontmatter for SKILL.md Files
SKILL.md files need two different frontmatter formats: the Agent Skills spec requires name and description, while Astro Starlight requires title and supports custom fields like visibility, status, and owner. Currently this conflict is resolved by excluding SKILL.md from the content collection and generating read-only .generated/ wrapper pages — which is the root cause of the complexity in R10.
Research confirms both systems tolerate extra fields: Astro’s Zod schema strips unknown keys silently (no error), and the Agent Skills spec does not forbid additional frontmatter fields. The metadata field in the Agent Skills spec exists as an explicit escape hatch for arbitrary key-value pairs. Combined frontmatter like name + title + visibility + status + owner works for both systems.
To make name and description accessible in Starlight templates (not just stripped), they must be added to the extend schema in content.config.ts.
Requirement: SKILL.md files must use combined frontmatter satisfying both the Agent Skills spec and Astro Starlight. The name and description fields must be added to the Starlight extend schema so they survive Zod parsing. The .generated/ SKILL.md wrapper generation must be removed.
Non-Goals
- Rewriting the entire FORGE.md structure.
- Adding new MCP servers or infrastructure.
- Changing the shell permission model itself (only fixing its known friction points).