⚠ This spec references a decommissioned Typesense host (typesense-updated-production-3c59.up.railway.app). The current host and keys are managed via environment variables. See Forge/Forge_Infrastructure.md for current configuration.

Typesense Spec

1. Purpose

Typesense-powered semantic search across two surfaces: the Uvilo OS website and the full repo. Humans get a better search experience on the site; Forge agents get a discoverability layer — the ability to find knowledge files by meaning rather than by explicit cross-references.

2. Architecture Overview

Typesense (Railway service)
├── Collection: uvilo (repo + website content)
├── Collection: uvilo_docs (DocSearch schema for website)
│
├── Consumers
│   ├── Humans (website) → starlight-docsearch-typesense plugin (uvilo_docs)
│   └── Agents (LLM) → typesense-mcp (stdio MCP server in LibreChat)
│
├── Embeddings
│   └── OpenAI text-embedding-3-small (1536 dimensions, client-side)
│
└── Pagefind removed

3. Typesense Instance

Hosting

Self-hosted on Railway using the official Typesense Docker image. A persistent volume stores the data directory.

Setting	Value
Service	typesense-updated-production-3c59.up.railway.app
Image	typesense/typesense:28.0
Data dir	/data (mounted persistent volume)
API port	8108
Public URL	https://typesense-updated-production-3c59.up.railway.app (CORS: *)
Serverless	Disabled (always-on)

API Keys

Key	Purpose	Scope
Admin key	Indexing, schema management, collection admin	Full access
Search-only key (nPWz...)	MCP queries	Search on uvilo collection
Search-only key (ImWe...)	Website plugin + MCP	Search on uvilo and uvilo_docs collections

4. Collections

uvilo — Repo + Website Content

Holds repo documents (indexed by the Python indexer) and website pages (indexed by the Python scraper). Supports hybrid search (keyword + semantic) with faceted filtering.

{
  "name": "uvilo",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "title", "type": "string" },
    { "name": "summary", "type": "string" },
    { "name": "content", "type": "string" },
    { "name": "path", "type": "string" },
    { "name": "department", "type": "string", "facet": true },
    { "name": "project", "type": "string", "facet": true },
    { "name": "type", "type": "string", "facet": true },
    { "name": "source", "type": "string", "facet": true },
    { "name": "status", "type": "string", "facet": true },
    { "name": "visibility", "type": "string", "facet": true },
    { "name": "owner", "type": "string", "facet": true },
    { "name": "embedding", "type": "float[]", "optional": true, "num_dim": 1536 }
  ],
  "default_sorting_field": ""
}

Field	Description
id	Unique document ID — path for single-chunk files, path#heading-slug for chunked files
title	From frontmatter title: or first # heading
summary	AI-generated summary (≤100 words) — produced at index time by gpt-5.4-nano
content	Body text (chunked for long files)
path	Relative file path (repo) or URL path (website)
department	Root-level folder: Forge, Product, Technology, etc.
project	Project name derived from path (e.g., TypeSense, Taxonomy) — empty for non-project files
type	knowledge, skill, spec, plan, state, project, readme, agents, index, other, page
source	repo or website
status	From frontmatter status:
visibility	From frontmatter visibility:
owner	From frontmatter owner:
embedding	Client-side generated via OpenAI text-embedding-3-small (not server-side embed field)

uvilo_docs — DocSearch Schema for Website

Populated by the typesense/docsearch-scraper Docker image. Uses the Algolia DocSearch record format (hierarchy, url, anchor, item_priority fields) required by the starlight-docsearch-typesense plugin. ~11,100 documents.

Chunking

Files longer than 800 words are split by markdown headings (##, ###). Each heading-defined section becomes its own chunk with a heading-slug ID (e.g., Forge/Forge_Infrastructure.md#alpine-container-constraints). Small sections (<50 words) are merged with their parent. Sections exceeding 800 words fall back to word-count splitting with 50-word overlap.

Deduplication

When the same content exists in both the repo and the built website, both documents are indexed in the uvilo collection. The source facet allows consumers to filter: agents search source=repo, the website plugin searches source=website.

5. Repo Indexer (.internal/index-department.py)

A Python script that walks the repo, extracts markdown content, generates AI summaries and embeddings, and upserts documents into Typesense.

Behavior

1. Walk the repo tree, find all .md files (excluding ignored directories)
2. Parse YAML frontmatter (title, status, owner, visibility)
3. Extract body text (stripped of frontmatter)
4. Derive department from root-level folder name
5. Derive type from path heuristics:
   • **/Knowledge/** → knowledge
   • **/Skills/**/SKILL.md → skill
   • **/*_Spec.md → spec
   • **/*_Plan*.md → plan
   • **/*_State.md → state
   • **/Projects/** → project
   • **/README.md → readme
   • **/AGENTS.md → agents
   • Everything else → other
6. Derive project from path — the folder name inside Projects/
7. Generate AI summary via gpt-5.4-nano (≤100 words, max_completion_tokens=300)
8. Generate embedding via OpenAI text-embedding-3-small (client-side)
9. Chunk files >800 words by headings
10. Upsert into Typesense collection uvilo
11. Support --incremental flag: only re-index files changed since a git ref
12. Support --all flag: reindex everything + stale document cleanup

Excluded Directories

.git, .internal, .trash, .generated, node_modules, .vscode, .vercel, _temp, .tmp

Environment

• OPENAI_API_KEY — read from /proc/1/environ (NOT os.environ)
• Typesense host and admin key are hardcoded in the script

Execution Modes

cd /workspace/erik/uvilo-os/.internal

python3 index-department.py --all                    # Full reindex + stale cleanup
python3 index-department.py --incremental            # Only changed since HEAD~1
python3 index-department.py --incremental --since=HEAD~5
python3 index-department.py Forge                    # Single department
python3 index-department.py --root                   # Root-level files only

6. Website Scraper (.internal/scrape-site.py)

Scrapes a local Astro build directory to index published website pages into the uvilo collection.

Behavior

1. Walk .internal/dist/ for HTML files
2. Extract content from <main data-pagefind-body> tags
3. Derive title, path, department, project from URL and content
4. Set source=website, type=page, visibility=public
5. Upsert into uvilo collection with website: ID prefix

Execution

cd /workspace/erik/uvilo-os/.internal && npm run build && python3 scrape-site.py

7. MCP Server (typesense-mcp)

A Python stdio MCP server using FastMCP, configured in librechat.yaml. Accessible to LibreChat as a local MCP server.

Tools

Tool	Parameters	Returns
search_knowledge	query, department, project, type, source, status, visibility, owner, limit (default 5)	Array of {path, title, summary, snippet, score}
get_file_summary	path	AI-generated summary field (≤100 words)

Deployment

Setting	Value
Transport	stdio
Framework	Python FastMCP
Config	librechat.yaml → mcpServers.uvilo-typesense
Runtime	Runs as a subprocess of LibreChat

LibreChat Integration

uvilo-typesense:
  title: "Typesense Search"
  description: "Semantic search over Uvilo OS documentation and repo content"
  type: stdio
  command: python3
  args: ["/workspace/typesense-mcp/typesense-mcp.py"]
  timeout: 30000

8. Starlight Integration

Plugin

starlight-docsearch-typesense provides a DocSearch-style search UI powered by Typesense. Queries the uvilo_docs collection (DocSearch schema populated by the typesense/docsearch-scraper).

Configuration

Configured inline in .internal/astro.config.mjs with Typesense host, search-only key, and uvilo_docs collection.

Pagefind

Removed. pagefind: false is set in the Starlight config.

9. Embedding Strategy

OpenAI text-embedding-3-small (1536 dimensions), generated client-side by the indexer script. The collection schema does NOT use the server-side embed field — embeddings are computed in Python and included in the upsert payload.

Why client-side: The server-side embed field was found to be unreliable during implementation. Client-side generation gives full control over the embedding input (title + summary + chunk content) and avoids Typesense server-side OpenAI API dependency.

10. Railway Services Summary

Service	Image/Source	Purpose
typesense	typesense/typesense:28.0	Search engine
typesense-mcp	Python FastMCP (local)	MCP server for agents

The repo indexer and website scraper are scripts, not services — they run on demand.

11. Security

• Admin API key is never exposed to the frontend or MCP clients
• Search-only key (ImWe...) is used in the Starlight plugin (visible in client-side JS)
• Typesense is publicly accessible with CORS access-control-allow-origin: * — the search-only key is safe to expose
• The MCP server runs as a local stdio process, not a network service