TypeSense Index Spec

The Typesense indexing pipeline is rewritten in TypeScript, runs concurrently, executes as a GitHub Action, and defaults to published-only search results. Operational tooling lives under Forge/Typesense/Maintenance/ (not .internal/). Two bundled TypeScript projects replace the Python scripts: index-department (department file indexer) and typesense-mcp (MCP server). The DocSearch scraper workflow remains unchanged in trigger config but benefits from the indexer’s incremental-by-default design.

Canonical plan: TypeSense_Index_Plan_2.md (active). Plan 1 complete; Plan 3 follows Plan 2.

---

1. DocSearch Scraper Workflow

The index-search.yml GitHub Action continues to trigger on pushes to dev that match **/*.md, .internal/docsearch.config.json, .internal/src/**, and the workflow file itself. No path narrowing is applied to the workflow trigger. The DocSearch scraper job (Docker-based) runs unchanged after a 120-second Vercel deployment wait.

This workflow handles only the DocSearch scraper (website crawl). Department file indexing is handled by the separate index-department.yml workflow (§3).

---

2. Typesense Maintenance (Forge/Typesense/Maintenance/)

Typesense operational scripts belong here, not in .internal/ (which is reserved for the Astro site build).

Artifact	Path
Collection setup	Forge/Typesense/Maintenance/create-typesense-collections.ts
Department indexer (bundled TS)	Forge/Typesense/Maintenance/index-department/

2.1 Location and Build

A bundled TypeScript project at Forge/Typesense/Maintenance/index-department/. Built with esbuild into a single JavaScript bundle (dist/index.js). External dependencies (p-limit, openai, typesense) are compiled into the output — no runtime node_modules required. Follows the Write TypeScript skill conventions for bundled projects.

REPO_ROOT resolves five levels up from dist/index.js (dist → index-department → Maintenance → Typesense → Forge → repo root).

2.2 Invocation

Run from repo root via node Forge/Typesense/Maintenance/index-department/dist/index.js with positional and flag arguments:

Argument	Description
department...	One or more department names to index
--subpath=<path>	Index only a subfolder within a department
--root	Index repo root-level files only
--all	Reindex everything + stale cleanup
--incremental	Only index files changed since a ref
--since=<ref>	Git ref for incremental mode (default: HEAD~1)

This preserves the interface of the original index-department.py (see Research_Decisions.md for exact original usage).

2.3 Environment Variables

Variable	Purpose
OPENAI_API_KEY	OpenAI API key for summaries and embeddings
TYPESENSE_URL	Typesense server URL
TYPESENSE_ADMIN_KEY	Typesense admin API key

The indexer reads process.env first. If a variable is not set, it falls back to reading /proc/1/environ for Railway container compatibility.

2.4 File Discovery

The indexer scans the repo root for departments in the set: Forge, Product, Technology, Marketing, Planning, Operations, Finance, Investors, Uvilo. It ignores directories: .git, .internal, .trash, .generated, node_modules, .vscode, .vercel, _temp, .tmp.

For --incremental mode, the indexer diffs changed files since the given ref using git diff --name-only --diff-filter=AM <sinceRef> and filters to .md files (excluding _WIP.md and ignored directories).

2.5 File Processing

For each markdown file:

1. Summary — The file content (truncated to 4000 characters) is sent to OpenAI gpt-5.4-nano with the summary prompt (see Research_Decisions.md for exact prompt text). Returns a ≤100-word summary.

2. Chunking — The file is split into chunks by headings. Each chunk has a word limit of 800 words. Heading-based chunks with fewer than 50 words are merged with their parent section.

3. Embeddings — Each chunk is sent to OpenAI text-embedding-3-small to generate a vector embedding.

4. Document — Each chunk is assembled into a Typesense document with fields: id, path, title, summary, content (chunk text), embedding, department, project, type, source, status, visibility, owner. Schema matches create-typesense-collections.ts (no heading field).

2.6 Concurrency

Summary and embedding calls for a single file run in parallel. Multiple files process in parallel. Concurrency is controlled by p-limit with a default of 5 and a configurable maximum (up to 8). The concurrency limit is set via a --concurrency CLI flag or the CONCURRENCY environment variable.

2.7 Upsert

Documents are batched and upserted to the Typesense uvilo collection in batches of 40. Each upsert is retried up to 3 times with a 2-second delay between retries.

2.8 Stale Cleanup

When --all mode is used, after all files are upserted, the indexer queries the Typesense collection for documents whose path no longer exists in the repo and deletes them.

---

3. Department Indexing Workflow (index-department.yml)

3.1 Trigger

A new GitHub Action at .github/workflows/index-department.yml triggered via workflow_dispatch with the following inputs:

Input	Type	Default	Description
mode	choice: incremental / all / department	incremental	Indexing mode
department	string	(empty)	Department name (required when mode is department)
since_ref	string	HEAD~1	Git ref for incremental mode

3.2 Runner

Runs on ubuntu-latest (2 vCPUs, 7 GB RAM, 6-hour job timeout). No Railway container constraints.

3.3 Steps

1. Checkout the repo (fetch-depth: 0)
2. Set up Node.js
3. cd Forge/Typesense/Maintenance/index-department && npm ci && npm run build
4. Run node Forge/Typesense/Maintenance/index-department/dist/index.js with the specified mode and inputs

3.4 Secrets

Secret	Environment Variable
OPENAI_API_KEY	OPENAI_API_KEY
TYPESENSE_ADMIN_KEY	TYPESENSE_ADMIN_KEY
TYPESENSE_URL	TYPESENSE_URL

---

4. Typesense MCP Server (typesense-mcp)

4.1 Location and Build

A bundled TypeScript project at Forge/Configs/MCP_Servers/typesense-mcp/. Built with esbuild. Follows the Write TypeScript skill conventions for bundled MCP servers. Replaces the current typesense-mcp.py.

4.2 Tools

The MCP server exposes two tools with the same signatures as the current Python server:

search_knowledge — Semantic search across the Typesense uvilo collection.

Parameter	Type	Default	Description
query	string	(required)	Search query
department	string	(none)	Filter by department
project	string	(none)	Filter by project
type	string	(none)	Filter by document type
source	string	(none)	Filter by source
status	string	"published"	Filter by status. "all" omits the filter
visibility	string	(none)	Filter by visibility
owner	string	(none)	Filter by owner
limit	number	5	Max results (1–50)

get_file_summary — Retrieve the AI-generated summary of a file from the Typesense index.

Parameter	Type	Description
path	string	Relative file path (e.g., Forge/Knowledge/Memory_Architecture.md)

4.3 search_knowledge Status Default

The status parameter defaults to "published". When a caller passes no status parameter, only documents with status=published are returned. When a caller passes status="all", the status filter is omitted entirely and all documents are returned regardless of status. Any other explicit status value (e.g., "draft", "archived") filters normally.

4.4 Environment Variables

Variable	Purpose
TYPESENSE_URL	Typesense server URL
TYPESENSE_SEARCH_KEY	Typesense search-only API key (read-only, not admin)

Reads process.env directly. Falls back to /proc/1/environ for Railway compatibility.

---

5. Typesense MCP Skill Documentation

The skill documentation for the Reindex Typesense skill (in Forge/Skills/) is updated to reflect the TypeScript implementation:

• All Python dependency references are removed
• The “no curl in container” limitation is removed (debian-slim includes curl)
• Environment variable access is described as process.env with Railway fallback to /proc/1/environ
• CLI invocation examples use node dist/index.js instead of python3 index-department.py
• The full reindex procedure references the index-department.yml GitHub Action (workflow_dispatch) instead of the Railway container

---

6. Artifacts Replaced

Current Artifact	New Artifact
.internal/index-department.py	Forge/Typesense/Maintenance/index-department/ (bundled TS project)
Forge/Configs/create-typesense-collections.ts	Forge/Typesense/Maintenance/create-typesense-collections.ts
Forge/Configs/MCP_Servers/typesense-mcp.py	Forge/Configs/MCP_Servers/typesense-mcp/ (bundled TS project)
Reindex Typesense skill (Python-based docs)	Updated skill documentation

---

Requirements Traceability

Requirement	Spec Section
R1: Refine GitHub Action path filtering	§1 (no trigger narrowing; incremental logic in indexer §2.4)
R2: Concurrent OpenAI API calls	§2.6
R3: Full reindex via GitHub Action	§3
R4: Migrate indexing scripts to TypeScript	§2
R5: Migrate Typesense MCP server to TypeScript	§4
R6: Update Typesense MCP skill documentation	§5
R7: Default search_knowledge to published-only	§4.3