TypeSense Index Vision

---

Problem

Two inefficiencies in the Typesense search indexing pipeline waste compute and cause operational failures:

1. The index-search.yml GitHub Action triggers on pushes to dev that touch **/*.md, .internal/src/**, or .internal/docsearch.config.json. While path filtering exists, the **/*.md glob is overly broad — many .md files (e.g., project State files, WIP files) don’t affect the website or search index. Additionally, there’s no separate GitHub Action for the department reindex (index-department.py); it only runs on the Railway container via the Reindex Typesense skill, where it frequently times out.

2. The index-department.py script (used by the Reindex Typesense skill) consistently times out when reindexing more than a small batch of files. Each file requires two sequential OpenAI API calls (summary + embedding), the script has no concurrency or batching, and it runs on a Railway container with resource constraints. An incremental reindex of 385 files takes 10+ minutes minimum and frequently fails.

---

Requirements

R1: Refine GitHub Action path filtering

The index-search.yml workflow currently triggers on **/*.md which is overly broad. The indexer must process only content-relevant files by using incremental diff-based filtering. The workflow may still trigger on any .md push, but the indexer performs no indexing work when no content-relevant files changed.

R2: Concurrent OpenAI API calls in index-department.py

The index-department.py script must make summary and embedding API calls concurrently rather than sequentially, so that reindexing throughput scales with concurrency instead of being limited to one file at a time.

R3: Full reindex via GitHub Action

The full department reindex (currently run on the Railway container via the Reindex Typesense skill) must run as a GitHub Action instead, eliminating Railway container timeout and resource constraints.

R4: Migrate indexing scripts to TypeScript

The indexing pipeline (index-department.py and related Python scripts) must be rewritten in TypeScript using the Write TypeScript skill, bringing them in line with the project’s standard language.

R5: Migrate Typesense MCP server to TypeScript

The Typesense MCP server (currently a Python FastMCP server) must be rewritten in TypeScript, conforming to the project’s standard language for MCP servers.

R6: Update Typesense MCP skill for current environment

The Typesense MCP skill documentation must be updated to reflect the current runtime environment (debian-slim base image): remove outdated limitations such as “no curl in container”, remove references to Python dependencies, and remove references to env var access patterns that are no longer valid.

R7: Default search_knowledge to published-only results

The search_knowledge tool must default to returning only results with status published. Callers may override this by explicitly specifying other statuses (e.g., archived, draft) or all.

Success Criteria

#	Criterion	Measured by
V1	The indexer processes no files when a push contains only non-content changes	Push a non-content change and verify the indexer performs no indexing work
V2	index-department.py processes files with ≥5 concurrent API calls	Inspect script code for concurrency implementation (asyncio, ThreadPoolExecutor, or equivalent)
V3	Incremental reindex of 385 files completes in under 4 minutes with no timeouts	Run the script and measure wall-clock time
V4	Full department reindex runs successfully as a GitHub Action without container timeout	Trigger the action and verify completion
V5	Indexing scripts are written in TypeScript, following the Write TypeScript skill conventions	Code review of rewritten scripts
V6	Typesense MCP server is written in TypeScript	Code review of rewritten MCP server
V7	Typesense MCP skill documentation contains no references to Python, curl limitations, or obsolete env var patterns	Grep skill docs for outdated terms
V8	search_knowledge returns only published results by default; override works for other statuses	Call search_knowledge with no status filter → only published; call with explicit status → returns matching results

---

Out Of Scope

• Changing the Typesense collection schema or indexing strategy
• Replacing the DocSearch scraper (only coexisting or consolidating with index-department.py)
• Modifying the Reindex Typesense skill’s user-facing procedure
• Switching OpenAI models (gpt-5.4-nano for summaries, text-embedding-3-small for embeddings)
• Changing the default search_knowledge status filter behavior beyond what is specified in R7