Reindex Typesense

Rebuild or update the Typesense search index for the Uvilo OS repo.

When to Use

• Index is stale (new files missing, deleted files still appearing)
• After bulk content changes across multiple departments
• As a quick refresh after a few commits

State Tracking

state.json in this skill folder tracks the last indexed commit. The default incremental reindex uses this commit as its --since ref, so it indexes everything changed since the last reindex — not just HEAD~1.

Procedure

1. Determine mode

User says	Mode	How to run
”Reindex Typesense”	Incremental (since last indexed commit)	--incremental --since=<state.json commit>
”Reindex Typesense all departments”	Full reindex	One department at a time (see step 2)
“Reindex Typesense “	Single department	<Department>

2. Run the indexer

Script location: Forge/Typesense/Maintenance/index-department/

Build first: cd Forge/Typesense/Maintenance/index-department && npm run build

Incremental or single-department — single invocation:

cd [[orgRepoRoot]] && node Forge/Typesense/Maintenance/index-department/dist/index.js <mode-args>

Full reindex — index one department at a time so a failure mid-run doesn’t require restarting from scratch. Large departments (Forge) must be indexed one subfolder at a time to avoid timeouts; smaller departments can be indexed in a single pass. Discover departments from the filesystem, then invoke the script per department (or subfolder) plus root:

# 1. Discover current department directories (any root-level dir that isn't in the ignored set)
cd [[orgRepoRoot]] && ls -d */ | sed 's|/||'

# 2. For SMALL departments, run the indexer in a single pass
node Forge/Typesense/Maintenance/index-department/dist/index.js Product
node Forge/Typesense/Maintenance/index-department/dist/index.js Technology
# ... repeat for every small department

# 3. For LARGE departments (Forge), index one subfolder at a time
#    List subfolders:
cd [[orgRepoRoot]]/Forge && ls -d */
#    Then index each subfolder separately:
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Agentic_Execution
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Bash_Refactor
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Container_Base_Image
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Forge_Optimizer
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Forge_Skill_Cleanup
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Forge_Skills_2
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Git_Dashboard
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Libre_Agents
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/OpenRouter
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/Sidebar_Reorg
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/TinaCMS
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Projects/TypeSense
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Skills
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Assets
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Configs
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Output
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=Archived
#    Then index the department's own root-level files:
node Forge/Typesense/Maintenance/index-department/dist/index.js Forge --subpath=root

# 4. Index repo root-level files last
node Forge/Typesense/Maintenance/index-department/dist/index.js --root

If a subfolder or department fails, re-run just that one — the others are already indexed.

3. Commit any repo changes

If the reindex produced file changes, commit and push to dev.

4. Update state.json

After indexing is done and committed, update Forge/Skills/Reindex_Typesense/state.json:

{
  "last_indexed_commit": "<current HEAD SHA>",
  "last_indexed_mode": "<incremental|all|department>",
  "last_indexed_at": "<ISO 8601 timestamp>"
}

Get the current commit: git rev-parse HEAD

5. Verify

1. search_knowledge — test a query that should find recently added files
2. get_file_summary — check a specific file path returns its summary

Indexer Modes (reference)

Mode	Command	When to use
Full reindex	One department/subfolder at a time + --root	Index is significantly stale; failure-safe
Incremental	--incremental --since=REF	Quick refresh; no stale cleanup
Single department	Forge	Only one (small) department changed
Single subfolder	Forge --subpath=Projects/TypeSense	One subfolder of a large department changed
Dept root files	Forge --subpath=root	Only the department’s own root .md files changed
Root files	--root	Only repo root-level files changed

Do not use --all. It indexes everything in one shot, so a failure partway through requires a complete restart. Indexing one department/subfolder at a time means only the failed unit needs to be re-run.

Large departments (Forge) must use --subpath to avoid timeouts. Run each subfolder separately, then --subpath=root for the department’s own root-level .md files.

Website Reindex

If website pages are also stale, rebuild and re-scrape:

cd [[orgRepoRoot]]/.internal && npm run build && python3 scrape-site.py

Excluded Directories

The indexer skips: .git, .internal, .trash, .generated, node_modules, .vscode, .vercel, _temp, .tmp. Also skips *_WIP.md files.

Gotchas

• OPENAI_API_KEY, TYPESENSE_URL, and TYPESENSE_ADMIN_KEY are read from /proc/1/environ, NOT os.environ — must be set in Railway environment
• Consult Forge/Skills/Choose_AI_Model/Models/ for the indexing use case’s model-specific quirks (required parameters, rejected parameters)
• Full reindex of 300+ files takes 5-10 minutes due to per-file OpenAI API calls; indexing one department at a time means a failure only requires re-running that department
• Rate limiting: script pauses 0.5s every 3 files to avoid OpenAI throttling
• Chunking is heading-based (##/### splits), not fixed word count
• Never hardcode Typesense hosts or API keys. All indexing scripts read from env vars via get_env(). If TYPESENSE_URL or TYPESENSE_ADMIN_KEY is missing, the script exits with an error.