claude-os/templates/skills/initialize-project/SMART_INDEXING_GUIDE.md

Claude OS Smart Incremental Indexing System

Overview

The analyze-project skill now includes an intelligent, multi-stage indexing strategy that gradually builds a complete project index without expensive full-reindex operations.

How It Works

Stage 1: Initial Analysis (Fast Setup)

When you run analyze-project: <project_id>:

1. Quick setup (~30-60 seconds for large projects)

   • Analyzes project structure and creates documentation
   • Registers 4 MCPs for on-demand loading
   • Indexes top 25 most important code files (source files prioritized)
   • Creates index state file (.claude-os/.index_state)
   • Installs smart git post-commit hook

2. Result

   • 400-500 code chunks indexed from priority files
   • Complete documentation for coding standards, architecture, practices
   • Foundation ready for AI assistance immediately

Stage 2: Smart Incremental Expansion (Automatic)

Every time you commit:

1. On EVERY commit

   • Git hook tracks commit count
   • Changed files are marked for incremental indexing

2. Every 10 commits

   • Git hook triggers incremental indexer
   • ~30 previously unindexed files are indexed
   • Index state is updated in .claude-os/.index_state
   • No disruption to your workflow

Timeline Example (PISTN: 3,326 files)

Day 1:     analyze-project → 25 files indexed (0.75%)
Days 2-3:  10 commits → 25 + 30 = 55 files (1.65%)
Days 4-5:  20 commits → 55 + 60 = 115 files (3.46%)
Weeks 2-3: 100 commits → gradually reaches 400+ files (12%)
Months:    Index grows naturally to 500+ files (15%+)

File Structure

Key Files Created

.claude-os/
├── .index_state          # Tracks which files have been indexed
├── .commit_count         # Tracks commits since last expansion
├── project-profile/
│   ├── CODING_STANDARDS.md
│   ├── ARCHITECTURE.md
│   └── DEVELOPMENT_PRACTICES.md

Git Hook

.git/hooks/post-commit    # Smart indexing trigger (executable)

Configuration

Expansion Frequency

Current default: Every 10 commits, index ~30 files

To customize, edit the git hook:

if [ $((COMMIT_COUNT % 10)) -eq 0 ]; then  # Change 10 to 5/15/20 as desired
    python3 "$INDEXER_PATH" "$PROJECT_ID" "$PROJECT_PATH" "$API_URL" 30  # Change 30 for batch size

Batch Size

Current default: 30 files per expansion

Recommendations:

• Large projects (3000+ files): 30-50 files per pass
• Medium projects (500-3000 files): 50-100 files per pass
• Small projects (<500 files): 100+ files per pass

Key Design Decisions

Why Not Index Everything Immediately?

1. Performance: Large projects (3000+ files) would take 5-10 minutes upfront
2. UX: Users want immediate feedback, not a blocking operation
3. Incremental growth: Mirrors natural development—as code changes, context grows

Why Every 10 Commits?

1. Timing: 10 commits ≈ 2-3 hours of typical development
2. Balance: Frequent enough to build index quickly, rare enough to minimize overhead
3. Deterministic: Easy to predict and adjust

Why Keep Unindexed Files?

1. Memory: Only recently used files need full context
2. Speed: Incremental scanning is much faster than full analysis
3. Coverage: Eventually, all important files get indexed as they're modified

CLI Scripts

incremental_indexer.py

Expands the project index with previously unindexed files.

python3 incremental_indexer.py <project_id> <project_path> [api_url] [batch_size]

# Example: Index next 50 files in your project
python3 incremental_indexer.py <project_id> ~/Projects/my-project http://localhost:8051 50

Returns JSON with progress:

{
  "indexed": 10,
  "chunks": 245,
  "progress": "35/3137",
  "percentage": 1,
  "status": "in_progress"
}

Triggering Expansion Early

If you want to expand the index before the scheduled 10-commit mark:

# Manually trigger expansion
python3 ~/.claude/skills/analyze-project/incremental_indexer.py <project_id> ~/Projects/my-project

Monitoring Progress

Check indexing progress:

# See current indexed files
cat .claude-os/.index_state

# See commit count (how many until next expansion)
cat .claude-os/.commit_count

MCPs and Memory

Native Memory

Save this to Claude's native memory for each project:

PROJECT: [Name] (ID: [ID])
TYPE: [Rails/Node/etc]
PATH: [path]

INDEXING STRATEGY:
- Initial: 25 key files indexed immediately
- Auto-expand: Every 10 commits, ~30 more files
- Incremental: Changed files indexed on each commit
- Timeline: Full index in ~6-10 weeks (typical dev pace)

Loading MCPs On-Demand

When working on a project, you can load MCPs explicitly:

load pistn-project-profile     # Get coding standards & architecture
load pistn-project-index       # Search indexed code
load pistn-knowledge-docs      # Access project documentation

Troubleshooting

Hook Not Firing

1. Check hook is executable: ls -la .git/hooks/post-commit
2. Make hook executable: chmod +x .git/hooks/post-commit
3. Check incremental_indexer.py path in hook matches your system

Index Not Growing

1. Check commit count: cat .claude-os/.commit_count
2. Make sure you're making actual commits (not --amend)
3. Manually trigger expansion: python3 incremental_indexer.py 4 [project_path]

Too Many Files Being Found

The incremental indexer may find more files than expected if:

1. node_modules wasn't properly ignored (check git status)
2. Vendor directories have many code files
3. Generated files or caches have code-like extensions

Solution: Expand skip_dirs list in incremental_indexer.py for your project.

Future Enhancements

Potential improvements to consider:

1. Smart file prioritization: Index files by change frequency
2. Selective expansion: Index only recently-modified files
3. Parallel indexing: Speed up expansion passes
4. ML-based prioritization: Index files most relevant to user queries
5. Background indexing: Run during idle time instead of on commits

---

Built with Claude OS - Intelligent context management for AI-assisted development