SheWhoReignsSupreme/claude-os
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
claude-os/README.md
Bob Roberts 80d658105b docs: Simplify MCP Integration section - auto-generated by initialize-project 
Changed:
- Removed detailed MCP integration instructions
- Removed old example MCP names (pistn, pistn-agent-os)
- Simplified to note that /initialize-project creates 4 MCPs automatically
- Explained the 4 auto-generated MCPs and their purposes
- Directed users to Project Setup modal for MCP URLs

The initialize-project skill handles all MCP creation, so manual
configuration instructions were unnecessary and outdated.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-27 22:27:48 -05:00

707 lines
22 KiB
Markdown
Raw Blame History

# Claude OS
```text
╔═══════════════════════════════════════════════════════════════╗
║ ║
║ ██████╗██╗ █████╗ ██╗ ██╗██████╗ ███████╗ ██████╗ ███████╗
║ ██╔════╝██║ ██╔══██╗██║ ██║██╔══██╗██╔════╝ ██╔═══██╗██╔════╝
║ ██║ ██║ ███████║██║ ██║██║ ██║█████╗ ██║ ██║███████╗
║ ██║ ██║ ██╔══██║██║ ██║██║ ██║██╔══╝ ██║ ██║╚════██║
║ ╚██████╗███████╗██║ ██║╚██████╔╝██████╔╝███████╗ ╚██████╔╝███████║
║ ╚═════╝╚══════╝╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚══════╝ ╚═════╝ ╚══════╝
║ ║
║ Localized Multi-Knowledge-Base RAG System ║
║ with MCP Integration ║
║ ║
╚═══════════════════════════════════════════════════════════════╝
```
> **Production-grade RAG system** with SQLite vector search, Ollama LLMs, and Model Context Protocol. Runs 100% locally on your machine.
[![License: MIT](https://img.shields.io/badge/License-MIT-purple.svg)](LICENSE)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![SQLite](https://img.shields.io/badge/SQLite-3.0+-green.svg)](https://www.sqlite.org/)
[![Ollama](https://img.shields.io/badge/Ollama-Latest-pink.svg)](https://ollama.ai/)
---
## 🚀 What is Claude OS?
**Claude OS** is a **localized RAG (Retrieval-Augmented Generation) system** designed to make AI assistants deeply knowledgeable about your codebase and documentation. It combines:
- **🗄️ SQLite + sqlite-vec** - Lightweight vector database for semantic search
- **🤖 Ollama** - Local LLMs (llama3.1) with no API keys needed
- **🔌 MCP Integration** - Claude Desktop integration via Model Context Protocol
- **⚡ React UI** - Modern dashboard for managing knowledge bases
- **📚 Multi-KB Support** - Separate knowledge bases for different projects
- **🧠 Advanced RAG** - Hybrid search, reranking, and agentic modes
### Perfect For
✅ Making Claude deeply familiar with your codebase
✅ Private, secure knowledge base (never leaves your machine)
✅ Team collaboration (share with coworkers easily)
✅ Integration with Claude Desktop via MCP
✅ Building AI-assisted development workflows
---
## ⚡ Quick Start (5-10 minutes)
### Prerequisites
**You MUST have these installed first:**
1. **Python 3.11+** (required)
```bash
python3 --version
```
- macOS: `brew install python@3.11`
- Linux: `sudo apt-get install python3.11`
- Windows: Download from https://www.python.org/downloads/
2. **Git** (required)
```bash
git --version
```
- macOS: `brew install git`
- Linux: `sudo apt-get install git`
- Windows: Download from https://git-scm.com/
3. **Node.js 16+** (optional, for React frontend)
```bash
node --version
```
- macOS: `brew install node`
- Linux: `sudo apt-get install nodejs`
- Windows: Download from https://nodejs.org/
---
### Quick Install (3 Steps)
**Once you have Python 3.11+ and Git, run these commands:**
```bash
# 1. Clone the repository
git clone https://github.com/brobertsaz/claude-os.git
cd claude-os
# 2. Run the setup script
./setup.sh
# 3. Start all services
./start_all_services.sh
```
**The setup script will automatically:**
- ✅ Install Ollama (if needed) + download LLM models
- ✅ Install Redis (if needed) for caching/queues
- ✅ Set up Python virtual environment
- ✅ Install all Python dependencies
- ✅ Create SQLite database
- ✅ Install frontend dependencies (if Node.js present)
**Done!** Visit http://localhost:5173 to start using Claude OS.
### Installation (Step by Step)
**Step 1: Install Prerequisites** (one-time only)
Choose your OS:
**macOS:**
```bash
# Install Python 3.11
brew install python@3.11
# Install Git
brew install git
# Optional: Install Node.js for frontend
brew install node
```
**Linux (Ubuntu/Debian):**
```bash
# Install Python 3.11
sudo apt-get update
sudo apt-get install python3.11 python3.11-venv
# Install Git
sudo apt-get install git
# Optional: Install Node.js for frontend
sudo apt-get install nodejs npm
```
**Verify installation:**
```bash
python3 --version # Should be 3.11+
git --version # Should exist
```
---
**Step 2: Clone the Repository**
```bash
git clone https://github.com/brobertsaz/claude-os.git
cd claude-os
```
---
**Step 3: Run the Installer**
```bash
./setup.sh
```
This automatically installs:
- ✅ **Ollama** - LLM engine (auto-installed if missing)
- ✅ **Redis** - Cache & queues (auto-installed if missing)
- ✅ **Python dependencies** - All required packages
- ✅ **SQLite database** - Local vector store
- ✅ **Frontend** - React UI (if Node.js present)
The script will download ~5-10 GB of LLM models (llama3.1, embeddings).
---
**Step 4: Start Services**
```bash
./start_all_services.sh
```
This starts:
- 🔌 **MCP Server** (port 8051) - Backend RAG engine
- 🎨 **React UI** (port 5173) - Web dashboard
- 🤖 **Ollama** (port 11434) - LLM service
- 💾 **Redis** (port 6379) - Cache & queues
- 🧠 **RQ Workers** - Real-time learning system
---
**Step 5: Access the Application**
Open your browser and visit:
- **Frontend:** http://localhost:5173
- **API Docs:** http://localhost:8051/docs
Start uploading documents to create knowledge bases!
---
## 🎯 Initialize Your Project (The Magic Sauce ✨)
Once Claude OS is running, follow these 3 simple steps to make Claude an expert on your codebase:
### Step 1: Create a Project in Claude OS UI
1. **Open the UI** - Visit http://localhost:5173 in your browser
2. **Click "Create Project"** button
3. **Fill in the form:**
- **Project Name** - e.g., "my-awesome-app"
- **Project Path** - Select your project directory (e.g., `/Users/you/Projects/my-awesome-app`)
- **Description** - (optional) Brief description of your project
4. **Click "Create Project"**
Your project is now registered in Claude OS!
### Step 2: Get Your Project ID
After creating your project, you'll see it listed in the Projects view. The **Project ID** is displayed right on the project card (e.g., `#1`, `#2`, etc.).
Simply note the number - that's your project ID!
**Example:**
- If you see `#1` on your project card, your project ID is `1`
- If you see `#3` on your project card, your project ID is `3`
### Step 3: Initialize Project with Claude Code
This is where the magic happens! The initialization will:
- ✅ Analyze your entire codebase (~5 minutes)
- ✅ Generate coding standards & architecture docs
- ✅ Index 50 key files (~800 code chunks)
- ✅ Set up Git hooks for auto-indexing
- ✅ Create 4 Knowledge Bases automatically
**Open Claude Code and copy-paste this command** (replace `1` with your project ID):
```
/initialize-project 1
```
**⚠️ Important:** Replace `1` with your actual project ID from Step 2
**Examples:**
- If your project ID is `1`: `/initialize-project 1`
- If your project ID is `2`: `/initialize-project 2`
- If your project ID is `3`: `/initialize-project 3`
**What happens after you run it:**
1. Claude analyzes your project structure, patterns, and conventions (takes ~5 minutes)
2. Generates 3 documentation files:
- `CODING_STANDARDS.md` - Your code style
- `ARCHITECTURE.md` - Your system design
- `DEVELOPMENT_PRACTICES.md` - How you build software
3. Creates semantic indexes for 50 key files (~800+ code chunks)
4. Installs Git hooks to auto-update knowledge on commits
5. Creates 4 Knowledge Bases for use with Claude Desktop
**When it's done:**
✅ Claude is now an expert on your project!
✅ Ask Claude about your codebase and it will know the answers
✅ Every git commit automatically updates Claude's knowledge
---
## 🔌 Auto-Generated MCPs
After running `/initialize-project`, Claude OS automatically creates **4 MCPs (Model Context Protocol endpoints)** for your project:
1. **📚 knowledge_docs** - Your project documentation
2. **📋 project_profile** - Project analysis and overview
3. **🗂️ project_index** - Code structure and file index
4. **💾 project_memories** - Persistent memory and insights
These MCPs are immediately available and can be added to Claude Desktop using the URLs shown in the Project Setup modal.
**To add them to Claude Desktop:**
- Open the Project Setup modal (⚙️ settings icon on your project card)
- Copy the MCP URLs displayed
- Use `claude mcp add` commands or configure manually
---
## 📚 Managing Knowledge Bases
### Create a Knowledge Base
1. Visit http://localhost:5173
2. Click "Create Knowledge Base"
3. Choose type:
- **GENERIC** - General documentation
- **CODE** - Source code repositories
- **DOCUMENTATION** - Technical docs
- **AGENT_OS** - Spec-driven development
### Upload Documents
1. Select a knowledge base
2. Click "Upload Documents"
3. Choose files (supports .md, .txt, .pdf, .py, .js, .ts, .json, .yaml)
4. Watch as documents are indexed
### Query the Knowledge Base
**Via Web UI:**
1. Select a KB from the dropdown
2. Type your question
3. View answer with source citations
**Via Claude Desktop:**
Once added as MCP server, Claude can query automatically when working on your project.
---
## 🏗️ Architecture
```
┌─────────────────────────────────────────────────┐
│ Claude Desktop │
│ (or any MCP client) │
└───────────────────┬─────────────────────────────┘
│ MCP HTTP
┌───────────────────▼─────────────────────────────┐
│ MCP Server (Port 8051) │
│ FastAPI Backend │
└───────────────────┬─────────────────────────────┘
│
┌───────────┴────────────┐
│ │
┌───────▼────────────┐ ┌────────▼──────────┐
│ RAG Engine │ │ React UI │
│ (llama-index) │ │ (Port 5173) │
│ • Vector Search │ │ │
│ • Hybrid Search │ └────────────────────┘
│ • Reranking │
└───────┬────────────┘
│
┌───────▼────────────────────────────────────────┐
│ SQLite + sqlite-vec (Local Database) │
│ • knowledge_bases │
│ • documents (with embeddings) │
│ • Single-file database │
└────────────────────────────────────────────────┘
│
└─────────────────────┐
│
┌─────────▼──────┐
│ Ollama │
│ (Port 11434) │
│ • llama3.1 │
│ • Embeddings │
└────────────────┘
```
---
## ⚙️ Configuration
### Environment Variables
The system uses sensible defaults, but you can customize via environment variables:
```bash
# SQLite Database
SQLITE_DB_PATH=data/claude-os.db # Default: data/claude-os.db
# Ollama
OLLAMA_HOST=http://localhost:11434 # Default: localhost:11434
OLLAMA_MODEL=llama3.1:latest # Default: llama3.1:latest
# MCP Server
MCP_SERVER_HOST=0.0.0.0 # Default: 0.0.0.0
MCP_SERVER_PORT=8051 # Default: 8051
```
---
## 📊 Performance
**Native Ollama Setup (Current)**
- Response time: ~40 seconds per query
- GPU acceleration: Full Metal GPU on Apple Silicon
- Memory usage: 8-10GB (models + context)
- CPU usage: 12 cores (M4 Pro, leaves 2 for system)
**Why it's fast:**
- Direct GPU acceleration (no virtualization)
- Efficient vector search in SQLite
- Optimized RAG engine with caching
- Single-file database with minimal overhead
---
## 🛠️ Scripts Guide
Claude OS includes comprehensive shell scripts for setup, service management, and testing. Here's what each one does:
### Installation & Setup
#### `./setup.sh` - Complete Setup ⭐ **USE THIS ONE**
```bash
./setup.sh
```
**This is the standard setup script that 99% of users need.**
**Prerequisites (must be installed first):**
- ✅ Python 3.11+ (required - script will fail if missing)
- ✅ Git (required - script will fail if missing)
- ⚠️ Node.js 16+ (optional - script skips frontend if missing)
**What it automatically installs:**
- ✅ **Ollama** - Downloads and starts if not installed
- ✅ **Redis** - Installs and starts if not present
- ✅ **Python virtual environment** - Isolated environment
- ✅ **Python dependencies** - All packages from `requirements.txt`
- ✅ **LLM models** - llama3.1 and nomic-embed-text (~5-10 GB)
- ✅ **SQLite database** - Local vector store
- ✅ **Frontend dependencies** - npm packages (if Node.js present)
**Supported platforms:** macOS & Linux
**When to use:**
- ✅ **First-time setup (recommended for 99% of users)**
- ✅ You have Python 3.11+ and Git already installed
- ✅ You don't have Ollama or Redis yet
- ✅ You want a complete, automated setup
- ✅ You're on any Unix system (macOS/Linux)
---
#### `./setup_native.sh` - Fast macOS Setup (Advanced Users Only)
```bash
./setup_native.sh
```
**This is an optional lightweight setup for macOS users who already have Ollama installed via Homebrew.**
**What it does:**
- ✅ Verifies Ollama is already installed via Homebrew (fails if not)
- ✅ Pulls required LLM models (llama3.1, nomic-embed-text)
- ✅ Sets up Python virtual environment
- ✅ Installs Python dependencies
- ✅ Creates data/ and logs/ directories
- ✅ Initializes SQLite database
- ⚠️ Does NOT install Redis
**When to use (rare):**
- ⚠️ Only if you already have **Ollama installed via `brew install ollama`**
- ⚠️ You're on macOS only
- ⚠️ You already have Redis running separately
**Note:** Most users should use `./setup.sh` instead. Only use this if you know what you're doing.
---
### Service Management
#### `./start_all_services.sh` - Start Everything
```bash
./start_all_services.sh
```
**Starts:**
- 🔌 **MCP Server** (port 8051) - Backend RAG engine
- 🎨 **React Frontend** (port 5173) - Web dashboard
- 🤖 **RQ Workers** - Real-time learning system
- 💾 **Redis** - Cache & message queue
- 🧠 **Ollama** - LLM service
**Features:**
- Automatic health checks for all services
- Creates required directories if missing
- Checks that ports are free
- Logs all services to `logs/` directory
- Shows service URLs and PIDs
**Output:**
```
Service URLs:
🎨 Frontend: http://localhost:5173
🔌 API Server: http://localhost:8051
📚 API Docs: http://localhost:8051/docs
Log Files:
MCP Server: logs/mcp_server.log
Frontend: logs/frontend.log
RQ Workers: logs/rq_workers.log
```
---
#### `./stop_all_services.sh` - Stop All Services
```bash
./stop_all_services.sh
```
**Stops:**
- 🔌 MCP Server (port 8051)
- 🎨 React Frontend (port 5173)
- 🤖 RQ Workers (real-time learning)
- 💾 Redis
**Note:** Ollama is preserved (may be used by other apps)
---
#### `./restart_services.sh` - Restart Everything
```bash
./restart_services.sh
```
**Does:**
1. Stops all services gracefully
2. Waits for ports to be released (3 second delay)
3. Starts all services fresh
**Use when:** Code changes aren't reflected or services need a clean restart
---
#### `./start_mcp_server.sh` - Start Only MCP Server
```bash
./start_mcp_server.sh
```
**Starts:**
- 🔌 MCP Server (port 8051)
**Checks:**
- ✅ Ollama is running
- ✅ SQLite database exists
- ✅ Python environment is ready
**Use when:** You only need the backend API (e.g., for testing)
---
#### `./start_redis_workers.sh` - Start RQ Workers
```bash
./start_redis_workers.sh
```
**Starts:**
- 🤖 RQ Workers for real-time learning system
- Listening on queues: `claude-os:learning`, `claude-os:prompts`, `claude-os:ingest`
- Scheduler for periodic tasks
**Checks:**
- ✅ Redis is running
- ✅ Python virtual environment exists
- ✅ RQ dependencies are installed
**Use when:** Running the real-time learning system separately
---
### Testing
#### `./run_tests.sh` - Run Test Suite
```bash
./run_tests.sh [OPTIONS]
```
**Options:**
```bash
./run_tests.sh # Run all tests
./run_tests.sh --unit # Unit tests only
./run_tests.sh --integration # Integration tests only
./run_tests.sh --vector # Vector DB tests
./run_tests.sh --rag # RAG engine tests
./run_tests.sh --api # API endpoint tests
./run_tests.sh --coverage # Generate coverage report
./run_tests.sh --verbose # Detailed output
./run_tests.sh --unit --coverage # Combine options
```
**Features:**
- Validates PostgreSQL connection
- Creates test database if needed
- Shows test configuration
- Generates HTML coverage report (with `--coverage`)
---
## 📁 Project Structure
```
claude-os/
├── app/ # Backend application
│ ├── core/ # Core modules
│ │ ├── config.py # Configuration management
│ │ ├── sqlite_manager.py # SQLite interface
│ │ ├── rag_engine.py # RAG logic
│ │ └── ...
│ ├── db/ # Database schemas
│ └── ...
├── frontend/ # React UI (Vite)
│ ├── src/
│ │ ├── pages/
│ │ ├── components/
│ │ └── ...
│ └── package.json
├── mcp_server/ # MCP Server (HTTP)
│ ├── server.py # FastAPI + MCP endpoints
│ └── ...
├── migrations/ # Database migrations
├── data/ # SQLite database (auto-created)
│ └── claude-os.db
├── logs/ # Service logs (auto-created)
├── .sh scripts # Utility scripts (see Scripts Guide above)
├── requirements.txt # Python dependencies
└── README.md # This file
```
---
## 🐛 Troubleshooting
### Database Issues
The SQLite database is automatically created at `data/claude-os.db`. If you encounter issues:
```bash
# Check if database exists
ls -lh data/claude-os.db
# Remove and recreate (WARNING: deletes all data)
rm data/claude-os.db
# Restart the server to recreate
```
### Ollama Issues
```bash
# Check if Ollama is running
ollama list
# Start Ollama manually
ollama serve
# Check for specific model
ollama list | grep llama3.1
```
### Port Already in Use
```bash
# Find process on port 8051 (MCP Server)
lsof -i :8051
# Find process on port 5173 (React UI)
lsof -i :5173
# Kill if needed (replace PID with actual process ID)
kill -9 <PID>
```
### MCP Server Not Starting
```bash
# Check logs
tail -f /tmp/mcp_server.log
# Verify Python environment
source venv/bin/activate
python mcp_server/server.py
# Check port 8051 is accessible
curl http://localhost:8051/health
```
---
## 📖 Additional Documentation
### Core Features
- **[docs/SELF_LEARNING_SYSTEM.md](docs/SELF_LEARNING_SYSTEM.md)** - 🧠 How Claude learns from your conversations automatically
- **[docs/REAL_TIME_LEARNING_GUIDE.md](docs/REAL_TIME_LEARNING_GUIDE.md)** - Real-time learning system usage guide
- **[docs/MEMORY_MCP_GUIDE.md](docs/MEMORY_MCP_GUIDE.md)** - Persistent memory across sessions
### Technical Documentation
- **[README_NATIVE_SETUP.md](README_NATIVE_SETUP.md)** - Detailed native setup guide
- **[NATIVE_VS_DOCKER_DECISION.md](NATIVE_VS_DOCKER_DECISION.md)** - Why we chose native Ollama
- **[PERFORMANCE_TEST_RESULTS.md](PERFORMANCE_TEST_RESULTS.md)** - Benchmark results
- **[SLUG_IMPLEMENTATION_SUMMARY.md](SLUG_IMPLEMENTATION_SUMMARY.md)** - URL slug system
- **[MCP_KB_ENDPOINTS.md](MCP_KB_ENDPOINTS.md)** - MCP endpoint documentation
---
## 🤝 Contributing
This is a personal development tool. Feel free to:
- Modify for your specific needs
- Add new KB types
- Optimize RAG strategies
- Contribute improvements back
---
## 📄 License
MIT License - Use it freely!
---
## 🎯 Next Steps
1. **First-time setup:** Run `./setup_native.sh`
2. **Start services:** Run `./start_all_services.sh`
3. **Create knowledge base:** Visit http://localhost:5173
4. **Upload documents:** Add your codebase/docs
5. **Integrate with Claude:** Run `claude mcp add ...`
6. **Start coding:** Ask Claude about your project!
Happy coding! 🚀
Reference in a new issue View git blame Copy permalink