Process and Analyze Large Files Efficiently

Large File MCP Server

MCP server for intelligent handling of large files - smart chunking, search, navigation, and streaming.

🔗 Live Demo / Landing Page - install with npx @willianpinho/large-file-mcp

📚 Full Documentation | API Reference | Examples

Features

• Smart Chunking - Automatically determines optimal chunk size based on file type
• Intelligent Navigation - Jump to specific lines with surrounding context
• Powerful Search - Regex support with context lines before/after matches
• File Analysis - Comprehensive metadata and statistical analysis
• Memory Efficient - Stream files of any size without loading into memory
• Performance Optimized - Built-in LRU caching for frequently accessed chunks
• Type Safe - Written in TypeScript with strict typing
• Cross-Platform - Works on Windows, macOS, and Linux

Installation

npm install -g @willianpinho/large-file-mcp

Or use directly with npx:

npx @willianpinho/large-file-mcp

Quick Start

Claude Code CLI

Add the MCP server using the CLI:

# Add for current project only (local scope)
claude mcp add --transport stdio --scope local large-file-mcp -- npx -y @willianpinho/large-file-mcp

# Add globally for all projects (user scope)
claude mcp add --transport stdio --scope user large-file-mcp -- npx -y @willianpinho/large-file-mcp

Verify installation:

claude mcp list
claude mcp get large-file-mcp

Remove if needed:

# Remove from local scope
claude mcp remove large-file-mcp -s local

# Remove from user scope
claude mcp remove large-file-mcp -s user

MCP Scopes:

• local - Available only in the current project directory
• user - Available globally for all projects
• project - Defined in .mcp.json for team sharing

Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "large-file": {
      "command": "npx",
      "args": ["-y", "@willianpinho/large-file-mcp"]
    }
  }
}

Config file locations:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

Restart Claude Desktop after editing.

Other AI Platforms

Gemini:

{
  "tools": [
    {
      "name": "large-file-mcp",
      "command": "npx @willianpinho/large-file-mcp",
      "protocol": "mcp"
    }
  ]
}

Usage

Once configured, you can use natural language to interact with large files:

Read the first chunk of /var/log/system.log

Find all ERROR messages in /var/log/app.log

Show me line 1234 of /code/app.ts with context

Get the structure of /data/sales.csv

Available Tools

read_large_file_chunk

Read a specific chunk of a large file with intelligent chunking.

Parameters:

• filePath (required): Absolute path to the file
• chunkIndex (optional): Zero-based chunk index (default: 0)
• linesPerChunk (optional): Lines per chunk (auto-detected if not provided)
• includeLineNumbers (optional): Include line numbers (default: false)

Example:

{
  "filePath": "/var/log/system.log",
  "chunkIndex": 0,
  "includeLineNumbers": true
}

search_in_large_file

Search for patterns in large files with context.

Parameters:

• filePath (required): Absolute path to the file
• pattern (required): Search pattern
• caseSensitive (optional): Case sensitive search (default: false)
• regex (optional): Use regex pattern (default: false)
• maxResults (optional): Maximum results (default: 100)
• contextBefore (optional): Context lines before match (default: 2)
• contextAfter (optional): Context lines after match (default: 2)

Example:

{
  "filePath": "/var/log/error.log",
  "pattern": "ERROR.*database",
  "regex": true,
  "maxResults": 50
}

get_file_structure

Analyze file structure and get comprehensive metadata.

Parameters:

• filePath (required): Absolute path to the file

Returns: File metadata, line statistics, recommended chunk size, and sample lines.

navigate_to_line

Jump to a specific line with surrounding context.

Parameters:

• filePath (required): Absolute path to the file
• lineNumber (required): Line number to navigate to (1-indexed)
• contextLines (optional): Context lines before/after (default: 5)

get_file_summary

Get comprehensive statistical summary of a file.

Parameters:

• filePath (required): Absolute path to the file

Returns: File metadata, line statistics, character statistics, and word count.

stream_large_file

Stream a file in chunks for processing very large files.

Parameters:

• filePath (required): Absolute path to the file
• chunkSize (optional): Chunk size in bytes (default: 64KB)
• startOffset (optional): Starting byte offset (default: 0)
• maxChunks (optional): Maximum chunks to return (default: 10)

Supported File Types

The server intelligently detects and optimizes for:

• Text files (.txt) - 500 lines/chunk
• Log files (.log) - 500 lines/chunk
• Code files (.ts, .js, .py, .java, .cpp, .go, .rs, etc.) - 300 lines/chunk
• CSV files (.csv) - 1000 lines/chunk
• JSON files (.json) - 100 lines/chunk
• XML files (.xml) - 200 lines/chunk
• Markdown files (.md) - 500 lines/chunk
• Configuration files (.yml, .yaml, .sh, .bash) - 300 lines/chunk

Configuration

Customize behavior using environment variables:

Example with custom settings (Claude Desktop):

{
  "mcpServers": {
    "large-file": {
      "command": "npx",
      "args": ["-y", "@willianpinho/large-file-mcp"],
      "env": {
        "CHUNK_SIZE": "1000",
        "CACHE_ENABLED": "true"
      }
    }
  }
}

Example with custom settings (Claude Code CLI):

claude mcp add --transport stdio --scope user large-file-mcp \
  --env CHUNK_SIZE=1000 \
  --env CACHE_ENABLED=true \
  -- npx -y @willianpinho/large-file-mcp

Examples

Analyzing Log Files

Analyze /var/log/nginx/access.log and find all 404 errors

The AI will use the search tool to find patterns and provide context around each match.

Code Navigation

Find all function definitions in /project/src/main.py

Uses regex search to locate function definitions with surrounding code context.

CSV Data Exploration

Show me the structure of /data/sales.csv

Returns metadata, line count, sample rows, and recommended chunk size.

Large File Processing

Stream the first 100MB of /data/huge_dataset.json

Uses streaming mode to handle very large files efficiently.

Performance

Caching

• LRU Cache with configurable size (default 100MB)
• TTL-based expiration (default 5 minutes)
• Speeds up repeated access to the same file chunk or structure lookup

Cache hit-rate depends entirely on your access pattern (how often you re-request
the same chunk), so there's no single universal number - run pnpm run benchmark
to measure it against a documented, repeatable "hot chunk" workload. On that
workload it currently measures 95% (95/100 accesses hit the cache).

Memory Management

• Streaming architecture - files are read line-by-line, never fully loaded
• Configurable chunk sizes - adjust based on your use case
• Smart buffering - minimal memory footprint for search operations

Read Latency (measured)

Uncached readChunk() latency from scripts/benchmark.mjs, generated fixtures,
Apple M1 Pro laptop - re-run the script on your own hardware for your own numbers:

Larger tiers (100MB-1GB, >1GB) aren't benchmarked here - the streaming
architecture (line-by-line readline, never loading the full file) means time
scales roughly linearly with size, and streamFile() switches to an
AsyncGenerator so memory stays flat regardless of file size.

Development

Building from Source

git clone https://github.com/willianpinho/large-file-mcp.git
cd large-file-mcp
pnpm install
pnpm build

Development Mode

pnpm dev    # Watch mode
pnpm lint   # Run linter
pnpm start  # Run server

Project Structure

src/
├── index.ts          # Entry point
├── server.ts         # MCP server implementation
├── fileHandler.ts    # Thin barrel — delegates to src/file-handler/*
├── file-handler/      # Chunking, search, navigation, streaming (split by concern)
├── cacheManager.ts   # Caching implementation
└── types.ts          # TypeScript type definitions

Troubleshooting

File not accessible

Ensure the file path is absolute and the file has read permissions:

chmod +r /path/to/file

Out of memory

1. Reduce CHUNK_SIZE environment variable
2. Disable cache with CACHE_ENABLED=false
3. Use stream_large_file for very large files

Slow search performance

1. Reduce maxResults parameter
2. Use startLine and endLine to limit search range
3. Ensure caching is enabled

Claude Code CLI: MCP server not found

Check if the server is installed:

claude mcp list

If not listed, reinstall:

claude mcp add --transport stdio --scope user large-file-mcp -- npx -y @willianpinho/large-file-mcp

Check server health:

claude mcp get large-file-mcp

Designed For

• Log Analysis: Processing multi-GB log files with search and navigation
• Data Processing: Reading large CSV/JSON files in manageable chunks
• Code Review: Navigating large codebases efficiently
• System Monitoring: Analyzing system logs and debug outputs
• Document Analysis: Processing large text documents

Compatible with Claude Code, Cursor, and Gemini CLI. Available on npm and the Glama.ai registry, and listed in community awesome-MCP directories.

scripts/benchmark.mjs

#!/usr/bin/env node
/**
 * Benchmark script for the README "Performance" section.
 *
 * Measures real numbers on real generated fixtures instead of hand-typed
 * estimates: (1) FileHandler.readChunk() latency for a few file-size tiers,
 * and (2) CacheManager hit-rate under a documented, repeatable access
 * pattern. Re-run with `pnpm run benchmark` whenever these numbers need
 * refreshing; results depend on the machine they run on, so treat them as
 * a reproducible reference point, not a universal guarantee.
 *
 * Requires a build first: `pnpm build && node scripts/benchmark.mjs`.
 */

import { mkdtempSync, rmSync, openSync, writeSync, closeSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { FileHandler } from '../dist/fileHandler.js';
import { CacheManager } from '../dist/cacheManager.js';

const LINE = 'The quick brown fox jumps over the lazy dog. 0123456789.\n';
const LINE_BYTES = Buffer.byteLength(LINE, 'utf-8');

function makeFile(dir, name, targetBytes) {
  const filePath = join(dir, name);
  const totalLines = Math.ceil(targetBytes / LINE_BYTES);
  const linesPerWrite = 10000;
  const bigChunk = LINE.repeat(linesPerWrite);

  const fd = openSync(filePath, 'w');
  let written = 0;
  while (written < totalLines) {
    const remaining = totalLines - written;
    if (remaining >= linesPerWrite) {
      writeSync(fd, bigChunk);
      written += linesPerWrite;
    } else {
      writeSync(fd, LINE.repeat(remaining));
      written += remaining;
    }
  }
  closeSync(fd);
  return filePath;
}

async function benchmarkReadLatency(filePath) {
  const start = performance.now();
  await FileHandler.readChunk(filePath, 0);
  return performance.now() - start;
}

/**
 * Simulates a realistic navigation pattern: a handful of "hot" chunks get
 * revisited far more often than "cold" ones (e.g. re-reading the same
 * function definition while a user asks follow-up questions).
 */
function benchmarkCacheHitRate() {
  const cache = new CacheManager({
    enabled: true,
    maxSize: 10 * 1024 * 1024,
    ttl: 5 * 60 * 1000,
  });

  const hotKeys = ['chunk:0', 'chunk:1', 'chunk:2'];
  const coldKeys = Array.from({ length: 10 }, (_, i) => `chunk:cold-${i}`);
  const accessPattern = [];
  for (let i = 0; i < 100; i++) {
    // 80% of accesses hit one of the 3 hot chunks, 20% hit a unique cold one.
    accessPattern.push(i % 5 === 0 ? coldKeys[i % coldKeys.length] : hotKeys[i % hotKeys.length]);
  }

  let hits = 0;
  let misses = 0;
  for (const key of accessPattern) {
    const cached = cache.get(key);
    if (cached !== undefined) {
      hits++;
    } else {
      misses++;
      cache.set(key, key.repeat(100));
    }
  }

  return {
    totalAccesses: accessPattern.length,
    hits,
    misses,
    hitRatePercent: Math.round((hits / accessPattern.length) * 100),
  };
}

async function main() {
  const dir = mkdtempSync(join(tmpdir(), 'large-file-mcp-bench-'));

  try {
    const tiers = [
      { label: '~500KB', bytes: 500 * 1024 },
      { label: '~5MB', bytes: 5 * 1024 * 1024 },
      { label: '~50MB', bytes: 50 * 1024 * 1024 },
    ];

    console.log('## Read latency (uncached, readChunk on chunk 0)\n');
    for (const tier of tiers) {
      const filePath = makeFile(dir, `${tier.label}.txt`, tier.bytes);
      const elapsedMs = await benchmarkReadLatency(filePath);
      console.log(`${tier.label.padEnd(8)} -> ${elapsedMs.toFixed(1)}ms`);
    }

    console.log('\n## Cache hit-rate (100 accesses, 3 hot + 10 cold keys, 80/20 split)\n');
    const cacheResult = benchmarkCacheHitRate();
    console.log(
      `${cacheResult.hits}/${cacheResult.totalAccesses} hits -> ${cacheResult.hitRatePercent}% hit rate`
    );
  } finally {
    rmSync(dir, { recursive: true, force: true });
  }
}

main().catch(err => {
  console.error(err);
  process.exit(1);
});