DeepWiki Rust High-Performance Markdown Engine

DeepWiki Rust High-Performance Markdown Engine

@sopaco/deepwiki-rs·v1.3.2·storage

A blazing-fast native Rust documentation compiler and BM25 indexer for DeepSeek Harness. Delivers millisecond Markdown parsing, bidirectional wikilink graphs, and zero-GC memory safety.

Rust-Engine · Markdown-Compiler · BM25-Search · Wikilinks

GitHubNPM
Installation & Integration
bash
manager:
$ pnpm add @sopaco/deepwiki-rs

Architecture

DeepWiki Rust eliminates document parsing bottlenecks across massive enterprise documentation libraries. When technical repositories scale past thousands of markdown files, JavaScript-based parsers hit severe V8 garbage collection pauses and memory bloat. This plugin integrates a native Rust-compiled dynamic engine directly into Cordis. Leveraging SIMD-accelerated lexical scanning, it tokenizes and compiles massive Markdown ASTs in microseconds. It generates bidirectional cross-link graphs instantly and powers lightning-fast local BM25 keyword searches with zero GC jitter.

Core Features

01

Native Rust Raw Throughput: 50x-80x faster than pure JS parsers, handling millions of words

02

Zero Garbage-Collection Overhead: Completely eliminates V8 GC pauses and memory spikes

03

Bidirectional Wikilink Graph: Native support for Obsidian-style networked thought associative links

04

SIMD-Accelerated BM25 Indexer: Blends keyword precision with extreme single-node search performance

Core Workflow

01

Parallel Native Parsing

Loads Markdown files concurrently with Rayon, building compact AST nodes.

02

Bidirectional Link Graph

Extracts `[[wikilink]]` references, constructing a bidirectional graph matrix.

03

SIMD BM25 Indexing

Indexes text with SIMD CPU instructions for sub-millisecond BM25 keyword recall.

04

Zero-Copy Data Delivery

Transfers search results directly into agent context via shared memory buffers.

Configuration Parameters Reference (YAML / JSON)

ParameterTypeDefaultDescription
workerThreadsnumber4Number of native Rayon worker threads
enableWikilinksbooleantrueEnable `[[wikilink]]` syntax parsing
maxSearchMatchesnumber10Maximum BM25 candidate results returned