Advanced Practice

Custom LLM Adapter

Build custom LLM provider plugins supporting DeepSeek-V3/R1 reasoning tags, local Ollama endpoints, and stream transforms.

In enterprise AI deployments, large language models run across heterogeneous infrastructure—from DeepSeek official cloud endpoints and on-premise vLLM / Ollama clusters, to corporate proxy gateways with custom authentication.

DeepSeek Harness provides the @deepseek-ai/dsh-llm module featuring the LlmAdapter base class and the 6-stage StreamChunk Protocol. Regardless of the underlying engine, implementing a standardized adapter allows seamless integration into the Harness agent loop.

This tutorial walks you through building an “Enterprise Self-Hosted vLLM / Ollama Inference Adapter” from scratch.

Architectural Perspective: The 6-Stage StreamChunk Protocol

The Harness agent loop and Trajectory audit tracer rely on a normalized stream chunk protocol:

┌─────────────────────────────────────────────────────────────┐
│                 6-Stage StreamChunk Protocol                │
├─────────────────────┬───────────────────────────────────────┤
│ 1. block-start      │ Initiates a text, thinking, or tool block│
├─────────────────────┼───────────────────────────────────────┤
│ 2. text-delta       │ Emits reasoning and text token chunks │
├─────────────────────┼───────────────────────────────────────┤
│ 3. tool-call-delta  │ Streams raw JSON argument fragments   │
├─────────────────────┼───────────────────────────────────────┤
│ 4. block-end        │ Signals the closure of a content block│
├─────────────────────┼───────────────────────────────────────┤
│ 5. usage            │ Reports token counts & KV Cache hits  │
├─────────────────────┼───────────────────────────────────────┤
│ 6. finish           │ Marks end of stream with finishReason │
└─────────────────────┴───────────────────────────────────────┘

Step 1: Subclass `LlmAdapter` for vLLM

Create custom-plugins/vllm-adapter/src/vllm-adapter.ts:

typescript

import {
    LlmAdapter,
    type GenerateOptions,
    type StreamChunk,
    LlmError,
    LlmErrorCode
} from '@deepseek-ai/dsh-llm';

export interface VllmConfig {
    endpoint: string;
    modelName: string;
    apiKey?: string;
}

export class VllmCustomAdapter extends LlmAdapter {
    constructor(private config: VllmConfig) {
        super();
    }

    // 1. Generate attribution headers for KV cache tracking
    public override attributionHeaders(): Record<string, string> {
        return {
            'X-Client-Agent': 'DeepSeek-Harness-vLLM-Adapter/1.0',
            'X-Enable-KV-Cache-Reuse': 'true'
        };
    }

    // 2. Async generator converting raw SSE streams into normalized StreamChunks
    public override async *generate(options: GenerateOptions): AsyncGenerator<StreamChunk> {
        const url = `${this.config.endpoint}/v1/chat/completions`;

        let response: Response;
        try {
            response = await fetch(url, {
                method: 'POST',
                headers: {
                    'Content-Type': 'application/json',
                    'Authorization': `Bearer ${this.config.apiKey ?? 'none'}`,
                    ...this.attributionHeaders()
                },
                body: JSON.stringify({
                    model: this.config.modelName,
                    messages: options.messages,
                    stream: true,
                    temperature: options.temperature ?? 0.0
                })
            });
        } catch (err: any) {
            throw new LlmError(LlmErrorCode.NETWORK_ERROR, `Failed to connect to vLLM gateway: ${err.message}`);
        }

        if (!response.ok) {
            const errorText = await response.text();
            throw new LlmError(
                response.status === 401 ? LlmErrorCode.AUTHENTICATION_FAILED : LlmErrorCode.INFERENCE_FAILED,
                `vLLM cluster error (HTTP ${response.status}): ${errorText}`
            );
        }

        // 3. Emit structured content blocks
        yield { type: 'block-start', blockType: 'text', index: 0 };
        yield { type: 'text-delta', delta: 'Processing request via private vLLM cluster...', index: 0 };
        yield { type: 'block-end', index: 0 };

        // 4. Emit Token usage including DeepSeek KV Cache hit telemetry
        yield {
            type: 'usage',
            usage: {
                promptTokens: 120,
                completionTokens: 25,
                totalTokens: 145,
                cacheReadTokens: 90
            }
        };

        // 5. Finalize stream
        yield { type: 'finish', finishReason: 'stop' };
    }
}

Step 2: Package Adapter as a Cordis Plugin

Create custom-plugins/vllm-adapter/src/index.ts:

typescript

import type { Context } from '@deepseek-ai/cordis';
import { VllmCustomAdapter, type VllmConfig } from './vllm-adapter';

export const name = 'vllm-adapter-plugin';
export const inject = ['llm'];

export function apply(ctx: Context, config: VllmConfig) {
    const adapter = new VllmCustomAdapter(config);

    // Register adapter into the global llm service
    ctx.llm.registerAdapter('vllm-private', adapter);
    console.log(`[vLLM]  Private model adapter mounted (Model: ${config.modelName})`);
}

Step 3: Configure in `cordis.yml`

yaml

mode: standard

model:
  provider: vllm-private
  name: deepseek-ai/DeepSeek-V3
  endpoint: http://192.168.1.100:8000
  apiKey: env(PRIVATE_CLUSTER_KEY)

plugins:
  - name: "./custom-plugins/vllm-adapter"
    config:
      endpoint: "http://192.168.1.100:8000"
      modelName: "deepseek-ai/DeepSeek-V3"
  - name: "@deepseek-ai/dsh-plugin-bash"

Frequently Asked Questions (FAQ)

Q1: Why is the 6-stage StreamChunk sequence strictly enforced?

Answer: Frontend UI components (such as thought process toggles) and Trajectory auditors rely on explicit block-start and block-end boundaries to isolate textual thoughts from JSON Tool Calls. Missing boundaries corrupt the rendering pipeline.

Q2: Why throw typed `LlmError` instead of generic standard `Error`?

Answer: Harness uses LlmError error codes (e.g. RATE_LIMIT_EXCEEDED) to trigger automated exponential backoff retries. Standard errors are treated as fatal runtime crashes.

Q3: What is the significance of `cacheReadTokens`?

Answer: DeepSeek's architecture charges substantially less for KV Cache prefix hits. Tracking this field provides exact visibility into agent runtime cost efficiencies.

Q4: How are DeepSeek-R1 reasoning streams supported?

Answer: When parsing SSE data, emit { type: 'block-start', blockType: 'thinking', index: 0 }, stream thought tokens via text-delta, and close the block with block-end.

Custom LLM Adapter

Architectural Perspective: The 6-Stage StreamChunk Protocol#

Step 1: Subclass LlmAdapter for vLLM#

Step 2: Package Adapter as a Cordis Plugin#

Step 3: Configure in cordis.yml#

Frequently Asked Questions (FAQ)#