Open Viking Headless Scraper & Content Extractor

Open Viking Headless Scraper & Content Extractor

@open-viking/scraper·v1.2.6·tools

A resilient headless web scraping and markdown distillation kernel for DeepSeek Harness. Features automated bot-defense evasion, dynamic SPA rendering, and token-efficient content distillation.

Scraper · Headless · Extraction · Readability

GitHubNPM
Installation & Integration
bash
manager:
$ pnpm add @open-viking/scraper

Architecture

Open Viking powers autonomous web exploration across resilient modern web applications. Modern web pages rely heavily on client-side JS hydration and anti-bot challenges that trip naive HTTP clients. Built on top of native Chromium CDP, Open Viking simulates natural human browsing behavior, transparently bypassing common bot defenses. Captured DOM trees undergo aggressive content distillation, eliminating sidebars, banners, and scripts to produce pristine, high-density Markdown ready for LLM consumption.

Core Features

01

Adaptive Bot Evasion: Circumvents bot challenges and TLS fingerprint tracking transparently

02

Full SPA Hydration: Accurately renders dynamic React, Vue, and Angular client-rendered data

03

Aggressive Token Pruning: Eliminates up to 75% of noisy DOM tags, saving valuable context

04

Table & Math Preservation: Accurately serializes complex data tables and formulas into Markdown

Core Workflow

01

Fingerprint Masquerade

Initializes randomized TLS fingerprints and user-agent rotations before fetching.

02

Headless DOM Hydration

Executes dynamic JS scripts and waits for main content nodes to hydrate.

03

Noise Pruning & Heuristics

Strips navbars, cookie banners, tracking iframes, and promotional artifacts.

04

Markdown Serialization

Converts clean content into token-compact Markdown, preserving tables and code.

Configuration Parameters Reference (YAML / JSON)

ParameterTypeDefaultDescription
renderTimeoutnumber15000Page hydration timeout in milliseconds
extractImagesbooleanfalsePreserve image links and captions in output
blockResourcesstringfont,mediaResource types blocked to accelerate throughput