Engineered Ingestion Architecture.
Explore Reworkd's high-speed crawling clusters, locality-sensitive hashing deduplication filters, and semantic vector formatting pipelines built specifically for model pretraining.
High-Throughput Scrapers
Reworkd operates a decentralized cluster of headless crawling nodes dynamically shifting user agents, viewport scopes, and request throttling configurations to emulate human interactions.
When encountering complex client-side applications (React, Angular, Next.js), we deploy a headless Chromium context (Playwright engine) that waits for async API calls to resolve and compile selectors into structured DOM representations, auto-throttling requests inline to prevent IP rate-limiting.
- Rotating Proxy subnets
- Automated Header spoofing
- CNN Visual slider solver
- Dynamic layout compilations
MinHash LSH Deduplicators
For generative models, repeating web page code elements represent overfitting risks. Standard deduplication methods require comparing every page to every other page, which becomes computationally impossible at billion-page scales.
Reworkd resolves this by chunking pages into 3-word shingles, generating integer hash keys, and selecting the minimum hash outputs (MinHash signatures). By partitioning signatures into index bands (Locality-Sensitive Hashing), workers find high Jaccard similarity indices in sub-millisecond rates, stripping repeating boilerplate rows automatically.
- Locality-Sensitive Hashing
- Word 3-Gram Shingling
- MinHash integer signature signatures
- Dynamic Jaccard parameters
Semantic Embedding Pools
Vector databases act as the knowledge retrieval system for Retrieval-Augmented Generation (RAG) models. Reworkd pipelines automatically format scraped text blocks into semantic units using recursive sliding character tokens.
We parse clean Markdown chunks, calculate text embeddings using host models (such as Cohere Embed or OpenAI text-embedding-3), and insert vectors directly into databases.
- 1536-dim float embeddings
- Sliding overlap padding
- Direct vector store sync
- Hugging Face parquet buffers
from pymilvus import Collection, connections
connections.connect("default", host="api.reworkd.ai", port="19530")
schema = Collection.load("pretraining_corpus")
# Insert payload containing crawled lineage metadata
schema.insert([
{"name": "vector", "type": "float32", "values": [0.1425, -0.0982, ...]},
{"name": "source_url", "type": "string", "values": ["github.com/trending"]},
{"name": "lineage_sha", "type": "string", "values": ["sha_89b2c01"]}
])