Data Lineage & Monitoring.
Review detailed telemetry charts tracking active exit nodes, run ethical robots.txt compliance tests, construct formatted JSON schemas, and audit raw textual scrubbing patterns.
Structured Data Schemas
Validate crawled HTML pages using strict JSON layouts. Choose a schema target to preview the metadata models streamed directly into Hugging Face datasets:
Optimized for LLM base-model pretraining. Highlights clean prose structure, sitemaps lineage, and word count statistics.
{
"$schema": "https://api.reworkd.ai/schemas/pretraining-v1.json",
"document_id": "doc_948a201c",
"source_domain": "wikipedia.org/wiki/Deep_learning",
"scraped_at": "2026-07-04T04:39:00Z",
"word_count": 4210,
"markdown_content": "# Deep Learning\nDeep learning is part of a broader family of machine learning methods...",
"meta_lineage": {
"ip_address": "108.34.12.98",
"http_response": 200,
"jaccard_similarity_coefficient": 0.12,
"pii_filtered": true
}
}Cluster Load telemetry
Real-time status overview of active scraping worker thread allocations, server node exit subnets, and average network response latencies.
Robots.txt Checker
Prior to executing a dataset gather, Reworkd checks sitemaps for crawl barriers. Define robots.txt rules and check if a target directory complies:
PII Anonymization Filter
NLP models require pre-training corpora free of personal coordinates. Type raw text inside our sandbox to test the regex and entity masking engine: