Security Rules & Architecture

1. Encryption and Data Custody

All crawl databases, shingle tables, and index storage zones are encrypted at rest using AES-256 with automatically rotated KMS keys. External synchronizations to destinations like Hugging Face, Pinecone, or AWS S3 are forced exclusively over TLS 1.3 tunnels.

Metadata database indices do not log target website user session values or authentication secrets. API credentials generated for sandbox workspaces are hashed via Argon2 and managed using isolated vault containers.

2. Audit Trails & Penetration Testing

Our operations infrastructure runs weekly automated network vulnerability audits. In addition, white-box penetration testing is performed annually by Schellman Labs, covering:

• API gateway authentication and session tokens validation.
• Container sandbox boundaries on Kubernetes crawl nodes.
• Database privilege configurations and network subnets separation.

We maintain detailed, immutable audit logs tracking raw URL ingestion timestamps, PII scrubbing logs, and token delivery counts.

3. Crawler Infrastructure Integrity

To verify our exit proxies remain clean and comply with network operators, our workers constantly run DNS validations, ensuring exit IPs do not trigger spam listings.

We run all crawler requests inside isolated Linux namespace sandboxes that prevent cross-container state propagation. Headless Chromium executions are bound to low-privilege pods with restricted network policies.