[Proto] MTS - Software Engineer, Data Acqusition & Crawling

THE ROLE

We’re hiring a Pre-Training Data & Acquisition Engineer to build the data systems powering Prometheus’s foundation models for the physical world. You’ll work closely with research and infrastructure teams to acquire, process, and deliver large-scale training datasets across engineering, scientific, and multimodal domains. This role spans distributed crawling, source integration, data processing, and production operation, with end-to-end ownership from raw content to training-ready datasets.

WHAT YOU’LL DO

• Identify and integrate valuable data sources across engineering and scientific domains.
• Build distributed crawlers, API integrations, and bulk ingestion systems with effective scheduling, rate limiting, retries, and incremental updates.
• Develop pipelines for parsing, extraction, normalization, deduplication, quality filtering, and tokenization across heterogeneous formats.
• Optimize throughput and cost across networking, compute, storage, and databases as acquisition and processing workloads scale.
• Build monitoring and tooling to track source coverage, ingestion failures, processing throughput, and usable data yield.
• Work closely with pre-training researchers to translate data requirements into reliable pipelines and deliver datasets ready for large-scale training.
• Own dataset reproducibility, versioning, provenance, and recovery from acquisition through delivery.

WHAT WE’RE LOOKING FOR

• Experience building and operating large-scale distributed systems, web crawlers, or data processing pipelines.
• Strong programming ability in Python and Rust, Go, C++, or a comparable systems language.
• A practical understanding of web infrastructure, including HTTP, DNS, concurrency, caching, and common failure modes.
• Hands-on experience with databases, object storage, and distributed batch or streaming processing.
• Experience designing fault-tolerant systems that handle partial failures, resume interrupted work, and prevent unintended duplication or data loss.
• Ability to profile and debug performance across CPU, memory, disk, and network usage.
• Strong technical judgment when integrating unfamiliar sources, APIs, and file formats.
• Bias toward fast iteration and end-to-end ownership, from initial implementation through reliable production operation.
• Experience with search indexing, document extraction, or foundation-model data pipelines is a plus.

WHY JOIN US

• Work with world-class researchers on frontier AI systems for the physical world.
• Build the acquisition and processing systems that supply engineering, scientific, and multimodal data to large-scale model training.
• Competitive compensation and flexible work arrangements.
• High-impact, mission-driven environment.