Web crawling Jobs

Prime Intellect is building the open superintelligence stack, providing infrastructure for AI labs. Their platform, Lab, unifies compute, environments, evaluations, and training for post-training at frontier scale. They are developing open-source models for long-horizon tasks and the platform their research team uses. The company has raised $150M from prominent investors and AI leaders. They are seeking individuals passionate about building at the intersection of frontier research, infrastructure, and go-to-market strategy for a nascent category. This role focuses on building the data foundation for Prime Intellect's next generation of open models. You will identify and process valuable data sources, transforming them into training-ready corpora. Your work will directly influence both broad pretraining efforts and the development of specific capabilities like reasoning, coding, and long-context learning. You will collaborate closely with researchers to test and refine data strategies, ensuring the data effectively improves model performance.

$150k - $350k

We are a dedicated research lab focused on building, understanding, using, and risk-managing foundation models. Our mission is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy. As a Data Engineer specializing in Natural Language Processing (NLP) and large-scale data processing, you will be instrumental in enabling researchers by quickly gathering, curating, and preparing high-quality datasets to support cutting-edge NLP research. You will achieve this through efficient and scalable engineering practices, including web crawling, LLM-generated content refinement, and robust data pipelines, primarily leveraging Python and related technologies.

$150k - $450k