Data Engineer
$150k - $450k • Sunnyvale, CA • Full-time
Posted 1y ago
Job Location
Sunnyvale, CA
Tech Stack
Remote Work Policy
On-site
Employment Type
Full-time
Categories
AI Engineer
About the job
We are a dedicated research lab focused on building, understanding, using, and risk-managing foundation models. Our mission is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy. As a Data Engineer specializing in Natural Language Processing (NLP) and large-scale data processing, you will be instrumental in enabling researchers by quickly gathering, curating, and preparing high-quality datasets to support cutting-edge NLP research. You will achieve this through efficient and scalable engineering practices, including web crawling, LLM-generated content refinement, and robust data pipelines, primarily leveraging Python and related technologies.
Responsibilities
- Collect, curate, and preprocess datasets based on researcher specifications within tight timelines.
- Develop and maintain efficient web crawling solutions, APIs, and automated workflows for data collection.
- Refine and evaluate Large Language Model (LLM) outputs to generate structured datasets for training and benchmarking.
- Implement scalable data pipelines for efficient data processing, storage, retrieval, and distribution.
- Collaborate with researchers and engineers to ensure data quality and relevance.
- Document data collection methodologies, dataset characteristics, and pipeline architecture.
- Participate in technical reviews to uphold best practices and data quality standards.
- Represent the institution at industry and research forums, showcasing technical capabilities.
Requirements
- Bachelor's degree in Computer Science, Data Science, Engineering, or a related technical field.
- Extensive experience in data engineering, data processing, and automation using Python.
- Proficiency in designing and deploying web crawling solutions, automated data extraction, and processing pipelines.
- Strong understanding of data structures, algorithms, databases, SQL, and performance optimization.
- Experience with cloud infrastructure and distributed data processing frameworks (e.g., AWS, Spark, Kafka, Kubernetes).
- Excellent problem-solving abilities and attention to detail.
- Strong communication and collaboration skills.
- Proven track record of supporting NLP or AI research teams with data delivery (preferred).
- Experience with large language models, including evaluation, efficient inference, and prompt engineering (preferred).
- Experience with refining outputs from large-scale AI models, such as LLM-generated data (preferred).