Data Annotation Specialist, Generalist
Remote • Canada • Contract
Posted 6d ago
Remote Work Policy
Fully remote
Employment Type
Contract
Categories
Applied AI Engineer
About the job
Cohere is seeking Generalist professionals with broad backgrounds for a Data Annotation Specialist role. In this judgment-driven position, you will play a crucial role in improving Large Language Models' performance by labeling, ranking, auditing, and correcting model output. You will evaluate, stress-test, and improve models on English-language data across text, image, and structured formats like JSON, CSV/TSV, and Markdown. This is an opportunity for individuals with strong analytical skills to contribute to high-impact annotation projects and help build machines that understand the world.
Responsibilities
- Evaluate and rank model outputs based on accuracy, helpfulness, tone, and safety, providing justifications.
- Stress-test models by probing for failure modes and unsafe behavior, documenting reproducible cases.
- Create datasets by authoring prompts, responses, and exemplars, and editing outputs to standard.
- Build and apply rubrics and taxonomies for consistent, high-quality annotations.
- Annotate and correct multimodal data (text, image, structured formats), ensuring data integrity.
- Participate in calibration exercises and inter-annotator agreement checks to maintain consistency.
- Adapt to new and evolving task types, applying sound judgment to developing guidelines.
- Report on model performance trends and provide feedback to cross-functional partners.
Requirements
- 1+ years of experience in AI data annotation, LLM evaluation, content moderation, research, or a related analytical role.
- Experience applying detailed guidelines to complex and ambiguous content with strong contextual and sociocultural judgment.
- Comfort with ambiguity and a willingness to flag unclear edge cases.
- A keen eye for inconsistencies and model failure modes, with an instinct to probe models adversarially.
- Familiarity with how large language models behave (e.g., hallucination, sycophancy, instruction-following gaps) is a plus.
- Excellent command of written English and strong reading comprehension, with the ability to clearly justify evaluations.
- Fluency in another language is a bonus.
- Strong attention to detail and commitment to accuracy, maintaining consistency across high-volume tasks.
- Comfort working with annotation platforms and structured data formats (JSON, CSV/TSV, Markdown, XML, YAML).
- Strong execution in a remote environment, including good time management and ability to work independently.
Benefits
- Part-time independent contractor position
- CAD $30/hour contract rate
- BYOD (Bring Your Own Device, laptop)