ML Engineer, LLM's

San Francisco • FullTime

Posted 11d ago

Job Location

San Francisco

Tech Stack

AWS Python Fine-Tuning PyTorch NLP Reinforcement Learning Distributed Training Model Serving LLMs LLM evaluation Transformer architectures SFT preference optimization distillation

Remote Work Policy

On-site

Employment Type

FullTime

About the job

Knowtex is seeking an ML Engineer / Researcher focused on Large Language Models to contribute to the next generation of their AI stack. This role involves developing and optimizing models for clinical documentation and structured clinical reasoning, aiming to enhance quality, cost, latency, and control. The position is research-heavy with a direct path to production, requiring the design of experiments, building datasets and evaluation systems, and fine-tuning and post-training models. The successful candidate will collaborate closely with engineering and clinical teams to deploy solutions at scale and will play a key role in shaping Knowtex's long-term LLM strategy.

Responsibilities

Develop and optimize models for high-quality clinical documentation generation.
Build models for downstream clinical tasks like medication extraction, orders, and coding.
Apply structured generation, tool use, and agentic approaches for reliable clinical outputs.
Evaluate open-weight and proprietary models to determine optimal fine-tuning or task-specific strategies.
Fine-tune and post-train open-weight LLMs on proprietary clinical datasets using various optimization techniques.
Research methods to reduce inference cost and latency while maintaining clinical quality.
Serve and optimize open-weight models in production at scale.
Build rigorous evaluation frameworks for clinical accuracy, hallucinations, completeness, and formatting.
Develop datasets, benchmarks, and evaluation infrastructure for measurable model improvements.
Design experiments to assess the impact of approaches on real-world clinical outcomes.
Translate research results into production systems, balancing quality with operational constraints.
Collaborate with clinicians, applied ML engineers, and platform engineers.

Requirements

4+ years of experience in machine learning research or ML engineering with deep expertise in large language models.
Hands-on experience fine-tuning or post-training open-weight LLMs (e.g., SFT, distillation, preference optimization, RL).
Strong Python and PyTorch skills.
Deep understanding of modern transformer architectures and LLM training techniques.
Strong experimental methodology and ability to independently design and execute research projects.
Experience with large-scale datasets and distributed training environments.
Strong understanding of LLM evaluation and benchmarking.
Ability to translate research results into production systems.
Bachelor's, Master's, or PhD in Computer Science, Machine Learning, or related technical field, or equivalent research experience.
Experience building LLM evaluation systems (preferred).
Experience serving and optimizing open-weight models at scale (preferred).
Experience with structured generation, tool use, or agentic systems (preferred).
Experience in healthcare AI or clinical NLP (preferred).
Familiarity with clinical documentation workflows and medical terminology (preferred).
Knowledge of coding systems like ICD-10, CPT, E&M, or SNOMED (preferred).
Publications at leading ML or NLP conferences (preferred).
Experience deploying ML systems in HIPAA-compliant or regulated environments (preferred).
Experience in fast-moving startups where researchers own projects from experimentation through production (preferred).

Benefits

Competitive salary
Meaningful equity compensation
Unlimited PTO
Premium health, dental, and vision coverage
401(k) plan

About Knowtex

View company profile