Senior Applied AI Evaluation Engineer - RAN Ticket Intelligence
Remote • Kfar Saba • Full Time
Posted 2mo ago
Job Location
Kfar Saba
Tech Stack
Remote Work Policy
Fully remote
Employment Type
Full Time
Categories
Applied AI Engineer
About the job
Parallel Wireless is seeking a Senior Applied AI Evaluation Engineer to enhance how their RAN R&D organization analyzes engineering tickets. This role will focus on improving Root-Cause Analysis, recommending ownership, and learning from resolved cases by building rigorous evaluation datasets, establishing baselines, and comparing internal and external tools. The primary goal is measurable, evidence-based improvement, assessing the accuracy, grounding, calibration, and usefulness of AI-generated conclusions for engineering teams. A secondary focus involves analyzing engineering workflows to identify bottlenecks and opportunities for process improvement.
Responsibilities
- Define use cases, acceptance criteria, evaluation metrics, and quality guardrails for RAN ticket intelligence.
- Build and maintain evaluation datasets using resolved tickets, RCA, logs, test evidence, code changes, reviews, and outcomes.
- Establish baselines for current tools and non-AI approaches before evaluating new LLM, RAG, search, or agent-based methods.
- Evaluate internal, commercial, local, and open-source solutions using secure and reproducible data handling.
- Measure retrieval quality, groundedness, diagnosis accuracy, citation support, routing recommendations, calibration, abstention, latency, cost, and human effort.
- Design held-out, time-based, edge, and adversarial test cases while preventing data leakage.
- Analyze failures and translate incorrect conclusions into prioritized improvements.
- Prototype improvements in search, metadata, context construction, prompting, reranking, classification, agent workflows, and model selection.
- Develop reusable evaluation pipelines, tools, services, APIs, dashboards, or documented workflows.
- Collaborate with AI, RAN, QA, System Integration, Release, Field, data, and engineering teams to review results and support decisions.
Requirements
- BSc or MSc in Computer Science, Data Science, Machine Learning, Statistics, Electrical Engineering, or related field, or equivalent practical experience.
- 5+ years of hands-on experience in applied machine learning, data science, search, natural-language processing, analytics engineering, or AI-enabled software systems.
- Recent experience evaluating LLM, RAG, search, or agent systems using representative datasets, task-specific metrics, human review, failure analysis, and regression testing.
- Strong Python and SQL skills, with experience building maintainable data pipelines, experiment workflows, services, or analytical tools.
- Practical experience with information retrieval, embeddings, hybrid search, reranking, classification, structured outputs, tool calling, or common LLM failure modes.
- Strong statistical judgment, including sampling, leakage prevention, uncertainty, calibration, precision and recall, temporal drift, and controlled comparison.
- Ability to work with semi-structured engineering data from issue-tracking systems, source control, CI, logs, and test systems.
- Clear communication skills to explain results, limitations, and tradeoffs to technical and business stakeholders.