Senior Software Engineer, AI Eval

Sentry(25 days ago)

HybridFull TimeSenior$240,000 - $280,000Engineering

About this role

As a Senior Software Engineer on Sentry’s AI/ML team, you will build evaluation infrastructure to assess the accuracy and reliability of AI systems, ensuring correct and predictable performance. This role includes designing datasets and benchmarks, creating automated test harnesses, and collaborating with AI engineers and product leaders to establish measurable criteria for AI behavior. With a focus on high-quality evaluation processes, you will oversee the lifecycle of major AI initiatives from experimentation to production monitoring.

View Original Listing

Required Skills

Evaluation Frameworks
Dataset Design
Benchmark Creation
Test Harnesses
Metrics Pipelines
AI Systems
Production Code
Data Infrastructure
Structured Datasets
Unstructured Datasets

+6 more

Qualifications

Bachelor’s degree in computer science, machine learning, or a related field
Minimum 5+ years of professional experience

About Sentry

sentry.io

Sentry is a powerful application performance monitoring and error tracking platform designed for developers and software teams. It provides tools that help users identify, resolve errors more effectively, and learn continuously from their applications. Sentry enables teams to gain better visibility into their codebase, enhance the user experience, and accelerate their development processes. By leveraging Sentry, organizations can ensure higher software quality and improved operational performance.