Senior AI Engineer (Evaluations - Canvas Agent)

At Instructure, we believe in the power of people to grow and succeed throughout their lives. Our goal is to amplify that power by creating intuitive products that simplify learning and personal development, facilitate meaningful relationships, and inspire people to go further in their education and careers. We do this by giving smart, creative, passionate people opportunities to create awesome. And that's where you come in: We are looking for an AI Engineer to join the team behind the Canvas Agent. As we build out agentic workflows that can "act" upon Canvas and assist educators and students, ensuring the quality, safety, and accuracy of these interactions is our top priority. In this role, you will design and implement robust LLM-as-a-judge evaluation pipelines that automatically assess the Canvas Agent’s multi-step reasoning, tool usage, and conversational helpfulness. What You’ll Do - Design the Evaluation Framework: Build and maintain scalable LLM-as-a-judge pipelines to automatically score the Canvas Agent’s actions, responses, and tool usage across a variety of complex educational workflows. - Develop Rubrics & Datasets: Create comprehensive grading rubrics and curate high-q... Click Apply to read the full job description.