Architect, ML Engineering
Icertis
Role Overview
Icertis is building AI-powered products and copilots that customers trust with high-impact decisions. We are looking for an AI Assurance Associate Architect to define, build, and own the quality, validation, benchmarking, and evaluation strategy for our AI systems—especially those built on Large Language Models (LLMs).
This role goes beyond traditional QA. You will operate at the intersection of AI engineering, evaluation science, and software quality, designing assurance frameworks that ensure our AI systems are reliable, measurable, explainable, and production-ready.
You will work closely with Applied AI Engineers, Product Managers, Data Scientists, Legal, and Platform teams, while remaining hands-on with scripts, automation, eval frameworks, and benchmarking pipelines.
Icertis is the global leader in AI-powered contract intelligence. The Icertis platform revolutionizes contract management, equipping customers with powerful insights and automation to grow revenue, control costs, mitigate risk, and ensure compliance - the pillars of business success. Today, more than one third of the Fortune 100 trust Icertis to realize the full intent of millions of commercial agreements in 90+ countries.
Who we are: Icertis is the only contract intelligence platform companies trust to keep them out in front, now and in the future. Our unwavering commitment to contract intelligence is grounded in our FORTE values—Fairness, Openness, Respect, Teamwork and Execution—which guide all our interactions with employees, customers, partners, and stakeholders. Because in our mission to be the contract intelligence platform of the world, we believe how we get there is as important as the destination.
Icertis, Inc. provides Equal Employment Opportunity to all employees and applicants for employment without regard to race, color, religion, gender identity or expression, sex, sexual orientation, national origin, age, disability, genetic information, marital status, amnesty, or status as a covered veteran in accordance with applicable federal, state and local laws. Icertis, Inc. complies with applicable state and local laws governing non-discrimination in employment in every location in which the company has facilities. If you are in need of accommodation or special assistance to navigate our website or to complete your application, please send an e-mail with your request to careers@icertis.com or get in touch with your recruiter.
Required Qualifications
Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field.
6–10 years of experience in software quality, AI engineering, validation, or related roles.
Strong understanding of AI/ML concepts, especially LLMs and AI copilots.
Proven experience with AI validation, benchmarking, or evaluation frameworks.
Hands-on experience writing scripts or code for testing, automation, or evaluation.
Strong foundation in software quality principles and test strategy.
Ability to work in fast-paced, agile product environments.
Excellent analytical, documentation, and communication skills.
Preferred Qualifications
Experience with LLM eval frameworks, prompt testing, or agent validation.
Familiarity with automation tools, CI/CD integration, and test orchestration.
Experience creating and managing sample or synthetic datasets for AI testing.
Exposure to AI assurance, responsible AI, or compliance-oriented validation.
Prior experience in architect-level or cross-team ownership roles.
Why This Role Matters
This role is central to trust in AI at Icertis. You will help shape how AI quality is defined, measured, and defended—ensuring our AI systems are not just powerful, but reliable, explainable, and enterprise-ready.
Key Responsibilities
AI Assurance & Strategy
Define and own the end-to-end AI assurance strategy for AI products and copilots, including validation, benchmarking, and release sign-off criteria.
Establish quality gates, metrics, and acceptance thresholds for LLM-based systems.
Design assurance approaches covering accuracy, robustness, safety, regression, and reliability.
AI Evaluation & Benchmarking
Design and execute AI evaluation (eval) frameworks for LLMs, prompts, agents, and copilots.
Define benchmarking methodologies using curated datasets, synthetic data, and real-world scenarios.
Analyze evaluation results to identify failure modes, drift, and improvement opportunities.
Partner with Product, Legal, and AI teams on benchmarking, compliance, and defensibility of AI behavior.
Hands-on Engineering & Automation
Build and maintain automation frameworks, scripts, and tools for AI validation and benchmarking.
Implement repeatable pipelines for regression testing of AI features and releases.
Contribute code to support evals, test harnesses, dataset management, and reporting/dashboarding etc.
Enable scalable validation across multiple SKUs and releases.
AI Test Design & Execution
Design test strategies for complex, high-risk AI features and workflows.
Perform hands-on validation for critical AI capabilities and customer-impacting scenarios.
Validate fixes and improvements to prevent regressions in production AI systems.
Collaboration & Enablement
Collaborate with Applied AI Engineers and Data Scientists during design, development, and release.
Support Product Managers in defining AI-specific acceptance criteria and release readiness.
Work with Legal and Compliance teams on benchmarking, auditability, and assurance evidence.
Mentor QA and validation engineers on AI-specific testing, evals, and best practices.
Documentation & Reporting
Maintain clear documentation of validation approaches, eval methods, benchmarks, and outcomes.
Prepare assurance summaries and release-readiness reports for stakeholders.
Continuously improve standards, tools, and processes based on learnings and industry trends.