QA Automation Engineer – AI/ML (Mumbai)

Location: Bangalore / Mumbai Role Overview Seeking a QA Engineer with AI/ML testing experience to ensure the quality, reliability, and performance of AI-powered applications. The role involves validating AI models, LLM solutions, and data pipelines while identifying issues such as hallucinations, bias, model drift, and data quality concerns. Key Responsibilities
- Test AI/ML models, LLM applications, RAG systems, chatbots, and data pipelines.
- Develop automated validation, regression, and performance testing frameworks.
- Evaluate model outputs for accuracy, consistency, relevance, bias, and hallucinations.
- Monitor production models for drift and performance degradation.
- Validate data quality and conduct fairness, bias, and explainability assessments. Must-Have Skills
- Bachelor's/Master's degree in Computer Science, Engineering, or related field.
- 3 6 years of QA experience, including 1 2 years in AI/ML testing.
- Strong Python skills for automation and data analysis.
- Experience with RAGAS, DeepEval, Promptfoo, LangSmith, PyTest, Selenium, and Postman.
- Good understanding of the ML lifecycle, data quality validation, and pipeline testing.
- Robust analytical, communication, documentation, and stakeholder management skills. Good-to-Have Skills
- Prompt Engineering and LLM Red Teaming.
- MLOps tools: MLflow, SageMaker, Vertex AI.
- Vector Databases, Responsible AI, AI Safety, and Fairness Frameworks.
- A/B Testing, Shadow Deployments, CI/CD, and DevOps practices. Apply on Kit Job: kitjob.in/job/4n6mer