Associate Lead - Testing (QA + MLOps)

While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.

If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!

Technical Skills

• Python (mandatory)
• Experience with ML libraries (Scikit-learn, TensorFlow, PyTorch)
• Experience with LLM frameworks (LangChain, etc.)
• API & automation testing frameworks
• Git-based workflows
• Leadership & Communication
• Strong client-facing communication
• Experience leading QA teams
• Ability to create strategy decks & solution proposals
• Strong stakeholder management

If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!

Define testing approaches for AI systems built on AWS services such as:

• Amazon SageMaker
• Amazon Bedrock
• AWS Lambda
• Amazon API Gateway
• Amazon Kinesis
• AWS Glue
• Amazon S3
• Amazon CloudWatch

Design validation frameworks covering:

• Model accuracy & performance validation
• Data drift & concept drift detection
• Hallucination detection for LLMs
• Prompt robustness testing
• RAG validation (retrieval accuracy + grounding)
• Bias & fairness validation
• Safety & toxicity testing

Validate the end-to-end ML lifecycle including:

• Data ingestion & feature pipelines
• Model training & hyperparameter tuning
• Model versioning & registry
• Deployment validation
• Canary & blue/green release validation

Work with AWS-native services such as:

• SageMaker Pipelines
• SageMaker Model Monitor
• SageMaker Feature Store
• Bedrock model evaluation workflows
• CloudWatch-based observability

Define quality engineering approaches for:

• LLM-based applications using Amazon Bedrock
• Prompt engineering validation
• Multi-agent orchestration testing
• Chatbot & Voice bot conversational testing
• Intent classification validation
• Conversation drift & fallback validation
• API contract validation for LLM integrations

Build reusable evaluation harnesses for:

• BLEU / ROUGE scoring
• Embedding similarity scoring
• Response consistency
• Safety scoring frameworks

Framework & Capability Development

• Design reusable AI testing accelerators
• Create AWS-aligned AI test automation frameworks (Python-first)
• Develop synthetic data generation strategies
• Establish AI quality scorecards
• Build an internal AI QA Center of Excellence

Client Engagement & Leadership

• Lead AI/ML quality strategy workshops
• Perform AI risk & readiness assessments
• Present quality architecture to CXOs
• Drive QA transformation programs
• Mentor QA teams on AWS-based AI testing
• Own delivery for AI testing engagements end-to-end

Testing Expertise

• 8–12+ years in Quality Engineering
• Strong test strategy, automation & governance experience
• Experience leading QA transformation initiatives
• Experience building frameworks from scratch AI/ML & GenAI Expertise
• Deep understanding of ML lifecycle
• Experience testing ML models (NLP preferred)
• Hands-on experience validating LLM applications
• Strong understanding of:
• Prompt engineering
• RAG architecture
• Embeddings
• Bias & explainability AWS AI/ML Expertise
• Hands-on experience with:
• Amazon SageMaker (training, deployment, monitoring)
• Amazon Bedrock (LLM integration & evaluation)
• S3-based data pipelines
• AWS IAM (security validation)
• CloudWatch monitoring
• Lambda & API Gateway integrations
• AWS CI/CD (CodePipeline / CodeBuild preferred)

Understanding of:

• Infrastructure as Code (Terraform / CloudFormation)
• Observability in AI systems
• Cost monitoring for ML workloads