AI Quality Assurance

We are looking for a full-time AI Quality Assurance Expert to ensure the quality, reliability, and robustness of AI-enabled solutions. This role combines traditional software QA with AI-specific validation, supporting the development of machine learning models, LLM-based applications, agentic systems, and other intelligent products.

Work Model: Remote (must be based in Portugal)

Responsibilities:

• Define and execute QA strategies for AI-powered applications.
• Design quality gates, evaluation datasets, benchmark scenarios, prompt test suites, regression packs, and automated tests.
• Validate AI models, LLMs, and agentic systems.
• Assess and mitigate risks such as hallucinations, model drift, bias, and prompt injection.
• Review Python-based validation and test automation code.
• Establish monitoring, observability, and quality metrics for AI systems.
• Manage defect tracking, test documentation, and audit-ready evidence.
• Translate technical findings into business impact assessments and recommendations.
• Collaborate with engineering, data science, product, compliance, cybersecurity, and business teams.

Requirements:

• 5–10 years of experience in Software QA, Test Automation, or Digital Product Validation.
• Practical experience with AI, Machine Learning, LLM-based applications, or data-driven systems.
• Strong Python skills, with the ability to review and assess test and validation code.
• Good understanding of test automation frameworks, API testing, MLOps, observability, monitoring, and defect management.
• Experience with AI validation, including model evaluation, LLM testing, agentic systems, robustness testing, simulation-based validation, and quality monitoring of non-deterministic systems.
• Knowledge of AI-specific risks such as hallucinations, drift, bias, and prompt injection.
• Strong analytical skills, attention to detail, and excellent communication abilities.
• Comfortable working in agile, international, and cross-functional environments.
• Strong documentation and evidence management discipline.