Position: LLM – AI Quality Analyst (Personalization) – Korean

Type: Short-Term Contract

Location: Remote

Commitment: 30–40 hours/week, 4-hour overlap with PST

Engagement Length: 2 months

Start Date: Immediate

Role Responsibilities

Design multi-turn conversational prompts (1–5 turns) based on personal context
Evaluate personalized AI responses for relevance, grounding, and helpfulness
Assess whether personal data is used appropriately without incorrect inferences or hallucinations
Review integration quality to ensure personalization feels natural and not over-narrated
Perform side-by-side (SxS) evaluation and ranking of model responses
Identify grounding issues, forced personalization, and reasoning gaps
Write clear, concise, and structured rationales referencing specific conversation turns
Extract and verify model “Debug Info” to confirm correct data source usage
Maintain strict data hygiene by deleting evaluation conversations

Requirements

Korean proficiency (reading and writing)
Strong experience in data annotation, AI quality evaluation, content moderation, or a related role
Strong analytical thinking and ability to evaluate nuanced, ambiguous responses
Understanding of personalization concepts and AI response evaluation
Creative prompt engineering experience preferred
High attention to detail for SxS comparisons and subtle language differences
Excellent written communication and structured feedback skills
BS/BA degree or equivalent experience in a relevant field
Willingness to use a primary personal Google account with enabled personal data sources
Self-motivated and able to work independently in a remote environment
Desktop/laptop with reliable internet connection

Application Process

LLM - AI Quality Analyst (Korean) | $11/hr - Remote

About this role