Lead AI Trainer / Prompt Evaluator

Social Discovery Group (SDG) is the 3rd largest social discovery company in the world, uniting 60+ brands with 500 million users. We solve the problems of loneliness, isolation, and disconnection by transforming virtual intimacy into the new normal. Our portfolio includes online communication platforms focusing on AI, game mechanics, and video streaming - Dating.com, DateMyAge, Cupid Media, Dil Mil, Kiseki, and others.

SDG invests in IT startups around the world. Our investments include Open AI, Patreon, Flo, Clubhouse, Woebot, Flure, Astry, Coursera, Academia.edu, and many others.

We bring together a team of like-minded people and IT professionals specializing in the creation and development of globally impactful social discovery products. Our international team of 1200 professionals and digital nomads works all over the world.

Our teams of digital nomads work remotely from Cyprus, Malta, the USA, Armenia, Georgia, Kazakhstan, Montenegro, Poland, Latvia, Serbia, Spain, Portugal, UAE, Israel, Turkey, Thailand, Indonesia, Japan, Hong Kong, Australia and many other locations.

In August 2024, we achieved Great Place to Work US Certification™! This achievement reflects our core belief that a truly exceptional workplace is built on trust, pride, and camaraderie—not just great perks.

We are looking for a meticulous Lead AI Trainer / Prompt Evaluator to oversee and enhance the quality of AI model dialogues with users.

Your main tasks will be:

• Review and assess the quality of datasets produced by the content team, ensuring accuracy, consistency, and adherence to labeling guidelines.
• Provide constructive feedback and help maintain a high quality bar across annotators.
• Personally perform high-quality annotation and dataset creation as a subject-matter expert.
• Validate conversational AI model outputs (e.g., SbS evaluations, error tagging, scenario stress-testing) to detect issues before production release.
• Work closely with the ML team to refine instructions, dataset specifications, and evaluation criteria for model training cycles.
• Identify recurring issues in annotations or model behavior and suggest improvements to workflows, QA processes, or validation pipelines.
• Contribute insights on prompts and data strategies to accelerate model quality improvements.

We expect from you:

• Experience in dataset annotation and content validation for conversational AI / chat models / NLP products.
• Hands-on experience with SbS (side-by-side) or other model evaluation frameworks (e.g., quality scoring, tagging, error categorization).
• Ability to analyze conversational responses and identify accuracy, tone, safety, and contextual relevance issues.
• Experience preparing or following annotation guidelines and QA checklists.
• Strong attention to detail and ability to consistently apply evaluation criteria.
• Ability to provide clear and constructive written feedback to other annotators.
• Familiarity with prompt engineering practices is a plus.
• Familiarity with LLM-based automated evaluators / AI-as-a-judge is a plus.
• English proficiency (B2+) in reading and writing.

What do we offer:

• REMOTE OPPORTUNITY to work full time;
• Vacation 28 calendar days per year;
• 7 wellness days per year (time off) that can be used to deal with household issues, to lie down and recover without taking sick leave;
• Bonuses up to $5000 for recommending successful applicants for positions in the company;
• 50% payment for professional training, international conferences and meetings;
• Corporate discount for English lessons;
• ​Health benefits. According to the paychecks, if you are not eligible for corporate medical insurance, the company will compensate you with up to $ 1,000 gross per year per employee. This can be spent on self-purchase of health insurance or on doctor’s fees for yourself and close relatives (spouse, children);
• ​Workplace organization. The company provides all employees with an equipped workplace and all the necessary equipment (table, armchair, wifi, etc.) in our offices or co-working locations. In the other locations, the company provides reimbursement of workplace costs up to $ 1000 gross once every 3 years, according to the paychecks. This money can be spent on the rent of the co-working room, on equipping the working place at home (desk, chair, Internet, etc.) during those 3 years;
• Internal gamified gratitude system: receive bonuses from colleagues and exchange them for our merchandise, team building activities, massage certificates, etc.

Sounds good? Join us now!