Generalist - English

Job Description

<p><b>Generalist - English ($45/hour)</b></p><p><b>Location</b>: Geography restricted to US, UK, Canada </p><p><b>Type</b>: Full-time or Part-time Contract Work </p><p><b>Fluent Language Skills Required:</b> English</p><p><br></p><p><b>Role Responsibilities</b></p><ul><li>Evaluate LLM-generated responses on their ability to effectively answer user queries</li><li>Conduct fact-checking using trusted public sources and external tools</li><li>Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies</li><li>Assess reasoning quality, clarity, tone, and completeness of responses</li><li>Ensure model responses align with expected conversational behavior and system guidelines</li><li>Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines</li></ul><p><br></p><p><b>Good Candidature</b></p><ul><li><b>Bachelor’s degree holder</b> </li><li><b>Significant experience using large language models</b> (LLMs) and understand how and why people use them</li><li><b>Excellent writing skills</b> and can clearly articulate nuanced feedback</li><li><b>Strong attention to detail</b> and consistently notice subtle issues others may overlook</li><li><b>Adaptable</b> and comfortable moving across topics, domains, and customer requirements</li><li>Background or experience in domains requiring <b>structured analytical thinking</b> (e.g., research, policy, analytics, linguistics, engineering)</li><li><b>Excellent college-level mathematics skills</b></li></ul><p><br></p><p><b>Nice-to-Have Specialties</b></p><ul><li>Prior experience with <b>RLHF, model evaluation, or data annotation work</b></li><li>Experience writing or editing <b>high-quality written content</b></li><li>Experience comparing multiple outputs and making <b>fine-grained qualitative judgments</b></li><li><b>Familiarity with evaluation rubrics</b>, benchmarks, or quality scoring systems</li></ul>