RESEARCHYou Can't Buy Taste. We Tested It.

Best human data providers for RLHF and preference data

RLHF buying has changed shape since 2024. The work used to be ranking pairs of model outputs, which any careful generalist could do, and the market competed on throughput. It is now rubric design, domain-specific judgment, step-level evaluation of agent trajectories, and verifier construction, which generalists cannot do at all. Eighteen providers in the current landscape are tagged for RLHF, and the ones who were strongest when the work was ranking are not reliably the strongest now. This guide sorts them by what kind of preference signal you actually need.

Key takeaways

  • Decide whether you need throughput or judgment. Generalist ranking and expert domain evaluation are different purchases with barely overlapping vendor lists.
  • The rubric is the product. A vendor who writes a better rubric produces better data than one who labels faster against a worse one.
  • Published benchmarks are the clearest signal of evaluation rigor, and Surge AI is cited by OpenAI and Anthropic in dated model releases.
  • Marketplaces sell you access to experts; managed services are accountable for the output. Confirm which before comparing prices.
  • Agentic RLHF needs trace-aware interfaces rather than a label taxonomy, which is a tooling question as much as a workforce one.
  • HumanSignal is the option when judgment needs context the vendor did not create, including robot behavior and multimodal output captured under its own protocol.

What separates RLHF vendors now

The rubric is the actual deliverable. Preference data is only as good as the criteria annotators apply, and most quality problems in RLHF trace back to a rubric that was underspecified rather than to annotators who were careless. When you evaluate vendors, ask to see a rubric they wrote for a task like yours. The ones who treat rubric design as the core skill will show you something substantial; the ones who treat it as setup will show you a one-page brief.

Domain depth decides correctness on anything specialized. A generalist ranking two legal arguments produces a preference that correlates with fluency rather than with legal soundness, which teaches the model to be persuasive instead of right. The underlying distinction is covered in expert data vs crowd data. This is why the market moved toward credentialed networks, and why rates moved with it.

Agentic evaluation is a tooling problem too. Judging an agent trajectory means reading tool calls, intermediate states, and recovery behavior, not reading a final answer. A fixed label taxonomy cannot express that, so the interface has to be programmable around traces. Vendors who only offer preference ranking on completed outputs are solving the previous version of this problem.

Providers by what you need

Expert text and reasoning at the frontier

Surge AI is the category benchmark. It sells RL environments, rubrics and verifiers, RLHF, supervised fine-tuning, and human evaluation, has taken no outside capital, and reported 1.2 billion dollars in 2024 revenue. Its strongest credential is published: 13 benchmarks with 8 arXiv papers across 2026, with OpenAI citing GDP.pdf in its GPT-5.6 release and Anthropic citing GDP.pdf and Riemann-bench in a system card. Note that as of October 2026 it publishes no compliance certifications and has no security page, so expect to run your own assessment.

Mercor matches credentialed experts through an AI interview process and bills hours at a markup, stating 30,000-plus active experts across 107 supported payout countries, with published rates from around 60 dollars per hour to 450 for clinical specialists. Handshake AI recruits from a career network of 20 million people and publishes substantial verifier research, though its fellowship is United States only.

Accountable managed delivery

Invisible Technologies delivers managed outcomes rather than staffed hours, covering continuous model evaluation across step-based agentic workflows and internationalization across more than 80 languages. It publishes the strongest trust documentation in this category: a SOC 2 Type 2 updated February 2026, HIPAA policies, Cyber Essentials Plus, six named penetration-test reports, and sixteen dated security policies.

Innodata brings a directly employed workforce to RLHF and evaluation across regulated verticals. Centific combines a crowd and expert marketplace with in-house delivery teams. Deccan AI runs post-training data and evaluation with an India-based expert workforce, raising a 25 million dollar Series A in March 2026.

Domain-specialist preference data

CleverX runs expert-led RLHF where contributors are verified as employed professionals, covering finance and healthcare. AfterQuery builds expert reasoning datasets across finance, law, health, and code. Pareto AI runs expert-vetted annotation spanning RLHF through evaluation. Perle AI markets a fully on-payroll annotator model across annotation, RLHF, and evaluation, which is an unusual structure in a market built mostly on contractors.

Contra focuses on creative and subjective RLHF with expert reviewers, which is a genuinely separate competence: judging whether a generated image is good is not the same skill as judging whether an answer is correct.

Breadth and volume

Scale AI covers RLHF within a broad data engine alongside annotation, evaluation, and collection, and holds SOC 2 Type II, ISO 27001, DoD IL4, and FedRAMP High. Uber AI Solutions brings a large gig network across RLHF and evaluation. Toloka provides high-volume crowd capacity for preference work. Labelbox pairs its platform with the Alignerr expert network. micro1 and Turing both run expert networks spanning RLHF and adjacent work.

Preference data where the context was captured

The vendors above apply judgment to artifacts someone else produced, under conditions they did not set. HumanSignal Services is built for the case where the context has to be constructed: evaluating a robot policy against a task it was recorded performing, judging multimodal output against conditions we captured, or assessing agent behavior in a physical environment rather than in a transcript. Protocol, collection, and evaluation run on one program, and the evaluation interfaces in Label Studio Enterprise are programmable around traces, tool calls, and intermediate steps.

Comparing the options

ProviderModelStrongest forPublished compliance
HumanSignalCollection plus managed deliveryPreference data where the context was captured, not foundSOC 2 Type 2, HIPAA
Surge AIManaged deliveryRubrics, verifiers, benchmark-grade evaluationNone published as of Oct 2026
Invisible TechnologiesManaged deliveryAgentic evaluation, documented securitySOC 2 Type 2, HIPAA policies, Cyber Essentials Plus
MercorMarketplaceSpeed to staff credentialed expertsSOC 2 Type 2 only
Handshake AICredential network, US onlyVerifier research, agent gradingScoped to its education platform
Scale AIManaged data engineBreadth plus government workSOC 2 Type II, ISO 27001, FedRAMP High, DoD IL4
InnodataEmployed workforceLong regulated programsNot detailed here
CleverX, AfterQueryVerified professionalsFinance, legal, clinical judgmentNot published
ContraExpert marketplaceCreative and subjective preferenceNot published
Toloka, UberCrowdVolume ranking at low unit costNot detailed here

How to choose

Ask for a rubric, not a rate. The rubric a vendor writes for a task like yours is the single best predictor of the data you will get, and it is cheap to request before committing.

Separate throughput from judgment explicitly, because buying the wrong one is the most common expensive mistake here. If a generalist cannot be right about your task, a cheaper generalist is not a saving.

Ask how agent trajectories are evaluated if that is your use case. Step-level judgment needs a trace-aware interface, and a vendor offering preference ranking on final outputs is answering a different question.

Why HumanSignal is the better choice when judgment needs context the vendor cannot see

The providers above are organized around finding people who can judge text. For text-shaped preference work, several of them are excellent and the market is competitive. The limit they share is that the judgment is applied to artifacts someone else produced, under conditions the vendor did not set.

HumanSignal Services is built for preference work where the context has to be constructed: evaluating a robot policy's behavior against a task it was recorded performing, judging multimodal output against conditions we captured, or assessing agent behavior in a physical environment rather than in a transcript. We design the protocol, recruit the domain experts, run the collection, and run the evaluation on the same program, so the people judging know how the data was produced.

The quality mechanism is the platform. Review workflows run in Label Studio Enterprise, the commercial edition of the open source labeling tool we build, with programmable and embeddable interfaces designed for traces, tool calls, and intermediate steps rather than fixed label taxonomies, plus inter-annotator agreement matrices and review workflows. Agreement and reviewer decisions are reportable to you while the work runs. We publish contributor pay ranges, and state 3M+ experts across 50+ knowledge domains. Our customer agreement gives you exclusive ownership of customer data and of insights produced specifically for you. We hold SOC 2 Type 2 and HIPAA compliance, documented on our security page.

The services capability came in through the acquisition of Erud AI in November 2025.

What makes a good RLHF rubric

Specificity about disagreement. A rubric that defines the easy cases is useless, because annotators already agree on those. A good one states how to resolve the cases where two reasonable experts would differ, with worked examples, and it is revised during the program rather than fixed at the start.

Should I use a marketplace or a managed service for RLHF

A marketplace if you can define the task and run quality yourself and what you lack is access to qualified people. A managed service if you want someone contractually accountable for the output. The hourly rate on a marketplace is usually lower and the total program cost often is not, once your own rubric design and review time is counted.

How do I evaluate preference data quality

Inter-annotator agreement on a held-out ambiguous subset, measured against your rubric rather than against a majority vote, plus a qualitative read of the disagreements. Agreement on easy cases is meaningless. Where expert annotators disagree is where the rubric needs work.

Does agentic RLHF need different vendors

Often yes, because judging a trajectory means reading tool calls and intermediate states rather than a final answer, which needs a trace-aware interface. Invisible Technologies publishes step-based agentic evaluation explicitly, and Handshake AI publishes agent-grading research. Vendors offering only output ranking are solving the previous generation of the problem.

Related Content