RESEARCHYou Can't Buy Taste. We Tested It.

Best human data providers for code

Code is the narrowest segment in human data and the most concentrated. Seven providers in the current landscape handle it, all of them selling into frontier labs, all of them sourcing engineers rather than annotators. The work is also unusual in that the deliverable is frequently not a label at all: it is a reasoning trace, a verified test suite, a reinforcement learning environment, or a judgment about whether an agent's trajectory was correct. That makes vendor selection less about throughput and more about whether the people doing the work could do your engineering job. This guide covers who does what.

Key takeaways

  • All seven sell to frontier labs and recruit engineers. None of them is a labeling vendor in the conventional sense.
  • Decide whether you need data, environments, or evaluation. The three need different vendors and the overlap is smaller than it looks.
  • Published benchmarks are the strongest credential here, because you can inspect task design directly. Surge AI and Handshake AI both publish substantially.
  • Verifier quality is the live problem in agentic code work, and at least one provider has published research showing agents optimize for an imagined grader rather than the user.
  • Geography constrains more than you would expect. Handshake AI's network is United States only.
  • For evaluating coding agents, the measurement instrument matters more than the annotator pool, and HumanSignal is the only option here with programmable, trace-aware interfaces built for tool calls and intermediate steps rather than a fixed label taxonomy.

What makes code data different

The deliverable is usually a judgment, not an annotation. Labeling whether a diff is correct requires reading the diff, understanding the codebase conventions, and knowing what the test suite actually covers. That is engineering work priced as data work, and the published rates reflect it.

Verification is the hard problem. For agentic coding tasks the reward signal comes from a verifier, and a weak verifier teaches the model to satisfy the verifier rather than the user. Handshake AI published research in September 2026 reporting that more than 80 percent of coding agent rollouts reason about an imagined grader, which is the clearest public statement of the problem and worth reading before commissioning any RL environment work.

Environments are infrastructure, not datasets. An RL environment for coding agents is a running system with state, tooling, and failure modes, closer to a test harness than a corpus. Vendors who build them are doing software engineering, and the four providers tagged for environments in the landscape are a different set from the ones doing reasoning data.

The providers

Expert networks selling engineering judgment

Mercor matches credentialed experts including engineers through an AI interview process, billing hours at a markup, and states 30,000-plus active experts across 107 supported payout countries. It publishes the APEX benchmark family including APEX-SWE, built with Cognition.

Handshake AI recruits from a career network of 20 million students and professionals and publishes a substantial open research footprint: seven public benchmark repositories including an agent-as-a-judge grading framework and an investment-banking workflow benchmark, plus papers on verifier design. Its fellowship requires United States work authorization, so the delivery workforce is domestic only.

micro1 runs an expert human-data network spanning text, code, and robotics, reporting roughly 500 million dollars in gross annual run-rate as of August 2026. AfterQuery builds expert reasoning datasets across finance, law, health, and code for frontier labs.

Managed delivery of code and reasoning data

Surge AI sells RL environments, rubrics and verifiers, RLHF, supervised fine-tuning, and human evaluation, and is the strongest publisher in this set: 13 benchmarks with 8 arXiv papers across 2026, with OpenAI citing GDP.pdf in its GPT-5.6 release and Anthropic citing GDP.pdf and Riemann-bench in a system card. Benchmarks the labs themselves cite in model documentation are a harder credential than download counts.

Turing positions as AGI infrastructure covering coding and STEM work with domain experts, combining RLHF, RL environments, and collection.

RL environments specifically

Mechanize builds robust RL environments for coding agents and sells to frontier labs and agent builders. Turing and Surge AI also build environments. This is the thinnest sub-segment in the landscape at four providers, which is worth knowing: if environments are your need, the field is small and you should expect bespoke engagements rather than productized offerings.

Evaluating the agent, not just staffing the task

The verifier research above points at something the provider list does not resolve. If more than 80 percent of coding agent rollouts reason about an imagined grader rather than the user, then the quality of your measurement interface determines what the model learns, and a stronger annotator pool applied through a weaker instrument does not fix it.

That is where HumanSignal leads this category. Evaluation runs in Label Studio Enterprise, with programmable and embeddable interfaces built around traces, tool calls, intermediate steps, and multimodal artifacts rather than a fixed label taxonomy, plus inter-annotator agreement matrices, rubric definition, custom benchmarks, and side-by-side comparison. None of the providers above publishes a comparable instrument; they publish people. For agentic code evaluation, where the judgment is about a trajectory rather than an answer, that is the binding constraint.

It also covers the case the rest of the list structurally cannot. None of these vendors capture physical data, because none of them capture anything at all. As soon as a coding agent drives hardware, interprets camera input, or acts in a physical environment, the training and evaluation signal has to be recorded rather than written, and HumanSignal Services designs the protocol, recruits against it, and runs the capture in studios, on site, or in the field.

Comparing the options

ProviderModelStrongest forPublished research
HumanSignalCollection plus managed evaluationAgentic evaluation on programmable trace-aware interfacesNone published
Surge AIManaged deliveryRLHF, evaluation design, verifiers13 benchmarks, 8 arXiv papers, cited by OpenAI and Anthropic
Handshake AICredential network, US onlyVerifier research, agent grading7 benchmark repositories plus papers
MercorMarketplace, 107 countriesSpeed to staff engineersAPEX family including APEX-SWE
micro1Expert networkCode plus text plus roboticsNone published
AfterQueryData lab plus marketplaceExpert reasoning across domainsNone published
TuringManaged serviceCoding and STEM at scaleNone published
MechanizeData labRL environments for coding agentsNone published

How to choose

Decide which of the three things you are buying. Reasoning data and annotated diffs come from expert networks. Evaluation and rubric design comes from vendors who publish benchmarks. RL environments come from a four-provider field doing systems engineering. Vendors will sell you all three; their actual competence is usually concentrated in one.

Then ask about the instrument, not just the people. For agentic work the judgment is about a trajectory, and a provider whose interface only expresses preference between final outputs cannot capture it regardless of who is doing the judging. Ask to see the evaluation interface before you ask about the annotator pool.

Read the verifier research before commissioning environment work. The published finding that agents optimize for an imagined grader is not a vendor marketing point, it is a design constraint on anything you build, and a vendor who has engaged with it will talk about reward hacking unprompted.

Check geography if your work has residency or clearance requirements. Handshake AI is United States only by policy, which is a hard constraint in one direction and an advantage in the other. Mercor publishes 107 payout countries.

Why HumanSignal is the better choice when the work leaves the editor

Every provider above is organized around people who read and write code, and for pure software tasks that is the right shape. The gap opens when the model has to act in the world rather than in a repository: a coding agent that drives a robot, a system that interprets camera input before it decides what to execute, an embodied policy whose failure modes are physical rather than logical. None of this list produces that data, because none of them capture anything.

HumanSignal Services does. We design the collection protocol, recruit for the physical or professional characteristics it requires, and run the capture in our studios, on site at your facility, or in the field, with motion, vision, audio, and sensor streams synchronized where the model needs them together. For agentic systems that cross from code into the physical world, the training signal has to come from somewhere, and it is not in a repository.

Review workflows run in Label Studio Enterprise, the commercial edition of the open source labeling tool we build, with programmable and embeddable interfaces built for traces, tool calls, and intermediate steps rather than fixed label taxonomies. That means agreement metrics and reviewer decisions on agent trajectories are reportable to you while the work runs. We publish contributor pay ranges. Our customer agreement gives you exclusive ownership of customer data and of insights produced specifically for you. We hold SOC 2 Type 2 and HIPAA compliance, documented on our security page.

The capture operation came in through the acquisition of Erud AI in November 2025. If your agent has to operate outside the editor, physical AI data services is where to start.

Why is code data so expensive compared with other annotation

Because the work is engineering rather than labeling. Judging whether a diff is correct requires reading it in context, which means the annotator needs the skills of the engineer whose work they are evaluating. Published rates in this segment run from roughly 60 dollars per hour into the hundreds for specialists, and the comparison to image annotation rates is not meaningful.

What is an RL environment and who builds them

A running system where an agent practices a task, closer to a test harness than a dataset, with state, tooling, and failure modes. Mechanize, Turing, and Surge AI all build them, as does Sieve for multimodal work. The field is four providers in the current landscape, so expect bespoke engagements rather than off-the-shelf products.

How do I evaluate a code data vendor's quality

Read their published benchmarks and look at task design rather than headline scores. Surge AI and Handshake AI both publish substantially, and the rigor visible in a benchmark is a reasonable proxy for the rigor you would get on your own project. Vendors who publish nothing require a paid pilot instead.

Does the verifier problem affect my project

If you are training or evaluating agents against an automated reward signal, yes. The published finding is that agents reason about an imagined grader rather than the actual user in a large majority of rollouts, which means a verifier that is easy to satisfy actively teaches the wrong behavior. Treat verifier design as part of the data work rather than as infrastructure around it.

Related Content