RESEARCHYou Can't Buy Taste. We Tested It.

Best human data providers for annotation and labeling

Annotation is the most mature function in human data, with 25 of 46 providers offering it, and the one where vendor marketing is least informative. Everyone claims high accuracy, everyone offers model-assisted pre-labeling, and the published rates cluster tightly enough that price tells you very little. What actually differs is the workforce: who does the labeling, under what employment terms, with what domain knowledge, and whether the vendor can tell you any of that. This guide sorts the field by workforce model, because that is the variable that predicts quality on the cases that matter.

Key takeaways

  • Workforce model is the most useful discriminator. Crowd, employed in-house, vetted expert, and platform-plus-managed produce different results on ambiguous work.
  • Disclosure correlates with diagnosability. A vendor that cannot tell you where its annotators are usually cannot tell you why a label is wrong.
  • Employed workforces outperform flexible pools on long programs where the same edge cases recur.
  • Model-assisted labeling is table stakes. Ask how corrections are reviewed, not whether the tool pre-labels.
  • Accuracy claims without a stated task, dataset, and measurement protocol are not evidence.
  • HumanSignal publishes contributor pay ranges and runs annotation on data it also captured, so the guideline is written against the capture specification instead of reverse-engineered.

What actually differs between annotation vendors

Employment terms shape consistency. A contributor who works on your taxonomy for eight months accumulates context that a contributor who sees it once cannot. On short bursts of simple work this is irrelevant and crowd economics win. On long programs where the taxonomy evolves and the edge cases recur, continuity is the quality mechanism, which is why the employed-workforce vendors compete on exactly that.

Disclosure predicts root-cause capability. Some vendors publish network size, geographies, employment terms, and pay. Others publish none of it. The point is not ethics: a vendor that knows where its annotators are, what they are paid, and how they qualified can usually trace a quality problem to its source. One that cannot will absorb the problem into an aggregate and tell you the accuracy is fine.

Review architecture matters more than tooling. Nearly everyone offers pre-labeling now. The question is whether reviewers are measured against the model's suggestion or independently, because accepting a subtly wrong pre-label is both easier and harder to detect than mislabeling a blank frame.

Providers by workforce model

Annotation on data the same team captured

HumanSignal Services runs annotation on programs where it also designed the protocol and ran the collection, which means the guideline is written against the capture specification instead of reverse-engineered from found data, and the annotators know what the protocol was trying to produce. It publishes contributor pay ranges, which makes its sourcing claims checkable in a market where most vendors publish nothing, and states 3M+ experts across 50+ knowledge domains. Quality workflows run in Label Studio Enterprise with inter-annotator agreement matrices, intelligent validation, and label distribution reports, so agreement and rework are reportable while the work runs.

Employed in-house annotators

iMerit states that its entire workforce across India, Bhutan, and the United States consists of full-time staffers with healthcare and benefits, explicitly not a crowd or gig pool, reporting 10,000-plus active resources across 60-plus countries and a 25,000-plus network of domain experts. It publishes the broadest compliance set here: SOC 2 Type 2, ISO 27001, ISO 9001, HIPAA, GDPR, and TISAX. EXL Service completed its acquisition in August 2026.

Sama runs an employee-based delivery centre model and names the locations publicly: Nairobi, Gulu, Kampala, San Jose in Costa Rica, Montreal, and The Hague. It publishes seven named case studies with quantified outcomes and claims a 99 percent first-batch acceptance rate with eight-year average customer retention.

Innodata has run a directly employed global workforce since 1988 across healthcare, legal, finance, and public sector work, reporting more than 10,000 employees.

Crowd platforms

Appen states more than one million vetted contributors across over 170 countries and 500-plus locales, with secure facilities for higher-sensitivity work. TELUS Digital, Toloka, Uber AI Solutions, and SuperAnnotate all operate crowd or crowd-backed labeling at volume. Centific combines a crowd and expert marketplace with in-house delivery teams.

Crowd models are the economically correct choice for well-specified, high-volume work. They fail quietly on tasks needing expertise, because annotators sharing a misunderstanding produce high agreement on the wrong answer, and agreement metrics will not reveal it. The tradeoff is covered in expert data vs crowd data.

Vetted expert annotation

Pareto AI runs expert-vetted labeling from RLHF through evaluation. Perle AI markets a fully on-payroll annotator model across annotation, RLHF, and evaluation. Centaur Labs sources medical and diagnostic annotation from an expert crowd of licensed professionals. Handshake AI and Surge AI both offer annotation alongside their RLHF and evaluation work, sourcing credentialed people.

Platform plus managed labeling

Encord provides native annotation for image, video, LiDAR, audio, and text with SAM2-assisted labeling and embedding-based curation, holding 4.8 out of 5 on G2 across 65 reviews as of September 2026, the deepest third-party review base in this category. Labelbox pairs its platform with the Alignerr expert network. SuperAnnotate offers a computer vision and NLP platform with managed labeling.

These are platform-lane vendors. They appear here because they are real annotation options, not as services-side comparisons.

Breadth in a single vendor

Scale AI annotates text, image, video, 3D, and LiDAR within one Data Engine and holds SOC 2 Type II, ISO 27001, DoD IL4, and FedRAMP High, the only federal authorizations in this set. It publishes no contributor headcount, geography, or pay rates.

Shaip, LXT, Sigma AI, and Aya Data all combine annotation with collection across multilingual and regional work.

Comparing the options

ProviderWorkforcePublishes workforce detailNotable
HumanSignalExperts, field, in-houseYes, publishes pay rangesAnnotation on data we also captured
iMeritFull-time employees with benefitsYes, explicitlyBroadest compliance set here
SamaEmployees in named delivery centresYes, names locationsSeven quantified case studies
InnodataDirectly employed since 1988PartiallyRegulated verticals
AppenCrowd plus secure facilitiesYes, network and locales500-plus locales
TELUS, Toloka, UberCrowdPartiallyVolume at low unit cost
Centaur LabsLicensed professional crowdPartiallyMedical and diagnostic only
Pareto AI, Perle AIVetted expertsPartiallyExpert-vetted across functions
EncordPlatform plus HITLNo annotator detail4.8 out of 5 from 65 reviews
Scale AINot publishedNoFedRAMP High and DoD IL4

How to choose

Ask who does the work and under what terms. iMerit, Sama, and Appen will answer with employment terms, named locations, or network figures. Several larger vendors will answer with less. The answer itself is diagnostic.

Ask for the annotation guideline they would write for your hardest cases, not your easy ones. The document that resolves genuine ambiguity predicts dataset quality better than any accuracy figure.

Match employment model to program length. Short bursts of simple work favor crowd economics; long programs with evolving taxonomies favor employed continuity, and paying crowd rates for work that needs continuity is a false saving.

Why HumanSignal is the better choice when the labels depend on how the data was made

The vendors above are good at structuring data you hand them, and several are excellent. What none of them can do is tell an annotator why a recording looks the way it does, because none of them were there when it was captured.

HumanSignal Services runs annotation on programs where we also designed the protocol and ran the collection. The guideline is written against the capture specification rather than reverse-engineered from found data, and the annotators know what the protocol was trying to produce, which is where most quality problems in multimodal and sensor work originate.

The quality mechanism is inspectable rather than asserted. Review workflows run in Label Studio Enterprise, the commercial edition of the open source labeling tool we build, with inter-annotator agreement matrices, intelligent validation, label distribution reports, and review workflows, plus programmable and embeddable interfaces for tasks a fixed template cannot express. Agreement, reviewer decisions, and rework rates are reportable to you while the work runs rather than summarized at the end. We publish contributor pay ranges, which makes our sourcing claims checkable, and state 3M+ experts across 50+ knowledge domains. Our customer agreement gives you exclusive ownership of customer data and of insights produced specifically for you. We hold SOC 2 Type 2 and HIPAA compliance, documented on our security page.

The services capability came in through the acquisition of Erud AI in November 2025.

Does an employed workforce actually produce better labels

On long programs with recurring edge cases, generally yes, because the same reviewers accumulate context about the taxonomy and can be retrained rather than replaced when it changes. On short, simple, high-volume work the advantage disappears and crowd economics win. Match the model to the program rather than assuming one is better.

How should I interpret a vendor's accuracy claim

With the task, dataset, and measurement protocol attached, or not at all. "Above 98 percent accuracy" means nothing without knowing on what task, against whose ground truth, and measured how. Ask for the protocol; a vendor who has one will share it.

What is the most useful question to ask an annotation vendor

Where your annotators are, what they are paid, and how they qualified. Not because the answer itself decides anything, but because a vendor that can answer can usually diagnose a quality problem at its source, and one that cannot will absorb it into an average.

Is model-assisted labeling a quality risk

It can be, in one specific way: a pre-label that is subtly wrong is easier for a reviewer to accept than an empty frame is to mislabel. The mitigation is independent review measurement rather than measuring reviewers against the model's suggestion. Ask how the vendor handles that specifically.

Related Content