RESEARCHYou Can't Buy Taste. We Tested It.

The six kinds of human-data vendor, and what each one actually does

Search for a data labeling company and you get a list that puts a billion-dollar defense contractor, a two-year-old robotics startup with one capture facility, and a self-serve annotation tool in the same ranking. They are not substitutes. They run on different workforces, sell different deliverables, and fail in different ways, and the shared category label hides all of it. This piece sorts the market into six operating models, describes what each one actually delivers, and names the conditions under which each is the right answer. The goal is to let you rule out four of the six before you take a single call.

Key takeaways

  • The market sorts cleanly by two questions: who does the work, and what arrives at the end.
  • Crowd platforms and expert marketplaces both sell access to people, but the first sells volume at low unit cost and the second sells credentialed judgment you still have to manage.
  • Managed annotation services deliver finished labels on data you already own. Collection specialists produce data that did not previously exist. These are the two most commonly confused categories and the most operationally different.
  • Tooling platforms sell software you run. Several have recently added managed labor, which blurs the line without changing who is accountable for the dataset.
  • The archetypes are converging at the edges, so verify what a vendor currently operates rather than what its category label implies.

Crowd platforms sell volume and manage quality statistically

The oldest model in the category. A large distributed pool of generalist contributors works through a task interface, with quality managed statistically through redundancy, gold-standard tasks, and agreement thresholds rather than through individual accountability.

Appen is the reference case, stating a network of more than one million vetted contributors across over 170 countries and more than 500 locales, with secure on-site facilities for higher-sensitivity work. Toloka and TELUS Digital operate comparable models. Uber AI Solutions is a newer entrant running a gig contributor network on Uber's existing dispatch and payments infrastructure, which is a genuine structural advantage for recruiting and paying people at volume.

Crowd platforms are the right answer when the task is well-specified, the judgment required is general rather than expert, and the volume is large enough that per-unit cost dominates. They are the wrong answer when a single wrong label is expensive, when the task requires credentials, or when you need to know why a specific annotation came out the way it did. The failure mode is quiet: agreement metrics stay healthy while the task is systematically misunderstood, and the problem surfaces in model evaluation rather than in the labeling pipeline.

Expert marketplaces sell access, not accountability

These vendors recruit, vet, and match credentialed people: physicians, attorneys, PhDs, engineers. The mechanism is screening rather than task management, and increasingly the screening itself is automated. Mercor, micro1, Handshake AI, Pareto AI, CleverX, and AfterQuery all operate versions of this model, mostly selling into frontier labs that need reasoning traces, preference data, and evaluation from people who know a domain.

The model has scaled fast. micro1 crossed 100 million dollars in annual recurring revenue in December 2025 and reported roughly 500 million in gross annual run-rate by August 2026.

The distinction that matters at purchase time is accountability. A marketplace sources and matches people; a managed service is answerable for the output. Many marketplaces now offer managed delivery on top, and the question to ask is which one you are buying, because the contract, the quality mechanism, and the person you call when the data is wrong all differ. Expert marketplaces are the right answer when you have the internal capacity to define the task and run quality yourself, and when what you lack is access to qualified people.

Managed annotation services are accountable for the output

A managed service takes data you already own and returns it structured, with an accountable delivery team, a defined quality process, and a service agreement. The workforce is typically employed rather than gig, which is the operational difference that produces consistency on long-running programs.

iMerit built this model around full-time in-house subject-matter experts in medical imaging, geospatial, and autonomous vehicle data, and was acquired by EXL Service in August 2026 for 170 million dollars upfront plus up to 140 million in earnouts. Sama runs an in-house workforce focused on computer vision and autonomous driving. Innodata has operated a directly employed global workforce since 1988, reporting more than 10,000 employees and 2025 revenue of 251.7 million dollars, up 48 percent year over year. Centific combines a crowd and expert marketplace with in-house delivery teams.

This is the right answer for sustained, high-volume annotation where consistency across months matters more than peak throughput, and particularly in regulated verticals where the same reviewers seeing the same edge cases repeatedly is the quality mechanism. It is the wrong answer when the data does not exist yet, which is the distinction the next category turns on.

Collection specialists produce data that did not exist before

These vendors produce data that did not previously exist. Someone performs a task while instrumented, a teleoperation rig records a manipulation sequence, a wearable captures egocentric video on a factory floor, a studio records synchronized audio, video, and sensor streams under a protocol designed for your model.

The segment has grown quickly around physical AI. Build AI open-sourced Egocentric-100K in December 2025, comprising 100,405 hours of footage from 14,228 workers and 10.8 billion frames. XDOF launched publicly in June 2026 with ABC-130k under Apache 2.0 and a companion arXiv paper with UC Berkeley faculty co-authors. MicroAGI exited stealth in July 2026 with a 55 million dollar seed round, reported as Germany's largest, capturing manual work inside client factories. Mecka AI raised a reported 60 million dollar Series A and hosts the Egoverse consortium. Human Archive joined Y Combinator's W26 batch and published the open ego500 dataset. Shaip, RoboStream, Cortex AI, and Adamo occupy adjacent positions.

HumanSignal Services sits in this category, launched in November 2025 through the acquisition of Erud AI, running on-site and in-studio collection for physical AI and robotics alongside biometric capture and expert annotation.

Collection is an operations problem rather than a software problem. The work is protocol design, recruiting for specific physical or professional characteristics, site logistics, consent and release management, instrumentation, and verification that what was captured matches what was specified. The failure modes are protocol errors that invalidate a whole capture session and consent gaps that surface during diligence. Evaluate these vendors on whether they have run the specific kind of capture you need, not on whether they describe it well.

Tooling platforms change who manages the people, not whether you need them

Software you operate, with your own team or contractors doing the labeling. Encord, Labelbox, SuperAnnotate, and our own Label Studio are the established names.

Several have added managed labor. Labelbox pairs its platform with the Alignerr expert network and acquired Upcraft in February 2026 to bring agent technology into how Alignerr recruits. Encord raised a 60 million dollar Series C in February 2026 and now operates a physical-AI data collection facility in San Leandro, California, running leader-follower teleoperation and egocentric capture, which places it in the collection category as well as this one.

Platforms are the right answer when you have labeling capacity and need workflow, quality instrumentation, and governance around it, and when the task is non-standard enough that a fixed template will not fit. They do not remove the need for people; they change who manages them.

Off-the-shelf licensors trade exclusivity for speed

Rather than producing data to your specification, these vendors license existing corpora. David AI licenses proprietary speech datasets across more than fifteen languages. Protege operates a compliant training-data exchange. Several collection specialists, Build AI among them, license previously captured data alongside bespoke work.

Licensing is faster and cheaper than commissioning, and it is the right answer when an existing corpus genuinely covers your distribution. It is the wrong answer when your model's advantage depends on data nobody else has, because a licensed corpus is by definition available to anyone willing to pay for it.

How the six compare

ModelWorkforceWhat arrivesStrongest when
Crowd platformDistributed generalistsLabeled data at volumeTask is simple, volume is high, unit cost dominates
Expert marketplaceVetted credentialed specialistsAccess to people, sometimes managed deliveryYou can run quality yourself and lack access
Managed annotationEmployed in-house annotatorsFinished labels under an SLALong programs where consistency beats peak throughput
Collection specialistField collectors, studio operators, expertsA dataset built to your protocolThe data does not exist yet
Tooling platformYoursSoftware and workflowYou have capacity and need governance
Off-the-shelf licensorAlready capturedAn existing corpusA corpus genuinely covers your distribution

Two caveats on reading this table. The archetypes are converging, so a vendor's category label describes its origin more reliably than its current operation, and you should verify what it runs today. And more than one model can be correct at once: a team frequently licenses a base corpus, commissions collection for the distribution gap, and runs its own annotation on a platform.

The useful question is therefore not which vendor is best, which is the question every ranking in this category answers and the one least connected to your decision. It is which of these six operations your project actually requires, and whether the vendor in front of you runs that operation today or acquired the words for it recently. That question you can answer from public artifacts: a capture facility, an open dataset, a published certification, a named case study with a number in it. Work out which model you need first, and the shortlist writes itself.

Working with HumanSignal

We operate as a collection specialist with a platform underneath, which means the quality workflows on a services engagement run in Label Studio Enterprise and the review decisions, agreement metrics, and reviewer oversight are visible to you rather than summarized for you. If you are deciding between commissioning collection and annotating what you have, we can work through the distribution gap with you before scoping anything.

What is the difference between an expert marketplace and a managed service

A marketplace sources and matches qualified people, and you typically define the task and run quality. A managed service is contractually accountable for the output and runs quality itself. Several marketplaces now sell managed delivery as a tier, so confirm which you are buying rather than inferring it from the vendor's category.

Can one vendor cover more than one of these models

Yes, and increasingly they do. Encord began as a tooling platform and now operates a collection facility. Labelbox pairs a platform with a managed expert network. Treat a vendor's self-description as a starting point and verify which of these operations it currently runs at the scale you need.

How do I know whether I need collection or annotation

Ask whether the examples your model needs exist anywhere. If you have the raw data and need structure on it, you need annotation. If the behavior, scenario, or expert judgment has never been recorded, no amount of annotation will produce it and you need collection.

Does open-sourcing datasets indicate a vendor is credible

It indicates a particular kind of credibility. Build AI, XDOF, and Human Archive publish open datasets that anyone can inspect for quality and distribution, which is a stronger signal than a capability claim. Many established vendors publish none, which is not disqualifying but does mean you will have to generate that evidence yourself through a pilot.

Related Content