RESEARCHYou Can't Buy Taste. We Tested It.

What to ask a data collection vendor before you sign

Most evaluations of a human-data vendor run on the wrong inputs. You get a deck, a logo wall, a throughput number with no denominator, and a reference call with a customer the vendor chose. None of that predicts whether the dataset you receive in four months will train the model you are actually building. The questions that do predict it are mostly questions about disclosure: what the vendor publishes about its own operation, and what it declines to publish. This piece gives you seven of those questions, the reason each one discriminates, and what a good answer looks like.

Key takeaways

  • Separate vendors that create data from vendors that process data you already have. These are different businesses with different failure modes, and the category word "data labeling" hides the distinction.
  • Workforce disclosure is the single most discriminating question in the category. Several large vendors publish no contributor headcount, no delivery geography, and no pay rates.
  • Read the contract for data ownership and IP indemnification before you read the capabilities deck. No vendor in this category, ours included, publishes indemnification covering the provenance of a delivered dataset, so that one has to be drafted rather than found.
  • Match compliance to your actual risk rather than to the longest certification list. One vendor holds FedRAMP High and DoD IL4 but no HIPAA attestation while naming a hospital system as a customer.
  • Ownership changes are a live risk in this category, not a hypothetical one. Three of the largest vendors changed hands or control between June 2025 and August 2026.
  • Check the sample size on every review score. A 4.5 out of 5 from two reviews is not evidence.

Ask whether they create data or process yours

The category is usually described as "data labeling," which collapses two different businesses. One takes data you already own and adds structure to it: bounding boxes, transcripts, preference rankings, rubric scores. The other produces data that did not exist before you commissioned it: someone performs a task on camera, a sensor rig records a manipulation sequence, a licensed clinician writes a reasoning trace.

The distinction matters because the failure modes do not overlap. An annotation vendor fails on consistency, agreement, and rework. A collection vendor fails on protocol design, recruiting, site logistics, and consent. Asking a vendor which of these they are, and listening for whether the answer is specific, filters a surprising amount of the market in one question. Many vendors now market both; fewer have operated both at scale. Our own split is documented in HumanSignal Services, and the related distinction between expert and crowd sourcing is covered in expert data vs crowd data.

Ask who does the work, and whether they will tell you

This is the question with the highest information value, because the answer is frequently nothing.

As of September 2026, Scale AI published no current contributor headcount, no delivery-geography breakdown, and no pay rates on its public pages. That is a notable omission from the company that defined the category, particularly next to vendors that do publish: Appen states a network of over one million vetted contributors across more than 170 countries and 500 locales, and Sama publishes a cumulative workforce figure. Encord discloses less about its workforce than about its customers, marketing "300+ AI teams" while naming no annotator countries or cities.

We publish contributor pay ranges on our own recruiting pages, which is unusual in this set and is deliberate. Published rates are checkable; "fair pay" is not.

What you are testing is not whether a vendor is ethical. You are testing whether the people doing your work are identifiable to the vendor itself. A vendor that cannot tell you where its contributors are, what they are paid, and how they were qualified is a vendor that cannot tell you why a label is wrong when it is wrong. The workforce question is a proxy for root-cause capability.

Ask who owns the output, and get it in writing

Read the customer agreement, not the security page. The terms diverge more than the marketing.

Three things to locate. First, who owns the labeled output: it should be you, named explicitly, with the labeled output defined as customer data rather than as a derived work the vendor retains rights in. Second, whether the vendor indemnifies you against third-party IP claims arising from the data it delivers, which matters more every quarter as training-data provenance litigation expands. Third, what the vendor retains: usage and performance logs are normal, and anything beyond that needs an explanation.

The spread here is real, and it is worth being precise about where we sit in it. Our customer agreement states that the customer exclusively owns all right, title, and interest in customer data and in insights produced specifically for the customer, which covers labeled output. Our indemnification clause is narrower than that: we defend the customer against claims that use of the HumanSignal product infringes a United States patent, copyright, or trade secret. That is product indemnification, not a warranty about the provenance of a delivered dataset.

The wider finding from checking the public terms across this category in October 2026 is that nobody publishes the second thing. Scale AI and Encord publish no IP indemnification commitment at all. Surge AI's public terms, last updated in December 2021, run the indemnity from customer to vendor with nothing coming back, and take a perpetual license to customer labeled data. Mercor's public terms also indemnify only in the vendor's direction and cap liability at the greater of fees paid or 100 dollars. None of these are necessarily the terms a large customer negotiates, and none of them are published. So treat data-provenance indemnification as something you raise early and get drafted specifically, rather than something you expect to find on a website, including ours.

Match compliance to the risk you actually carry

Longer certification lists are not better lists. They are different lists, and the right one depends on what you are building.

Scale AI holds SOC 2 Type II, ISO 27001, DoD IL4 Provisional Authorization, and FedRAMP High, which is the strongest government posture in the category and effectively decides the question if you are delivering into a defense environment. The same research found no HIPAA attestation on its public compliance pages, which is worth noting given Mayo Clinic appears as a named customer. Encord claims SOC 2, HIPAA with Vanta verification, and GDPR compliance, but not ISO 27001. We hold SOC 2 Type 2 and HIPAA compliance, documented on our security page, and not FedRAMP. Appen holds SOC 2 Type II and ISO 27001, but the ISO certification is facility-scoped to named sites in the UK, the Philippines, and China rather than company-wide, and its HIPAA claim covers UK secure facilities only.

Scope is the part that gets skipped. A certification that covers one delivery site does not cover the site your project runs in, and a vendor that answers the scope question precisely is telling you something about its operational maturity beyond the certificate itself.

Ask what happens if the vendor changes hands

Ownership risk in this category stopped being hypothetical in 2025.

In June 2025, Meta invested roughly 14.3 billion dollars for a 49 percent stake in Scale AI at a valuation above 29 billion. Within weeks, Google, OpenAI, and xAI were reported to have pulled back or halted work, on the straightforward logic that routing unreleased-model data through a pipeline part-owned by a competitor is a risk a frontier lab does not need to take. By August 2025, reporting indicated that even Meta's own AI unit was favoring other vendors. In August 2026, EXL Service completed its acquisition of iMerit for 170 million dollars upfront plus up to 140 million in earnouts. TELUS Digital was taken private by TELUS Corporation in October 2025, and in March 2026 a threat actor publicly claimed theft of more than a petabyte of data across TELUS and affiliated companies.

The question to ask is narrow and answerable: what happens to our data, our dedicated team, and our contractual terms if you are acquired. A vendor that has thought about it will have an answer about data segregation and assignment clauses. A vendor that has not will tell you it is not planning to be acquired, which is not an answer.

Ask for named outcomes, and read the sample size

Logo walls are the weakest form of evidence in this category and the most common. What you want is a named customer, a described engagement, and a number with a denominator.

The disclosure spread is wide enough to be diagnostic. Encord publishes twelve named case studies carrying specific metrics, including Hudl, UiPath, and Pickle Robot; the metrics are self-published and unaudited, but they are attributable, which most of the category cannot match. Scale AI presents mostly logos, with case-study-level description for Mayo Clinic and British Petroleum and a claim that 90 percent of leading generative AI model builders are powered by Scale, stated without methodology. Appen's case-studies page returned no retrievable content when checked.

Apply the same discipline to review scores. Scale AI's GenAI Platform holds 4.5 out of 5 on Gartner Peer Insights from two reviews. Encord holds 4.8 out of 5 on G2 from 65. Appen holds 4.2 out of 5 from 35, but that review base mixes crowdworker feedback with enterprise-buyer feedback, which dilutes the buyer signal you are trying to read.

We should be held to the same standard, and by it we have a gap. HumanSignal Services launched in November 2025 with the acquisition of Erud AI, and there is no independent third-party review coverage of the services line yet. Our published case studies predate the launch and describe the labeling platform. If track record on a services engagement is your top criterion, that is a real consideration and you should weigh it.

Turning the answers into a decision

The seven questions are not equally weighted, and the weighting depends on what you are buying. For a regulated deployment, compliance scope and ownership stability dominate. For a physical AI program, collection capability and workforce disclosure dominate, because the work happens in rooms and in the field rather than in a browser. For a long-horizon program, the contract terms matter more than any current capability, because capabilities change inside the term and the contract does not.

The common thread is that every one of these questions is answerable from public artifacts before you ever take a call, which means the evaluation can start before the sales process does. Where a vendor has not published something, that is not proof of a deficiency, but it does tell you which parts of diligence you will have to run yourself rather than verify from the record.

Working with HumanSignal

We designed HumanSignal Services around the questions above: published pay ranges, customer-owned output with IP indemnification, SOC 2 Type 2 and HIPAA compliance, and quality workflows that run in Label Studio Enterprise so review decisions and agreement metrics are visible to you rather than summarized for you. If you are scoping a collection program, particularly in physical AI and robotics, we can walk through the protocol design and the acceptance criteria before any commitment.

What is the difference between a data collection vendor and a data labeling vendor

A labeling vendor adds structure to data you already own. A collection vendor produces data that did not previously exist, through recruiting, protocol design, and capture in a studio, on site, or in the field. Many vendors market both. The operational skills are different, and the questions you should ask differ accordingly.

Why does contributor pay transparency matter to a buyer

It is a proxy for traceability. A vendor that knows what its contributors are paid, where they are, and how they qualified can usually tell you why a specific label is wrong. A vendor that cannot publish any of it is less likely to be able to diagnose quality problems at the root.

Is a longer compliance certification list better

No. The relevant question is whether the certifications match your risk and what their scope covers. A facility-scoped ISO 27001 certification does not cover a site it does not name, and a vendor with FedRAMP High but no HIPAA attestation is the right choice for a defense program and the wrong one for a clinical dataset.

How should I weigh a vendor with no third-party reviews

Treat it as a real gap and price it, rather than treating it as disqualifying. Ask for references you select rather than references the vendor selects, and ask for a paid pilot with a defined acceptance test. A pilot generates the evidence that reviews would have given you, on your own task.

Related Content