Research Report 003

Human Expertise as Infrastructure for the Next Generation of Artificial Intelligence

Human knowledge enters AI through demonstrations, preferences, graders and real tasks. We review how, what the evidence says about its value, and the consent, privacy and fairness questions that determine whether it can be used responsibly.

HUTE ResearchPublished 12 min readVersion 1.0
In this report
  1. Abstract
  2. 1. Scope
  3. 2. Method and status
  4. 3. Where human knowledge enters the pipeline
  5. 4. Expert demonstrations and the value of curation
  6. 5. Preference learning and reinforcement learning
  7. 6. Rubrics, graders and the problem of subjective tasks
  8. 7. Production-derived tasks and "traces"
  9. 8. Documentation, provenance and licensing
  10. 9. Consent and privacy
  11. 10. Supply, synthetic data and the economics of human data
  12. 11. People who produce the data
  13. 12. Regional and linguistic diversity
  14. 13. Implications for HUTE (position, not result)
  15. 14. Limitations
  16. 15. Conclusion

Abstract

Frontier AI is often described as the product of compute and data, but a growing share of what makes models useful comes from people: experts who write demonstrations, annotators who rank outputs, graders who write rubrics, and users whose real tasks reveal where systems fail. This report reviews how human expertise enters the AI pipeline, what the published evidence says about its value, and what must be true for it to be used legitimately. We cover supervised fine-tuning, preference learning and reinforcement learning, expert rubrics and verification, production-derived tasks, human feedback quality and disagreement, provenance and licensing, consent and privacy, the supply of human data, and the conditions of the people who produce it. We conclude with implications for how an organisation such as HUTE should build human-expertise infrastructure, and flag the points at which the evidence is thin.

1. Scope

The question is deliberately practical: given that AI development depends on human knowledge, what is the right way to build infrastructure around it? We interpret "infrastructure" broadly: the processes, records, contracts and tools that turn human knowledge into reliable, rights-cleared training and evaluation material. We do not review general pretraining on web text except where it bears on the supply of human-generated data. We do not evaluate any specific commercial provider.

2. Method and status

This is a structured literature review with analysis. Sources were selected for relevance to the pipeline stages above, favouring peer-reviewed work and recognised archives, and including studies that complicate the case for human data. All references were verified against Crossref or arXiv (see methodology). Findings are attributed; interpretation is labelled. We did not reproduce any study.

3. Where human knowledge enters the pipeline

StageRole of peopleRepresentative evidence
PretrainingAuthorship of the text and images models learn from[1, 2]
Supervised fine-tuning (SFT)Demonstrations of desired behaviour[3, 4]
Preference learningRankings or comparisons of outputs[5, 3, 6]
Reinforcement learningReward models, rubrics, verifiers and environments[7, 8, 9]
EvaluationExpert-written questions, expert grading, human reference baselines[10, 11]
Real-world task dataUsers and professionals whose tasks become benchmarks[12, 13, 14]

Interpretation: the later the stage, the more the contribution is judgement and expertise rather than raw text, and the more the quality of an individual contribution matters.

4. Expert demonstrations and the value of curation

InstructGPT began with prompts written by labelers and submitted through an API, collected labeler demonstrations of desired behaviour for supervised fine-tuning, and then collected rankings of model outputs for reinforcement learning from human feedback. In human evaluations on the authors' prompt distribution, outputs from the 1.3-billion-parameter InstructGPT model were preferred to those of the 175-billion-parameter GPT-3, despite the model being a hundred times smaller [3]. The LIMA study fine-tuned a 65-billion-parameter model on only 1,000 carefully curated prompts and responses, with no reinforcement learning, and reported remarkably strong performance, supporting the view that much capability is learned in pretraining and that limited, high-quality instruction data can suffice to elicit it [4].

Interpretation: these studies point in one direction. Quality and curation can matter more than volume, which raises the value of people who can judge what "good" looks like. They are not a proof that small data always suffices. LIMA's results were measured on its own evaluation setup and for its own target behaviours.

Expert-written evaluations make a related point. GPQA consists of 448 multiple-choice questions written by domain experts in biology, physics and chemistry. Experts who held or were pursuing PhDs in the relevant field reached 65% accuracy, while skilled non-experts with unrestricted web access reached 34%, a gap the authors used to argue that the questions are difficult and "Google-proof" [10].

5. Preference learning and reinforcement learning

Learning from human preferences lets people specify goals they find hard to formalise. Christiano and colleagues showed that goals defined through non-expert human preferences between pairs of trajectory segments could solve complex tasks while requiring feedback on less than one percent of the agent's interactions [5]. This line of work became the basis for RLHF in language models [3]. Direct preference optimisation avoids training a separate reward model [6], and Constitutional AI uses a list of written principles so that an AI can generate the preference labels (RL from AI feedback), with human oversight provided by the principles rather than by labels for each case [15].

Casper and colleagues surveyed open problems and fundamental limitations of RLHF and proposed auditing and disclosure standards [16]. For our purposes the important lesson is that preference data inherit the limits of the people who provide them: who they are, what they were asked, and how well they could judge.

Reinforcement learning also uses signals that need no human in the loop at training time, but still depend on human expertise to design. Training verifiers to judge candidate solutions improved performance on grade-school maths problems [8]; supervising each reasoning step outperformed supervising only final answers on a challenging maths benchmark [7]; and the open Tulu 3 recipe combines supervised fine-tuning, preference optimisation and a reinforcement-learning method using verifiable rewards, released with its data, code and evaluation scheme [9]. Interpretation: verifiable rewards shift the human contribution from labelling outputs to specifying tasks, tests and rubrics, which requires deep domain knowledge.

6. Rubrics, graders and the problem of subjective tasks

For tasks that cannot be checked by code, such as tact, consent, empathy and fairness, the grader is a rubric applied by qualified people. Three issues recur.

  • Disagreement is signal. Plank argued that human label variation often reflects genuine disagreement or multiple plausible answers and should not be treated as noise [17]. Aroyo and Welty argued for measuring the spread of reasonable interpretations across a crowd [18].
  • Whose judgement. Gordon and colleagues' jury learning makes explicit which groups' labels a model should emulate, and in what proportion, instead of settling disagreements by majority vote [19].
  • Model graders have biases. Strong language-model judges can match human preferences on benchmarks but exhibit position, verbosity and self-enhancement biases [20]; preference-trained assistants can favour agreeable over truthful responses [21].

Interpretation: expert rubrics need documented qualifications, calibration on anchor cases, multiple independent judges and a record of disagreement. The deliverable is not a single label, but a label with its evidence. See Research Report 002 for a proposed protocol.

7. Production-derived tasks and "traces"

Real use reveals what systems cannot do. SWE-bench assembled 2,294 software-engineering problems drawn from real GitHub issues and pull requests across 12 popular Python repositories, and gave language models the task of editing a codebase to resolve the issue [12]. WebArena created a realistic, reproducible environment of functional websites for evaluating web agents [14], and Mind2Web collected more than 2,000 open-ended tasks from 137 real websites across 31 domains, with crowdsourced action sequences [22]. τ-bench emulates conversations between a simulated user and an agent equipped with domain tools and policy guidelines, scoring the final database state against an annotated goal and introducing a metric for reliability across repeated trials; it found that even strong agents succeeded on fewer than half of tasks and were inconsistent [13].

These works illustrate a design pattern that we call, in HUTE's vocabulary, trace-to-task reconstruction: take records of real work, recover the original goal and context, rebuild an environment in which the task can be re-run, and add a check that tells success from failure. See the article What are AI production traces, and why do they matter?

Interpretation: three difficulties make this harder than it sounds. A final artefact hides the constraints, corrections and changed requirements that make a task instructive. Real records contain personal and confidential information. And the verifier must not be gameable. Where the evidence is thin: the public literature is strongest for software and web tasks, where checks are mechanical. We found far less published work on reconstructing socially rich, high-stakes tasks, where success is partly a matter of judgement.

8. Documentation, provenance and licensing

A dataset without a record of its origins cannot be responsibly used. Gebru and colleagues proposed datasheets for datasets to document motivation, composition, collection process and recommended uses [23], and Mitchell and colleagues proposed model cards for reporting models' intended use and performance across conditions [24]. Paullada and colleagues surveyed dataset development and use across machine learning research [25].

The state of practice in the wild is concerning. The Data Provenance Initiative audited more than 1,800 text datasets and found sharp divides between commercially open and closed data and inconsistent documentation of licences and attribution, and built tools to trace dataset lineage [2]. A longitudinal audit of 14,000 web domains found a proliferation of AI-specific restrictions, inconsistencies between sites' terms of service and their robots.txt files, and an observed decline in the crawlable data commons [26].

Interpretation: provenance is infrastructure. For each source an organisation should be able to answer: who produced it, who owns it, what permission covered what use, what was promised to contributors, and what must be removed on request.

Consent. The EU General Data Protection Regulation defines valid consent as freely given, specific, informed and unambiguous, and gives people rights over their data, including erasure in defined circumstances [27]. Interpretation: consent to use a product is not consent to a different purpose; training, evaluation and licensing each need their own basis, and a company's rights over its software do not convey rights over its users' private content.

Re-identification. Removing names is not the same as anonymising. In a study of mobility data, a small number of spatio-temporal points sufficed to single out most individuals [28]. A generative model estimated that 15 demographic attributes could re-identify the great majority of Americans in a heavily incomplete dataset [29]. Sparse high-dimensional records can be de-anonymised even when data are perturbed [30].

Memorisation. Training-data extraction attacks recovered verbatim sequences, including personal information, from a language model trained on public web scrapes [31].

Interpretation: a responsible pipeline minimises collection, redacts and transforms with an awareness that redaction is imperfect, restricts access, isolates sensitive raw records, and excludes categories (for example clinical, children's and confidential employment information) where safe use cannot be demonstrated.

10. Supply, synthetic data and the economics of human data

Is human-generated data becoming scarcer? Villalobos and colleagues forecast that if current trends continue, models will be trained on datasets roughly equal in size to the available stock of public human-generated text between 2026 and 2032, and discussed synthetic data, transfer learning and data efficiency as ways to progress beyond it [1]. Shumailov and colleagues found that indiscriminate use of model-generated content in training causes irreversible defects, a phenomenon they call model collapse, in which the tails of the original distribution disappear [32].

Interpretation, flagged as such: these findings together support a plausible argument that fresh, high-quality human contributions, especially those that cover the tails (rare situations, languages, local knowledge, expert judgement), keep their value even as synthetic data grow. They do not show that any particular company's data will command a price, nor how large a market will be. We found no independent, verified market-size figures and make no estimate. Demand for such data should be validated with customers and measured by outcomes: whether training on a dataset produces a measured improvement on held-out tasks.

11. People who produce the data

Human data work is labour. Hara and colleagues analysed workers' earnings on Amazon Mechanical Turk and reported a median hourly wage of roughly two US dollars, with only a small share of workers earning the US federal minimum wage [33]. Irani and Silberman described Turkopticon, a tool through which workers on that platform reviewed employers and made working conditions visible [34].

Interpretation: if expertise is infrastructure, the people who provide it need conditions that make quality possible, namely transparent pay, clear acceptance criteria with explanations, dispute channels, and voluntary participation. This is also a quality argument: experts will not spend time on poorly paid, opaque tasks. Contributors in every region should be paid fairly, and local expertise deserves recognition as expertise.

12. Regional and linguistic diversity

Most of the evidence and tooling in this field is English-centred. Multilingual models can be Anglocentric even when they handle other languages [35]; cultural variation is distinct from linguistic variation [36]; and the psychological literature itself rests heavily on Western, educated, industrialised, rich and democratic samples [37]. Interpretation: this is a strong argument for sourcing expertise from many regions, and designing evaluations with local communities, but it must not turn into a demand that individuals represent a culture. Individual variation within every region is large.

13. Implications for HUTE (position, not result)

The following is a statement of HUTE's intended approach, informed by the review. It describes design intent, and makes no claim about customers, revenue or results.

  1. Treat quality as the product. Invest in qualified experts, calibrated rubrics and documented disagreement over volume.
  2. Start from failures. Build datasets and environments around a specific capability gap, and measure improvement on held-out tasks.
  3. Make provenance a first-class record. Every item should carry its origin, rights and restrictions.
  4. Keep private product data out. Private HUMAN content is not part of the data business; see Responsible AI.
  5. Pay people well and explain decisions.
  6. Publish evidence status. Report both gains and failures.

14. Limitations

  • The literature on trace-to-task conversion for social and professional work is thin; we have drawn on software and web agent studies.
  • We found no verified public data on the market economics of expert data, and say nothing quantitative about it.
  • Many cited studies are preprints, and many concern specific model versions that are quickly replaced.
  • This is a structured, not systematic, review, and selection reflects our judgement.

15. Conclusion

The evidence is consistent with a view of human expertise as infrastructure: a set of capabilities, records and relationships that has to be built deliberately. Expert demonstrations and curated data can be disproportionately valuable; preference and reward signals are only as good as the people and rubrics behind them; real tasks reveal gaps that synthetic tests miss; and none of it can be used responsibly without provenance, consent and fair treatment of contributors. The hardest open questions are the ones in the middle: how to convert socially rich real work into reliable, safe, rights-cleared tasks, and how to measure whether a dataset made a system better at what mattered. Those are the questions HUTE Intelligence Infrastructure is designed around, and the questions we will report on, with evidence status stated, as the work proceeds. Related reading: Human expertise, training data and the economics of AI development.

References

  1. Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L. & Hobbhahn, M. (2022). Will we run out of data? Limits of LLM scaling based on human-generated data. arXiv:2211.04325. arxiv.org/abs/2211.04325
  2. Longpre, S., Mahari, R., Chen, A., Obeng-Marnu, N., Sileo, D., et al. (2023). The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI. arXiv:2310.16787. arxiv.org/abs/2310.16787
  3. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. arXiv:2203.02155. arxiv.org/abs/2203.02155
  4. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., et al. (2023). LIMA: Less Is More for Alignment. NeurIPS 2023. arXiv:2305.11206. arxiv.org/abs/2305.11206
  5. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S. & Amodei, D. (2017). Deep reinforcement learning from human preferences. NeurIPS 2017. arXiv:1706.03741. arxiv.org/abs/1706.03741
  6. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D. & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290. arxiv.org/abs/2305.18290
  7. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., et al. (2024). Let's Verify Step by Step. ICLR 2024. arXiv:2305.20050. arxiv.org/abs/2305.20050
  8. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. arxiv.org/abs/2110.14168
  9. Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. arxiv.org/abs/2411.15124
  10. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., et al. (2024). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. COLM 2024. arXiv:2311.12022. arxiv.org/abs/2311.12022
  11. Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., et al. (2024). Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7), 1285-1295. doi.org/10.1038/s41562-024-01882-z
  12. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR 2024. arXiv:2310.06770. arxiv.org/abs/2310.06770
  13. Yao, S., Shinn, N., Razavi, P. & Narasimhan, K. (2024). $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. arxiv.org/abs/2406.12045
  14. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024. arXiv:2307.13854. arxiv.org/abs/2307.13854
  15. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. arxiv.org/abs/2212.08073
  16. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., et al. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. TMLR 2023. arXiv:2307.15217. arxiv.org/abs/2307.15217
  17. Plank, B. (2022). The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation. EMNLP 2022. arXiv:2211.02570. arxiv.org/abs/2211.02570
  18. Aroyo, L. & Welty, C. (2015). Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine, 36(1), 15-24. doi.org/10.1609/aimag.v36i1.2564
  19. Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J. T., et al. (2022). Jury Learning: Integrating Dissenting Voices into Machine Learning Models. CHI 2022. arXiv:2202.02950. arxiv.org/abs/2202.02950
  20. Zheng, L., Chiang, W. L., Sheng, Y., Zhuang, S., Wu, Z., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv:2306.05685. arxiv.org/abs/2306.05685
  21. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., et al. (2024). Towards Understanding Sycophancy in Language Models. ICLR 2024. arXiv:2310.13548. arxiv.org/abs/2310.13548
  22. Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., et al. (2023). Mind2Web: Towards a Generalist Agent for the Web. NeurIPS 2023. arXiv:2306.06070. arxiv.org/abs/2306.06070
  23. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., et al. (2018). Datasheets for Datasets. arXiv:1803.09010. arxiv.org/abs/1803.09010
  24. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., et al. (2019). Model Cards for Model Reporting. FAT* 2019. arXiv:1810.03993. arxiv.org/abs/1810.03993
  25. Paullada, A., Raji, I. D., Bender, E. M., Denton, E. & Hanna, A. (2021). Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 2(11), 100336. doi.org/10.1016/j.patter.2021.100336
  26. Longpre, S., Mahari, R., Lee, A., Lund, C., Oderinwale, H., et al. (2024). Consent in Crisis: The Rapid Decline of the AI Data Commons. arXiv:2407.14933. arxiv.org/abs/2407.14933
  27. European Union (2016). Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data (General Data Protection Regulation). Official Journal of the European Union. eur-lex.europa.eu/eli/reg/2016/679/oj
  28. de Montjoye, Y. A., Hidalgo, C. A., Verleysen, M. & Blondel, V. D. (2013). Unique in the Crowd: The privacy bounds of human mobility. Scientific Reports, 3(1). doi.org/10.1038/srep01376
  29. Rocher, L., Hendrickx, J. M. & de Montjoye, Y. A. (2019). Estimating the success of re-identifications in incomplete datasets using generative models. Nature Communications, 10(1). doi.org/10.1038/s41467-019-10933-3
  30. Narayanan, A. & Shmatikov, V. (2008). How To Break Anonymity of the Netflix Prize Dataset. IEEE Symposium on Security and Privacy 2008 (preprint). arXiv:cs/0610105. arxiv.org/abs/cs/0610105
  31. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., et al. (2021). Extracting Training Data from Large Language Models. USENIX Security 2021. arXiv:2012.07805. arxiv.org/abs/2012.07805
  32. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631(8022), 755-759. doi.org/10.1038/s41586-024-07566-y
  33. Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C. & Bigham, J. P. (2018). A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. doi.org/10.1145/3173574.3174023
  34. Irani, L. C. & Silberman, M. S. (2013). Turkopticon: interrupting worker invisibility in amazon mechanical turk. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. doi.org/10.1145/2470654.2470742
  35. Havaldar, S., Rai, S., Singhal, B., Liu, L., Guntuku, S. C. & Ungar, L. (2023). Multilingual Language Models are not Multicultural: A Case Study in Emotion. EMNLP 2023. arXiv:2307.01370. arxiv.org/abs/2307.01370
  36. Hershcovich, D., Frank, S., Lent, H., Lhoneux, M. D., Abdou, M., et al. (2022). Challenges and Strategies in Cross-Cultural NLP. ACL 2022. arXiv:2203.10020. arxiv.org/abs/2203.10020
  37. Henrich, J., Heine, S. J. & Norenzayan, A. (2010). The weirdest people in the world?. Behavioral and Brain Sciences, 33(2-3), 61-83. doi.org/10.1017/S0140525X0999152X

Human Expertise, Training Data, and the Economics of AI Development

Where human knowledge enters AI, why curated expert data can beat volume, how data scarcity and model collapse matter, and what consent and fair pay require.

Infrastructure

What Are AI Production Traces, and Why Do They Matter?

A production trace is the record of an AI doing real work. Why traces reveal what benchmarks miss, how they become training tasks, and what rights and privacy limits apply.

Infrastructure

The State of Social Intelligence in Artificial Intelligence: Capabilities, Limitations, and Open Problems

A literature review of social intelligence in AI: theory of mind, emotion, pragmatics, multi-agent behavior, evaluation limits, culture, safety risks and open problems.

Research Report 001