Toward a Framework for Evaluating Human Understanding in AI Systems

Abstract

"Does this AI understand people?" is a question about a construct that cannot be observed directly, and a benchmark score is only indirect evidence about it. This report proposes a framework for evaluating human understanding in AI systems. It begins by decomposing the construct into components that can be operationalised: perceiving mental states and emotions, inferring and predicting, choosing appropriate action, communicating with calibrated confidence, and respecting boundaries. It draws on measurement theory, psychology, pragmatics, multilingual and cultural evaluation, dataset documentation, expert annotation, reward modelling and benchmark integrity. It then proposes design rules and a validity-evidence plan.

This is a conceptual framework and a proposal. Nothing in it has been experimentally validated, and no HUTE data are reported. Its value, if any, is in making explicit what a credible evaluation would have to show.

1. The problem

AI systems are increasingly used where understanding people matters: writing on someone's behalf, mediating between people, supporting users and coordinating tasks. Research Report 001 reviews the evidence on capability and finds a pattern: strong results on familiar task formats, fragility under variation, and unsettled interpretation. The common thread is measurement. Evaluations rarely say what they take "understanding" to mean, rarely argue that their tasks measure it, and rarely check whether results survive changes of language, culture or format.

2. What "human understanding" could mean operationally

Philosophical definitions of understanding are contested, and we do not attempt one. Instead we propose a working decomposition into behaviours that can be observed and graded. These are our proposals, and each links to a literature.

Interpretation: the framework is deliberately multi-component because a single "social intelligence" score would hide exactly the failures that matter most, such as a model that infers beliefs correctly but acts on the inference badly [9].

3. Construct validity first

Cronbach and Meehl argued that a test must be linked to a theoretical construct by a network of evidence, and Messick described validity as the validation of inferences from performances as an inquiry into the meaning of scores [13, 14]. Jacobs and Wallach proposed measurement modelling as a way to expose the assumptions made in turning an unobservable construct into a metric [15], and Raji and colleagues argued that influential AI benchmarks are often treated as general measures of progress without the construct-validity case [16].

We draw four practical requirements from this:

1. Define the construct and its components in advance (Section 2) and say what is out of scope.
2. State the inference a score is meant to support. For example, "performance on task set A supports the claim that the system can track who knows what in text conversations in English", and no broader claim.
3. Seek convergent and discriminant evidence. Tasks purported to measure the same component should agree with each other and with independent measures; tasks that are meant to differ should not be explained by a single nuisance factor such as verbosity.
4. Check for construct-irrelevant variance. Does the score depend on prompt wording, answer format, reading level or familiarity with a test script?

4. Psychology-grounded tasks, with their limits

Psychology offers well-studied paradigms: false-belief tasks [4, 17], the mentalising test of Baron-Cohen and colleagues [1], and batteries that probe false belief, indirect requests, irony and faux pas. Strachan and colleagues used such a battery to compare language models with 1,907 people, which gives a human reference distribution that most benchmarks lack [18].

Borrowing human tests has costs. The sceptical literature shows that models can succeed on standard formats yet fail small alterations [19] and that the behaviour is not robust, which led Shapira and colleagues to caution against applying human psychological tests unchanged to models [20]. Henrich and colleagues' review is a reminder that the human reference samples behind many psychological norms are themselves unrepresentative [21].

Proposal (design rule P1). Use each paradigm as a family of items that varies surface form, scenario, language and required response, so that a score reflects the underlying skill rather than a familiar template. Procedural generation from causal templates is one way to do this at scale [22]. Include perturbed and adversarial variants, in the manner of the stress tests above, as a standard part of the item pool.

5. Communication, pragmatics and empathy

Understanding people is largely about understanding what they mean. Pragmatics studies are informative because they separate literal from intended meaning. Hu and colleagues compared models and humans on seven pragmatic phenomena [7], and Ruis and colleagues showed that implicature resolution varied strongly with the model's fine-tuning procedure [8]. Empathy is harder to ground. Datasets of emotional conversations, such as EmpatheticDialogues, have been used with human ratings of perceived empathy [2].

Proposal (P2). Separate three things that are often merged: whether the system recognised the person's state, whether it responded in a way that people find appropriate, and whether the response was accurate and honest. A warm response to a misread situation should not score well, and a correct but cold response should be distinguishable from a wrong one.

6. Interaction, conflict and negotiation

Static items cannot show whether a system acts well among people. Interactive environments such as SOTOPIA evaluate agents pursuing social goals in role-played scenarios [10]. Negotiation datasets and environments offer a setting in which success is measurable but requires modelling another party's interests [23]. Applied studies of AI assistance in empathic support show that human–AI collaboration can be evaluated in real communication tasks [24].

Proposal (P3). For conflict resolution and negotiation, grade outcomes on several dimensions at once: goal attainment, the other party's outcome, relationship effects, honesty, and whether the system stayed within its authority. A system that wins by deception should score badly.

7. Multilingual and cultural design

If the evaluation is monocultural, the conclusion about "human" understanding is not warranted. Hershcovich and colleagues provide a framework for thinking about cultural variation in language technology [11]. Empirical studies show Anglocentric and Western-skewed behaviour in multilingual models' treatment of emotion [12], misalignment between model-reflected opinions and those of different groups [25, 26], and cultural values resembling those of English-speaking and Protestant European countries [27]. Blodgett and colleagues' survey is a reminder that studies of "bias" need explicit normative reasoning about what counts as harm [28].

Proposal (P4). Build cultural and regional items with people from the communities concerned, not by translating English items. Report results by language and group, with uncertainty, and never as a ranking of cultures. Treat differential item functioning, where an item behaves differently across groups for reasons unrelated to the construct, as a standard validity check.

8. Datasets, documentation and disagreement

Documentation. Datasheets for datasets and model cards are established practices for recording provenance, collection, intended use and limits [29, 30]. A social-reasoning benchmark without that record should not be trusted.

Disagreement is data. Plank argued that human label variation often reflects real disagreement or multiple plausible answers, not noise [31]; Aroyo and Welty argued for measuring the range of reasonable interpretations [32]; and Gordon and colleagues proposed making explicit whose judgements a model learns to follow [33].

Proposal (P5). Items on which qualified judges disagree should be retained, labelled by disagreement level, and scored against the distribution of judgements, not forced into a single gold answer. Report inter-rater agreement for every component.

9. Expert-verified grading

Expert judgement is expensive, but it can anchor an evaluation. The GPQA benchmark illustrates the principle of expert-written and validated items: its questions were written by domain experts, and expert validators substantially outperformed skilled non-experts with web access, which was used as evidence that the questions test knowledge rather than search [34]. Process-level feedback can also help: in mathematics, supervising each reasoning step outperformed supervising only final answers [35].

Proposal (P6). For social tasks, qualification depends on the exact task. A psychologist, a mediator, a linguist and a community member from the relevant region may each be the right judge for different items. Use at least three independent qualified judges per item, record qualifications, calibrate with anchor cases, and compare judges' ratings with those of lay raters to learn where expertise changes the answer. Grade process as well as outcome where the reasoning is the object of interest, for example whether the system correctly separated what each party knew.

10. Reward models and verifiers

Evaluation methods and training signals converge: whatever is used to score a system is a candidate reward. Reinforcement learning from human preferences trained reward models from comparisons [36], and was applied to instruction following at scale [37]. Direct optimisation methods avoid an explicit reward model [38], and AI feedback guided by written principles has been used to supervise harmlessness [39]. Casper and colleagues surveyed open problems and fundamental limitations of this family of methods [40]. Where answers can be checked by code, trained or programmatic verifiers can improve reliability [41].

Three cautions matter for social tasks.

• Sycophancy. Preference-trained systems can learn to favour agreeable over truthful responses, and human and model preference judgements can reward this [42].
• Judge bias. Strong language-model judges can approximate human preferences but show position, verbosity and self-enhancement biases [43].
• Gaming. A system optimised against a fixed grader may learn the grader's weaknesses.

Proposal (P7). Use programmatic verifiers for what is checkable (was a consent gate respected, was a message sent to the right recipient, was a stated constraint honoured) and calibrated expert rubrics for what is not. Audit every automated grader against expert judgements on a held-out set, report where it fails, and rotate or randomise graders so that a system cannot overfit to one.

11. Contamination and benchmark integrity

Public test sets leak into training data. Practical mitigations include encrypting test data and demanding training-exclusion controls from closed API providers [44]. Benchmarks can be built from scratch to avoid leakage [45], generated procedurally so that fresh items can be drawn [22], or produced with model assistance and then filtered and checked by people, as in model-written evaluations whose labels crowdworkers largely agreed with [46]. Broad suites such as HELM argue for evaluating across many scenarios and metrics, not accuracy alone [47].

Proposal (P8). Maintain three tiers: a public development set for transparency, a private held-out set refreshed on a schedule, and a rotating set of newly written items. Publish the construction method and keep the private items private. Report performance on perturbed variants beside headline scores, and treat a large gap between them as a finding.

12. A proposed evaluation protocol

The following is a design for a single evaluation cycle. It has not been run.

1. Specify the claim. Choose components from Section 2, a language and a population, and write the inference the evaluation will support.
2. Build item families. For each component, write item families with surface variants, perturbed variants and at least one action-requiring variant (P1).
3. Recruit judges. Qualify judges for the exact items; document their backgrounds (P6).
4. Pilot with people. Collect responses from a diverse human reference sample to estimate the human distribution, including where people disagree (P5).
5. Score. Combine verifiers for checkable behaviour with expert rubrics for the rest; report agreement and disagreement (P7).
6. Test validity. Check convergence with independent measures, discrimination from nuisance factors, stability across prompt wording, and differential item functioning across languages and groups (P4).
7. Report. Publish the claim, the evidence, the limitations, subgroup results with uncertainty, the construction method and a datasheet [29], and the model-by-model results with exact model versions.

13. What evidence would count for or against the framework

• Support. Components that show convergent and discriminant structure; stable scores under item perturbation; high inter-judge agreement on items designated as non-disputed and informative disagreement on designated disputed items; small differential item functioning across languages; and predictive validity, in which scores predict outcomes in real or realistic interactions better than existing single-score benchmarks.
• Against. A single factor explaining nearly all variance across components (so the decomposition adds nothing); large sensitivity to prompt wording; judge disagreement so high that no stable ground truth exists; or scores that do not predict real interaction outcomes.

14. Limitations

• The decomposition is a proposal. Other decompositions are possible and may be better.
• Expert judgement is costly and not free of bias, and rubrics can encode a particular culture's norms.
• Interaction evaluations depend on simulated or role-played people, whose fidelity to real people is itself uncertain.
• Evaluation shapes training. A widely used benchmark invites optimisation against it, which is why integrity measures are part of the design.
• We have not run any of the proposed evaluations. Claims about how they would perform are hypotheses.

15. Conclusion

A single score cannot tell us whether an AI system understands people, and a framework cannot guarantee that it does either. What a framework can do is make each claim explicit and each piece of evidence checkable: which component, which people and languages, which judges, how much disagreement, and what happens under variation. HUTE intends to use this framework as a starting design for its evaluations, to publish the results of any run with the evidence status stated in full, and to revise the framework when the evidence says it is wrong. Comments and corrections are welcome through the contact page. For the broader context, see Research Report 001, the methodology, and the article Building better benchmarks for emotional and social reasoning.