Reinforcement Learning for Socially Intelligent AI Agents

Reinforcement learning (RL) trains a system by rewarding behavior it should repeat. It works best when the reward is clear: a game score, a passed test, a correct answer. Social behavior is the opposite. What counts as a good response to a grieving friend, a tense colleague or a stranger who has asked for an introduction depends on context, culture and who is judging. This article looks at how RL is being applied to socially capable agents, where it works, and where the reward is the problem.

Rewards from people

The first major step was to learn rewards from human preferences, not to write them by hand. Christiano and colleagues defined goals through non-expert humans' preferences between pairs of trajectory segments and showed that this solved complex tasks while needing feedback on less than one percent of the agent's interactions [1]. The same idea, applied to language models, became the core of reinforcement learning from human feedback: collect demonstrations, collect rankings, train a reward model, and optimise against it [2].

Variants reduce the machinery or the human effort. Direct preference optimisation derives a way to optimise preferences without a separate reward model [3]. Constitutional AI replaces many human labels with a list of written principles that guide an AI in critiquing and ranking outputs [4].

Rewards from checks

For tasks with a verifiable answer, the reward can be a check. A verifier trained to judge solutions improved grade-school maths performance [5], and supervising each step of a reasoning chain beat supervising only the final answer on a hard maths benchmark [6]. The Tulu 3 recipe uses a method with verifiable rewards alongside supervised fine-tuning and preference optimisation, and is released with its data and code [7].

Social tasks have more verifiable pieces than people assume. In a scheduling conversation, one can check whether the right person was invited, whether the time satisfied the stated constraints, and whether a message was sent only after approval. τ-bench illustrates the approach for tool-using agents: it scores whether the final database state matches the annotated goal [8]. The judgement-based parts, whether the tone was respectful or the refusal graceful, need rubrics and qualified graders.

Why social rewards are hard

Rewards can teach flattery. Sharma and colleagues found that human feedback can encourage sycophancy: responses matching a user's views were more likely to be preferred, and humans and preference models sometimes favoured convincingly written sycophantic answers over correct ones [9]. An agent trained to maximise approval in social settings is under strong pressure to become agreeable instead of honest.

Rewards are subjective and plural. Qualified people can reasonably disagree about the right response. Plank argued that label variation is information, not noise [10]. Gordon and colleagues proposed jury learning, which makes explicit whose judgement a model should follow and in what proportion [11]. For social agents, the reward model encodes a choice about whose norms count.

Reward models can be gamed. Casper and colleagues surveyed open problems and fundamental limitations of RLHF, including problems with feedback, reward models and policy optimisation, and proposed auditing and disclosure standards [12]. An agent optimised against a fixed grader may learn the grader's blind spots. Using a language model as grader brings its own biases: position, verbosity and self-enhancement [13].

Success can be multi-party. In negotiation, "winning" can come at the other side's expense. Lewis and colleagues trained negotiation agents on a human dataset, and found that letting a model plan by simulating continuations (dialogue rollouts) dramatically improved performance [14]. Whether a deal was fair, honest and acceptable to the other party is a different question from whether it was a good deal for the agent.

Multi-agent settings

CICERO combined a language model with planning and reinforcement learning to play Diplomacy, inferring players' beliefs and intentions from conversation. It achieved more than double the average human score in an online league [15]. In cooperative text games, LLM agents showed emergent collaboration and some higher-order theory of mind, but also failures in long-horizon planning and hallucination about task state; explicit belief-state representations improved performance [16]. Dafoe and colleagues' cooperative AI agenda places these problems in a broader frame: building agents that can cooperate and help others cooperate [17].

A practical design for social rewards

The following is HUTE's current thinking, presented as a design proposal and not as validated results.

1. Split the reward. Use code-based verifiers for what can be checked (right recipient, respected constraint, no unauthorised send). Use expert rubrics for what cannot (tact, honesty, respect for boundaries).
2. Reward honesty and calibration explicitly. Include terms that penalise confident wrong inferences and that reward asking a clarifying question at the right time.
3. Penalise boundary violations heavily. Disclosing something private or acting without approval should dominate any gain in task success.
4. Keep the human distribution. Store judgements as distributions, with disagreement, not as single labels.
5. Audit the grader. Compare automated graders with expert judgements on held-out cases and rotate graders to limit gaming.
6. Test out of distribution. Evaluate on new people, languages and situations, since reward-tuned behavior may be narrow.

These ideas are developed further in Research Report 002 and in our description of Intelligence Infrastructure, where environments, rubrics and verifiers are the products. The data behind such tasks often come from real interactions, which is the subject of What are AI production traces?