The Future of Human-AI Relationships and Personal AI Agents

Two ideas about the future of AI and relationships are often confused. One is the AI companion: a system that plays the role of a friend or partner. The other is the personal agent: a system that acts on your behalf, handling messages, schedules and tasks, and perhaps representing you to others. The first is about a relationship with an AI. The second is about how AI changes your relationships with other people. They raise different questions, and the research is thinner on both than the confidence of the commentary suggests.

Relationships with AI

People form attachments to technology. Nass and Moon's review of experiments showed that people mindlessly apply social rules, such as politeness and reciprocity, to computers [1]. Companion chatbots lean into that tendency. Skjuve and colleagues conducted an interview study of people who use a companion chatbot, documenting how those human-chatbot relationships develop [2].

What we can say responsibly is limited. We found no long-term, independent evidence in the sources we reviewed on whether such relationships help or harm over years, for whom, and under what designs. That is a research gap, and it is itself a reason for caution. A related risk that is better documented concerns what systems learn to say: preference-trained assistants show sycophancy, and human raters and preference models can favour agreeable responses over correct ones [3]. An assistant that is rewarded for approval has an incentive to flatter, and flattery is a poor basis for a relationship of any kind.

Relationships through AI

Hancock, Naaman and Levy defined AI-mediated communication as interpersonal communication in which an intelligent agent modifies, augments or generates messages on behalf of a communicator [4]. A personal agent that drafts your replies, schedules your meetings or introduces you to someone is a case of this. The authors argued for a research agenda covering psychological, linguistic, relational, policy and ethical implications.

A practical question hides in the definition: who is speaking? If an agent writes in your voice, the recipient may form a view of you based on words you did not choose. Jakesch and colleagues examined how the perception that profile text was written by AI affects trustworthiness [5]. As such agents spread, people will form expectations about when AI is involved, and what that implies about the sender.

There are upsides. In text-based peer support, human–AI collaboration was reported to enable more empathic conversations, with the AI helping the human supporter respond better [6]. The design lesson is that the human stays the author of the relationship, and the AI helps.

Agents that remember

A personal agent that is useful over months needs memory: who people are, what was promised, what is sensitive. Research on generative agents showed that giving a language-model agent a memory stream, periodic reflection and planning produced believable individual and social behavior in a simulated town [7]. In a real product, memory raises hard questions of consent, correction and deletion, and the cost of a wrong inference grows when the memory concerns real people.

Risks

Weidinger and colleagues' taxonomy of risks from language models includes harms arising from human–computer interaction as one of six areas [8]. For personal agents, the following seem most important, and we present this list as our own analysis.

• Impersonation and disclosure. The recipient should know when they are dealing with an AI, and the agent should not pass as the person.
• Overreach. Sending, paying, disclosing location or committing to something are actions that should require clear permission.
• Leakage of third-party information. An agent that knows about your contacts must not reveal them to others.
• Dependence. Systems optimised for engagement can encourage reliance that serves the product more than the person.
• Manipulation. An agent that is skilled at modelling people can influence them; limits on that power are a safety requirement.

Design principles we hold to

For HUMAN, which is in development, the principles below describe our intent:

1. Identify as AI. The agent says what it is.
2. Act only within granted permissions. High-stakes actions need explicit approval.
3. Layer what is shared. Public, relationship-specific and private information are separated, and the private layer is never available to outside questions.
4. Keep humans connected to humans. The goal is more and better human relationships, not substitution.
5. Make memory inspectable. People can see, correct and delete what the agent remembers.
6. Do not turn private data into training data. See Responsible AI.

These principles are commitments about design. Whether any system lives up to them is a matter for evidence, which is why we publish how we evaluate. See HUMAN for status. For the wider social picture, read the future of human connection in an AI-powered world; for why modelling people well matters, why AI needs to understand human behavior.