SocialIA: a social AI engine for interactive agents
Learning how people behave together, from data collected in a multi-player game, to give robots and avatars a working model of social situations
Overview
Robots have become much better at walking, grasping and carrying. They have not become much better at being in a room with people: knowing when to join a conversation, where to stand in a queue, when to offer help and when to step back. Without that understanding, a capable robot is still an intrusive one.
SocialIA sets out to give interactive agents, physical robots as well as virtual avatars, this foundational social understanding. The project is a three-year Proyecto de Generación de Conocimiento funded by the Spanish Agencia Estatal de Investigación (reference PID2025-175726NB-I00), co-led by Raquel Ros, Jordi Sabater-Mir and Séverin Lemaignan.
The starting hypothesis
Recent progress in robot manipulation owes a great deal to large datasets: MIME, RH20T and DROID each record thousands to hundreds of thousands of demonstrations. Nothing comparable exists for social interaction. Existing datasets are small, recorded in laboratories, and often place people around a robot as passive bystanders rather than as participants.
Our hypothesis is that agents can acquire the social competencies they lack when trained on large amounts of interaction data collected in multi-player social games. There is good evidence that this is not as odd as it sounds: studies of online games have shown that players keep their real-world social norms, interpersonal distance included, when they move an avatar through a virtual world.
Objectives
- Map the social mechanisms that matter. Build a taxonomy of social situations that are hard for an agent (small group conversations, waiting in line, collaborating on a task) and of the mechanisms they involve: speech, gaze, gesture, posture, interpersonal distance and orientation.
- Build a game that produces social data. Develop a casual, online, multi-player game in which these situations arise naturally, and record what players do and say.
- Learn a model of social dynamics. Train models that represent a social situation and predict how it will unfold, and use them to generate socially appropriate actions for an agent.
- Test it with people. Evaluate the resulting agents, first as AI players inside the game, then on a robot interacting with people in the real world.
Approach
The game is built on RHYMAS, the agent-based simulation platform developed at IIIA by Jordi Sabater-Mir’s team for emergency training, which already lets human participants and artificial agents share the same virtual environment. We will extend it with generative AI, so that varied environments (“a busy food market, in the evening”) can be assembled from a short description. The game also draws on earlier work in the lab with OfficeBots, a multi-player game for studying human-robot interaction.
A first scenario under study is a cooperative escape room for two players, starting in a lift that has broken down between two floors: a situation that tends to make people talk, and to coordinate.
On the learning side, we build on two ideas. Joint-Embedding Predictive Architectures (JEPA, from Yann LeCun’s group at Meta) learn about the physical world by predicting the abstract representation of what comes next, rather than every pixel. Social embeddings, which we introduced in 2024, offer a compact vector representation of a social situation. Combined, they suggest a “social world model”: a network that predicts how a social situation is likely to evolve, and so lets an agent anticipate rather than react.
The data collection, training and evaluation in the game run in two iterations, so that what we learn from the first model shapes the second. The final year brings the model onto a physical robot.
Expected outcomes
- An open dataset of socially situated interactions, anonymised and released following the FAIR principles, for the wider HRI, multi-agent and machine learning communities.
- The game and the extended RHYMAS platform, as open-source software.
- A social AI engine: learned models of social situation assessment and action selection, integrated on a robot and evaluated with people.
The game also gives non-specialists a way to shape how robots behave: players express their expectations and social norms by playing, not by writing specifications.
Team and collaborators
At IIIA, the three co-leads bring social robotics (Raquel Ros and Séverin Lemaignan) and agent-based simulation (Jordi Sabater-Mir); Jesús Cerquides and Mehmet Oğuz Mülâyim bring machine learning. The project funds a research engineer and a doctoral candidate.
Four colleagues abroad contribute to the work:
- Selma Šabanović, Indiana University Bloomington, on the design of social scenarios and the final evaluation;
- Tony Belpaeme, Ghent University, on the machine learning approaches and their evaluation;
- Iolanda Leite, KTH Royal Institute of Technology, on scenarios, learning and evaluation;
- Wafa Johal, University of Melbourne, on social mechanisms in collaboration, crowd-sourced data collection and annotation.
Our colleagues Núria Vallès-Peris and Margarida Romero advise the project on the sociological side.
Project Team
0000-0002-8295-6932
