From 27 September to 1 October, the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS'26, took place in Pittsburgh (USA). As announced, we were there: PhD student Luca Pallonetto presented the paper “The SOSI Toolkit: Ontology, Synthetic Dataset, and Benchmark for Social Situation Understanding in Robotics” in a short talk and at the poster session.

The question at the heart of the paper is a short one: what makes a social situation? Picture the same people in the same room. In one moment they are engaged in a conversation; in the next, each of them is absorbed in a phone. The geometry is nearly identical, but the social situation is not, and neither is the right behaviour for a robot that has to decide whether to approach, wait, interrupt or reroute. Pose, proximity and speech can already be estimated. What is still missing is a principled way to combine them into a scene-level understanding of what is going on socially.

SOSI (SOcial SItuation Interpretation) addresses this with three coordinated pillars:

• an ontology that describes each social situation through the five constructs of Kenneth Burke’s dramatistic pentad: Act (what is happening), Scene (where it happens), Agent (who is involved), Agency (how the interaction is organised) and Purpose (the goal of the interaction);
• a synthetic dataset in which ontology parameters are sampled, written into prompt templates and rendered into images, so that every scene is labelled by construction (19,200 images across 12 classes in the first release);
• a benchmark protocol based on semantic separability, retrieval and synthetic-to-real transfer. With a CLIP baseline in which only the projection heads are trained, Top-3 retrieval on real images captured by our TIAGo robot rises from 76.7 % to 90.0 %.

The discussions at the poster were an opportunity to show where the work has gone since submission. We have recorded a real-robot dataset to close the gap with real data, extended the ontology from three to 17 social dimensions, and started to move beyond CLIP. A learned projection on top of a frozen Qwen3-VL-8B already lifts accuracy on real observations well above chance, where the off-the-shelf model performs at chance. Video is the next step, since motion is essential to reading a social situation.

The ontology is openly available as a standalone resource (SOSI – Social Situation Ontology v1.0.1).

The paper is part of the SWEET project, funded by the EU’s Horizon Europe programme under the Marie Skłodowska-Curie Actions (grant No 101168792). It is co-authored with Lorenzo Ferrini and Viviana Morlando (PAL Robotics), Silvia Rossi (University of Naples Federico II) and Séverin Lemaignan.

Thanks to everyone who stopped by the poster. See you at the next one.

Related project

SWEET (Horizon Europe)

active

Social aWareness for sErvicE roboTs