First contact is a project aimed at exploring the way two entities can interact upon meeting each other for the first time. One of the two is a person, while the other is either a physical or a virtual creature, with non-humanoid shape and features different from human ones.
From November 9th 2023 to January 20th 2024 experiences of these ways of interacting are offered within the X-Cities exhibition at Politecnico di Milano. The exhibition is in Spazio Nardi, Via Ampère, 2, in the Architecture Area of Politecnico di Milano.
Possibilities for different experiences of the general idea of meeting an alien digital entity are available upon booking on the calendar below. Each slot is intended for two persons, but if you come alone we can manage to run the experience anyway.
An administration manual for this website is available at
https://docs.google.com/document/d/1YQHSmfYedhOWCBrFSKqS9xBvWZN1LW4-ZWMf5M9_5zc/edit?usp=sharing
This page introduces “POLIMI-ITW-S” dataset “POLIMI-ITW-S” contains 37 action classes and 22,164 video samples with total . The average duration of each clip is about 7 seconds. The dataset contains RGB videos, 2-D skeletal data, bounding boxes and labels for each sample. This dataset was taken from RGB cameras of two smartphones by two recorders with resolution 1920×1080 pixels, 30 fps, held by hands about 90 cm from the floor. The 2-D skeletal data and person bounding boxes are generated by the OpenPifPaf. The 2-D skeletal data contains the 2-D coordinates of 17 body joints at each frame. The recorders imitated the mobile robot, keeping moving or staying till by looking around to capture the persons who are performing the actions. We did not mount the camera on a robot in order to avoid uncommon situations that the presence of a robot could trigger.
1. Action Classes
As shown in the tables below, the actions in the dataset are distributed on three levels: • General Level: labels are used for single action. • Modifier Level: labels are used for actions of multiple persons. • Aggregate Level detailed labels aim at describing multiple actions in a single label.
1.1 General Level Actions (10)
A1: cleaning
A2: crouching
A3: jumping
A4: laying
A5: riding
A6: running
A7: scooter
A8: sitting
A16: standing
A27: walking
–
–
1.2 Modifier Level Actions (3)
A8: sittingTogether
A17: standingTogether
A27: walkingTogether
1.3 Aggregate Level Actions (24)
A10: sittingWhileCalling
A11: sittingWhileDrinking
A12: sittingWhileEating
A13: sittingWhileHoldingBabyInArms
A14: sittingWhileTalkingTogether
A15: sittingWhileWatchingPhone
A18: standingWhileCalling
A19: standingWhileDrinking
A20: standingWhileEating
A21: standingWhileHoldingBabyInArms
A22: standingWhileHoldingCart
A23: standingWhileHoldingStroller
A24: standingWhileLookingAtShops
A25: standingWhileTalkingTogether
A26: standingWhileWatchingPhone
A29: walkingWhileCalling
A30: walkingWhileDrinking
A31: walkingWhileEating
A32: walkingWhileHoldingBabyInArms
A33: walkingWhileHoldingCart
A34: walkingWhileHoldingStroller
A35: walkingWhileLookingAtShops
A36: walkingWhileTalkingTogether
A37: walkingWhileWatchingPhone
2. Size of Datasets
The dataset includes three types of files:
RGB videos: collected RGB videos.
2-D skeletons + bounding boxes + labels: JSON format files including 2-D skeletons, bounding boxes and labels for each RGB video.
pre-processed data: splitted into “training” (70%) and “test” (30%) data with format “.npy” for joint body data and “.pkl” for label data.
The size of each type is shown in the below table:
Data Type
POLIMI-ITW-S
RGB videos
335 GB
2-D skeletons + bounding boxes + labels
39.4 GB
pre-processed data (.npy and .pkl)
17.7 GB
Total
392.1 GB
3. More Information (FAQs and Sample Codes)
We have developed the annotation tool which should be used to visualize poses, bounding boxes and labels on the video clips.
We provide the developed annotation tool, data pre-processing script, information about the data, answers to FAQs, samples codes to read the data, and the latest published results on our datasets here.
The datasets are released for academic research only, and are free to researchers from educational or research institutes for non-commercial purposes. The use of the dataset is governed by the following terms and conditions:
Without the expressed permission of the AIRLab, any of the following will be considered illegal: redistribution, derivation or generation of a new dataset from this dataset, and commercial usage of any of these datasets in any way or form, either partially or in its entirety.
For the sake of privacy, images of all subjects in any of these datasets are only allowed for the demonstration in academic publications and presentations.
All users of “POLIMI-ITW-S” dataset agree to indemnify, defend and hold harmless, the AIRLab and its officers, employees, and agents, individually and collectively, from any and all losses, expenses, and damages.
Everyday objects can be animated to improve their functionalities and to provide a more interesting environment. Moreover, it is interesting to explore new interaction situations. Proper design of shape and interaction is needed to obtain interesting objects. Emotional expression is an interesting aspect to explore.
We have developed a couple of emotional trash bins, going around to invite to trash selected materials by using the lid movements and sounds, a coat-hanger (IGHOR), welcoming people entering and asking to have their coats, showing sadness if they keep it on, a naughty fan, coming close and suddenly investing the person with an air flow, a naughty money saver that has to be chased to give it money, a kind of pillow that react with sounds to the way it is touched.
Robots can be used as artistic media, able to perform and interact with people in artistic representations.
Interacting with a different entity: First contact
In order to explore the interaction possibilities, we are developing experiences where a person can interact with another entity, having different form and abilities, and should find strategies to communicate, possibly to perform shared tasks. The experiences are done both with robots and in virtual reality. More details on this post.
Robot actor
We have developed an autonomous robotic actor, able to participate to a public representation with a defined role and script, and we are developing an Improv robotic actor able to adapt its performance to external stimuli. We have firstly developed a robot able to move in classical scenes (e.g. the balcony scene of Romeo and Juliet) selecting the proper emotional expressions for the situation, and a framework to define emotional expressions according to the social setting among characters, and the situation. We then developed Robocchio, a robot able to implement a script and adapt to the timing of the partner on stage, tested on an adapted excerpt of Pinocchio by Collodi. We then improved Robocchio to improvise by visually recognizing 17 “scenic actions” and reacting to them with scenic actions influenced by one of 16 different psychological types. The final step to include improvised verbal interaction is under development.
Interactive robotic art
Robots can have different shapes and play different roles in interactive artistic performances. We are exploiting materials like nets, polyethilene sheets, polyurethane foams and other materials to obtain shapes interesting to move in interactive exhibits. Emotional expression is also in this area, an interesting feature to explore.
METRICS (Metrological Evaluation and Testing of Robots in International CompetitionS) organises challenge-led and industry-relevant competitions in the four Priority Areas (PAs) identified by ICT-09-2019-2020: Healthcare, Infrastructure Inspection and Maintenance (I&M), Agri-Food, and Agile Production.
Within METRICS, AIRLab is in charge of the ACRE (Agri-food Competition for Robot Evaluation) competition, dedicated to benchmarking agricultural robots.
TEINVEIN (TEcnologie INnovative per VEicoli INtelligenti) is a project financed by Regione Lombardia for the construction of a platform for intelligent autonomous vehicles.
