diff --git a/README.md b/README.md index fb464b3ff7eb326d0e0f05796c02ca09c88d2f12..54c1abef461a9c25ae64b3638dc4223ab8ca56d9 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,15 @@ architecture, as well as mock ROS nodes for all the architecture modules.  +You can view a textual version of the architecture here: +[spring-architecture.md](spring-architecture.md). + The [architecture.json](architecture.json) file contains the details of the -nodes and interfaces in `json` format, and can be visualised with [the SPRING branch of Boxology](https://github.com/severin-lemaignan/boxology/tree/spring). You can [download a binary for Linux here](https://github.com/severin-lemaignan/boxology/releases), or compile it yourself. Note that Boxology tends to crash often! Ctrl+S is your friend! +nodes and interfaces in `json` format, and can be visualised with [the SPRING +branch of Boxology](https://github.com/severin-lemaignan/boxology/tree/spring). +You can [download a binary for Linux +here](https://github.com/severin-lemaignan/boxology/releases), or compile it +yourself. Note that Boxology tends to crash often! Ctrl+S is your friend! ## Running the mock nodes diff --git a/spring-architecture.md b/spring-architecture.md index 13c28715c42923b4581bc7c8e5580ae5baab6bf6..6a09adc77450991773ab093a1dce4981d9415dc5 100644 --- a/spring-architecture.md +++ b/spring-architecture.md @@ -1,6 +1,6 @@ # SPRING architecture -**Version:** 0.3.1 +**Version:** 0.3.2 EU H2020 SPRING architecture @@ -8,46 +8,46 @@ EU H2020 SPRING architecture | **Node** | **id** | **Partner** | **Status** | |----------|--------|-------------|------------| -| [robot_behavior](#robot_behavior) | robot_behavior | INRIA | mock-up | -| [Sound source localisation](#soundsourcelocalisation) | soundsourcelocalisation | BIU | mock-up | -| [Robot functional layer](#robotfunctionallayer) | robotfunctionallayer | PAL | mock-up | -| [Voice speech matching](#voicespeechmatching) | voicespeechmatching | BIU | mock-up | -| [Soft biometrics](#softbiometrics) | softbiometrics | UNITN | mock-up | -| [User attention estimation](#userattentionestimation) | userattentionestimation | UNITN | mock-up | -| [ hri_msgs](#hri_msgs) | hri_msgs | Other | released (version 0.1.1) (dependency) | -| [Speech synthesis](#speechsynthesis) | speechsynthesis | PAL | mock-up | -| [ROS openpose](#rosopenpose) | rosopenpose | INRIA | mock-up | -| [Robot GUI](#robotgui) | robotgui | ERM | mock-up | -| [interaction_manager](#interaction_manager) | interaction_manager | HWU | mock-up | -| [Object detection/identification/localisation](#objectdetectionidentificationlocalisation) | objectdetectionidentificationlocalisation | CVUT | mock-up | +| [ASR](#asr) | asr | BIU | mock-up | | [Depth estimation from monocular](#depthestimationfrommonocular) | depthestimationfrommonocular | UNITN | mock-up | -| [Scene understanding](#sceneunderstanding) | sceneunderstanding | HWU | mock-up | -| [Occupancy map](#occupancymap) | occupancymap | CVUT | mock-up | -| [dialogue arbiter](#dialoguearbiter) | dialoguearbiter | HWU | mock-up | -| [People 3D tracker](#people3dtracker) | people3dtracker | INRIA | mock-up | -| [Speaker identification](#speakeridentification) | speakeridentification | BIU | mock-up | -| [Person re-identification](#personreidentification) | personreidentification | UNITN | mock-up | -| [ORB SLAM](#orbslam) | orbslam | PAL | mock-up | +| [Multi-people body tracker](#multipeoplebodytracker) | multipeoplebodytracker | INRIA | mock-up | | [hri_fullbody](#hri_fullbody) | hri_fullbody | PAL | mock-up | +| [Sound source localisation](#soundsourcelocalisation) | soundsourcelocalisation | BIU | mock-up | | [RGB head](#rgbhead) | rgbhead | PAL | mock-up | -| [Multi-people body tracker](#multipeoplebodytracker) | multipeoplebodytracker | INRIA | mock-up | -| [RGB-D + camera_info](#rgbdcamera_info) | rgbdcamera_info | PAL | mock-up | | [Activity reco](#activityreco) | activityreco | UNITN | mock-up | | [Face detection](#facedetection) | facedetection | UNITN | mock-up | | [ spring_msgs](#spring_msgs) | spring_msgs | Other | released (version 0.0.2) (dependency) | +| [Robot GUI](#robotgui) | robotgui | ERM | mock-up | +| [respeaker_ros](#respeaker_ros) | respeaker_ros | PAL | released (version master) | +| [Person re-identification](#personreidentification) | personreidentification | UNITN | mock-up | +| [ORB SLAM](#orbslam) | orbslam | PAL | mock-up | +| [People 3D tracker](#people3dtracker) | people3dtracker | INRIA | mock-up | +| [Fisheye](#fisheye) | fisheye | PAL | mock-up | +| [F-formation](#fformation) | fformation | UNITN | mock-up | +| [Robot functional layer](#robotfunctionallayer) | robotfunctionallayer | PAL | mock-up | +| [robot_behavior](#robot_behavior) | robot_behavior | INRIA | mock-up | +| [User attention estimation](#userattentionestimation) | userattentionestimation | UNITN | mock-up | +| [RGB-D + camera_info](#rgbdcamera_info) | rgbdcamera_info | PAL | mock-up | +| [Non-verbal behaviours](#nonverbalbehaviours) | nonverbalbehaviours | UNITN | mock-up | +| [Speaker identification](#speakeridentification) | speakeridentification | BIU | mock-up | | [Mask detection](#maskdetection) | maskdetection | UNITN | mock-up | +| [Semantic mapping](#semanticmapping) | semanticmapping | CVUT | mock-up | +| [Speech analysis](#speechanalysis) | speechanalysis | BIU | mock-up | +| [Soft biometrics](#softbiometrics) | softbiometrics | UNITN | mock-up | +| [Occupancy map](#occupancymap) | occupancymap | CVUT | mock-up | +| [dialogue arbiter](#dialoguearbiter) | dialoguearbiter | HWU | mock-up | | [plan_actions](#plan_actions) | plan_actions | HWU | mock-up | -| [Person manager](#personmanager) | personmanager | INRIA | mock-up | -| [Non-verbal behaviours](#nonverbalbehaviours) | nonverbalbehaviours | UNITN | mock-up | | [User visual focus](#uservisualfocus) | uservisualfocus | UNITN | mock-up | -| [Speech analysis](#speechanalysis) | speechanalysis | BIU | mock-up | +| [ hri_msgs](#hri_msgs) | hri_msgs | Other | released (version 0.1.1) (dependency) | +| [Speech synthesis](#speechsynthesis) | speechsynthesis | PAL | mock-up | +| [Scene understanding](#sceneunderstanding) | sceneunderstanding | HWU | mock-up | +| [ROS openpose](#rosopenpose) | rosopenpose | INRIA | mock-up | | [Visual SLAM + 3D map](#visualslam3dmap) | visualslam3dmap | CVUT | mock-up | -| [Semantic mapping](#semanticmapping) | semanticmapping | CVUT | mock-up | -| [Fisheye](#fisheye) | fisheye | PAL | mock-up | -| [F-formation](#fformation) | fformation | UNITN | mock-up | -| [respeaker_ros](#respeaker_ros) | respeaker_ros | PAL | released (version master) | -| [ASR](#asr) | asr | BIU | mock-up | +| [Object detection/identification/localisation](#objectdetectionidentificationlocalisation) | objectdetectionidentificationlocalisation | CVUT | mock-up | | [Body 2D pose](#body2dpose) | body2dpose | UNITN | mock-up | +| [interaction_manager](#interaction_manager) | interaction_manager | HWU | mock-up | +| [Person manager](#personmanager) | personmanager | INRIA | mock-up | +| [Voice speech matching](#voicespeechmatching) | voicespeechmatching | BIU | mock-up | ## Detailed description @@ -55,13 +55,11 @@ EU H2020 SPRING architecture --- -### robot_behavior - -Node *robot_behavior* (id: `robot_behavior`) is overseen by INRIA. +### asr -REPO: https://gitlab.inria.fr/spring/wp6_robot_behavior/robot_behavior/-/tree/devel/src/robot_behavior +Node *ASR* (id: `asr`) is overseen by BIU. -The code is primarily developed at INRIA by Timothée Wintz. +REPO [TBC]: https://gitlab.inria.fr/spring/wp5_spoken_conversations/asr #### Status @@ -69,26 +67,21 @@ The code is primarily developed at INRIA by Timothée Wintz. #### Inputs/outputs - - Input: `status` (undefined) - - Input: `look at` (undefined) - - Input: `following/nav goals` (undefined) - - Input: `occupancy map` (undefined) - - Input: `/h/i/groups [hri_msgs/GroupsStamped]` (topic) + - Input: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) - - Output: `status` (undefined) - - Output: `low-level actions` (undefined) + - Output: `/audio/speech_streams -- array of std_msgs/String` (undefined) #### Dependencies +- `audio_common_msgs/AudioData` - `std_msgs/Empty` -- `hri_msgs/GroupsStamped` --- -### soundsourcelocalisation +### depthestimationfrommonocular -Node *Sound source localisation* (id: `soundsourcelocalisation`) is overseen by BIU. +Node *Depth estimation from monocular* (id: `depthestimationfrommonocular`) is overseen by UNITN. @@ -98,27 +91,24 @@ Node *Sound source localisation* (id: `soundsourcelocalisation`) is overseen by #### Inputs/outputs - - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) + - Input: `input` (undefined) - - Output: `output` (undefined) - - Output: `source angle` (undefined) - - Output: `tracking information` (undefined) - - Output: `tf: /voice_id` (tf) + - Output: `depth` (undefined) #### Dependencies - `std_msgs/Empty` -- `tf/transform_broadcaster` -- `audio_common_msgs/AudioData` --- -### robotfunctionallayer +### multipeoplebodytracker -Node *Robot functional layer* (id: `robotfunctionallayer`) is overseen by PAL. +Node *Multi-people body tracker* (id: `multipeoplebodytracker`) is overseen by INRIA. +REPO (Docker): https://gitlab.inria.fr/spring/wp3_av_perception/docker-tracking +This code is primarily developed at INRIA by Luis Gomez Camara. #### Status @@ -126,19 +116,21 @@ Node *Robot functional layer* (id: `robotfunctionallayer`) is overseen by PAL. #### Inputs/outputs - - Input: `input` (undefined) + - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Output: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) #### Dependencies -- `std_msgs/Empty` +- `sensor_msgs/Image` +- `hri_msgs/RegionOfInterestStamped` --- -### voicespeechmatching +### hri_fullbody -Node *Voice speech matching* (id: `voicespeechmatching`) is overseen by BIU. +Node *hri_fullbody* (id: `hri_fullbody`) is overseen by PAL. @@ -148,23 +140,26 @@ Node *Voice speech matching* (id: `voicespeechmatching`) is overseen by BIU. #### Inputs/outputs - - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) - - Input: `/audio/speech_streams [array of std_msgs/String]` (topic) + - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Input: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Output: `/h/v/*/speech [std_msgs/String]` (topic) + - Output: `tf: /full_body` (tf) + - Output: `/h/b/*/skeleton2d [hri_msgs/Skeleton2D]` (topic) + - Output: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) #### Dependencies -- `std_msgs/String` -- `audio_common_msgs/AudioData` -- `array of std_msgs/String` +- `tf/transform_broadcaster` +- `hri_msgs/Skeleton2D` +- `sensor_msgs/Image` +- `hri_msgs/RegionOfInterestStamped` --- -### softbiometrics +### soundsourcelocalisation -Node *Soft biometrics* (id: `softbiometrics`) is overseen by UNITN. +Node *Sound source localisation* (id: `soundsourcelocalisation`) is overseen by BIU. @@ -174,21 +169,25 @@ Node *Soft biometrics* (id: `softbiometrics`) is overseen by UNITN. #### Inputs/outputs - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) - - Output: `/h/f/*/demographics [hri_msgs/AgeAndGender]` (topic) + - Output: `source angle` (undefined) + - Output: `tracking information` (undefined) + - Output: `tf: /voice_id` (tf) + - Output: `output` (undefined) #### Dependencies -- `hri_msgs/AgeAndGender` -- `hri_msgs/RegionOfInterestStamped` +- `std_msgs/Empty` +- `tf/transform_broadcaster` +- `audio_common_msgs/AudioData` --- -### userattentionestimation +### rgbhead -Node *User attention estimation* (id: `userattentionestimation`) is overseen by UNITN. +Node *RGB head* (id: `rgbhead`) is overseen by PAL. @@ -198,24 +197,19 @@ Node *User attention estimation* (id: `userattentionestimation`) is overseen by #### Inputs/outputs - - Input: `TF (faces)` (undefined) - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Output: `tf: /face_id_gaze` (tf) - - Output: `x,y + attention heatmap` (undefined) + - Output: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) #### Dependencies -- `std_msgs/Empty` -- `tf/transform_broadcaster` -- `hri_msgs/RegionOfInterestStamped` +- `sensor_msgs/Image` --- -### speechsynthesis +### activityreco -Node *Speech synthesis* (id: `speechsynthesis`) is overseen by PAL. +Node *Activity reco* (id: `activityreco`) is overseen by UNITN. @@ -225,9 +219,10 @@ Node *Speech synthesis* (id: `speechsynthesis`) is overseen by PAL. #### Inputs/outputs - - Input: `speech [std_msgs/String]` (undefined) + - Input: `gaze direction` (undefined) + - Input: `TF (bodies)` (undefined) - - Output: `/tts/feedback` (undefined) + - Output: `[?] output` (undefined) #### Dependencies @@ -236,9 +231,9 @@ Node *Speech synthesis* (id: `speechsynthesis`) is overseen by PAL. --- -### rosopenpose +### facedetection -Node *ROS openpose* (id: `rosopenpose`) is overseen by INRIA. +Node *Face detection* (id: `facedetection`) is overseen by UNITN. @@ -248,13 +243,14 @@ Node *ROS openpose* (id: `rosopenpose`) is overseen by INRIA. #### Inputs/outputs - - Input: `input` (undefined) + - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `output` (undefined) + - Output: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) #### Dependencies -- `std_msgs/Empty` +- `sensor_msgs/Image` +- `hri_msgs/RegionOfInterestStamped` --- @@ -285,41 +281,34 @@ Node *Robot GUI* (id: `robotgui`) is overseen by ERM. --- -### interaction_manager +### respeaker_ros -Node *interaction_manager* (id: `interaction_manager`) is overseen by HWU. +Node *respeaker_ros* (id: `respeaker_ros`) is overseen by PAL. -REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/interaction/-/tree/spring_dev/interaction_manager +REPO:git@gitlab.inria.fr:spring/wp7_ari/respeaker_ros.git +BIN:respeaker_multichan_node.py #### Status -**This node is currently auto-generated (mock-up)** +**Current release: master** #### Inputs/outputs - - Input: `/h/p/personID` (undefined) - - Input: `semantic scene description` (undefined) - - Input: `input` (undefined) - - Input: `dialogue state` (undefined) - - Input: `robot state` (undefined) - - Input: `TF` (undefined) - - Output: `gestures` (undefined) - - Output: `who to look at` (undefined) - - Output: `verbal command` (undefined) - - Output: `nav goals` (undefined) - - Output: `active personID` (undefined) + - Output: `/audio/raw_audio [respeaker_ros/RawAudioData]` (topic) + - Output: `/audio/ego_audio [audio_common_msgs/AudioData]` (topic) #### Dependencies -- `std_msgs/Empty` +- `respeaker_ros/RawAudioData` +- `audio_common_msgs/AudioData` --- -### objectdetectionidentificationlocalisation +### personreidentification -Node *Object detection/identification/localisation* (id: `objectdetectionidentificationlocalisation`) is overseen by CVUT. +Node *Person re-identification* (id: `personreidentification`) is overseen by UNITN. @@ -329,21 +318,21 @@ Node *Object detection/identification/localisation* (id: `objectdetectionidentif #### Inputs/outputs - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Output: `/detected_objects [spring_msgs/DetectedObjectArray]` (topic) + - Output: `person_id_candidate [hri_msgs/IdsMatch]` (undefined) #### Dependencies -- `spring_msgs/DetectedObjectArray` -- `sensor_msgs/Image` +- `std_msgs/Empty` +- `hri_msgs/RegionOfInterestStamped` --- -### depthestimationfrommonocular +### orbslam -Node *Depth estimation from monocular* (id: `depthestimationfrommonocular`) is overseen by UNITN. +Node *ORB SLAM* (id: `orbslam`) is overseen by PAL. @@ -353,20 +342,23 @@ Node *Depth estimation from monocular* (id: `depthestimationfrommonocular`) is o #### Inputs/outputs - - Input: `input` (undefined) + - Input: `/camera_torso/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `depth` (undefined) + - Output: `tf: /odom` (tf) + - Output: `/map [nav_msgs/OccupancyGrid]` (topic) #### Dependencies -- `std_msgs/Empty` +- `tf/transform_broadcaster` +- `sensor_msgs/Image` +- `nav_msgs/OccupancyGrid` --- -### sceneunderstanding +### people3dtracker -Node *Scene understanding* (id: `sceneunderstanding`) is overseen by HWU. +Node *People 3D tracker* (id: `people3dtracker`) is overseen by INRIA. @@ -376,20 +368,23 @@ Node *Scene understanding* (id: `sceneunderstanding`) is overseen by HWU. #### Inputs/outputs - - Input: `scene graph` (undefined) + - Input: `feet position` (undefined) + - Input: `people RoIs` (undefined) + - Input: `ground plane` (undefined) - - Output: `semantic description` (undefined) + - Output: `tf: /body_id` (tf) #### Dependencies - `std_msgs/Empty` +- `tf/transform_broadcaster` --- -### occupancymap +### fisheye -Node *Occupancy map* (id: `occupancymap`) is overseen by CVUT. +Node *Fisheye* (id: `fisheye`) is overseen by PAL. @@ -399,24 +394,21 @@ Node *Occupancy map* (id: `occupancymap`) is overseen by CVUT. #### Inputs/outputs - - Input: `TF (bodies)` (undefined) - - Input: `dense 3d map` (undefined) - - Output: `/map_refined [nav_msgs/OccupancyGrid]` (topic) + - Output: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) #### Dependencies -- `std_msgs/Empty` -- `nav_msgs/OccupancyGrid` +- `sensor_msgs/Image` --- -### dialoguearbiter +### fformation + +Node *F-formation* (id: `fformation`) is overseen by UNITN. -Node *dialogue arbiter* (id: `dialoguearbiter`) is overseen by HWU. -REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/dialogue/-/tree/spring-dev/dialogue_arbiter #### Status @@ -424,23 +416,23 @@ REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/dialogue/-/tree/sp #### Inputs/outputs - - Input: `interaction messages` (undefined) - - Input: `/h/v/*/speech [std_msgs/String]` (topic) + - Input: `tf: /person_id` (tf) + - Input: `/h/i/gaze [hri_msgs/GazesStamped]` (topic) - - Output: `DialogueState` (undefined) - - Output: `next utterance` (undefined) + - Output: `/h/i/groups [hri_msgs/GroupsStamped]` (topic) #### Dependencies -- `std_msgs/Empty` -- `std_msgs/String` +- `tf/transform_listener` +- `hri_msgs/GazesStamped` +- `hri_msgs/GroupsStamped` --- -### people3dtracker +### robotfunctionallayer -Node *People 3D tracker* (id: `people3dtracker`) is overseen by INRIA. +Node *Robot functional layer* (id: `robotfunctionallayer`) is overseen by PAL. @@ -450,26 +442,23 @@ Node *People 3D tracker* (id: `people3dtracker`) is overseen by INRIA. #### Inputs/outputs - - Input: `feet position` (undefined) - - Input: `people RoIs` (undefined) - - Input: `ground plane` (undefined) + - Input: `input` (undefined) - - Output: `tf: /body_id` (tf) #### Dependencies - `std_msgs/Empty` -- `tf/transform_broadcaster` --- -### speakeridentification +### robot_behavior -Node *Speaker identification* (id: `speakeridentification`) is overseen by BIU. +Node *robot_behavior* (id: `robot_behavior`) is overseen by INRIA. -- online services -- not started yet +REPO: https://gitlab.inria.fr/spring/wp6_robot_behavior/robot_behavior/-/tree/devel/src/robot_behavior + +The code is primarily developed at INRIA by Timothée Wintz. #### Status @@ -477,23 +466,26 @@ Node *Speaker identification* (id: `speakeridentification`) is overseen by BIU. #### Inputs/outputs - - Input: `tracking information` (undefined) - - Input: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) + - Input: `look at` (undefined) + - Input: `following/nav goals` (undefined) + - Input: `occupancy map` (undefined) + - Input: `status` (undefined) + - Input: `/h/i/groups [hri_msgs/GroupsStamped]` (topic) - - Output: `person_id_candidate [hri_msgs/IdsMatch]` (undefined) - - Output: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) + - Output: `status` (undefined) + - Output: `low-level actions` (undefined) #### Dependencies - `std_msgs/Empty` -- `audio_common_msgs/AudioData` +- `hri_msgs/GroupsStamped` --- -### personreidentification +### userattentionestimation -Node *Person re-identification* (id: `personreidentification`) is overseen by UNITN. +Node *User attention estimation* (id: `userattentionestimation`) is overseen by UNITN. @@ -503,21 +495,24 @@ Node *Person re-identification* (id: `personreidentification`) is overseen by UN #### Inputs/outputs + - Input: `TF (faces)` (undefined) - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Output: `person_id_candidate [hri_msgs/IdsMatch]` (undefined) + - Output: `tf: /face_id_gaze` (tf) + - Output: `x,y + attention heatmap` (undefined) #### Dependencies +- `tf/transform_broadcaster` - `std_msgs/Empty` - `hri_msgs/RegionOfInterestStamped` --- -### orbslam +### rgbdcamera_info -Node *ORB SLAM* (id: `orbslam`) is overseen by PAL. +Node *RGB-D + camera_info* (id: `rgbdcamera_info`) is overseen by PAL. @@ -527,23 +522,19 @@ Node *ORB SLAM* (id: `orbslam`) is overseen by PAL. #### Inputs/outputs - - Input: `/camera_torso/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `tf: /odom` (tf) - - Output: `/map [nav_msgs/OccupancyGrid]` (topic) + - Output: `/camera_torso/color/image_raw [sensor_msgs/Image]` (topic) #### Dependencies -- `tf/transform_broadcaster` -- `nav_msgs/OccupancyGrid` - `sensor_msgs/Image` --- -### hri_fullbody +### nonverbalbehaviours -Node *hri_fullbody* (id: `hri_fullbody`) is overseen by PAL. +Node *Non-verbal behaviours* (id: `nonverbalbehaviours`) is overseen by UNITN. @@ -553,28 +544,26 @@ Node *hri_fullbody* (id: `hri_fullbody`) is overseen by PAL. #### Inputs/outputs - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Input: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) + - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Output: `/h/b/*/skeleton2d [hri_msgs/Skeleton2D]` (topic) - - Output: `tf: /full_body` (tf) - - Output: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Output: `/h/f/*/expression [hri_msgs/Expression]` (topic) #### Dependencies -- `hri_msgs/Skeleton2D` -- `tf/transform_broadcaster` -- `sensor_msgs/Image` +- `hri_msgs/Expression` +- `audio_common_msgs/AudioData` - `hri_msgs/RegionOfInterestStamped` --- -### rgbhead - -Node *RGB head* (id: `rgbhead`) is overseen by PAL. +### speakeridentification +Node *Speaker identification* (id: `speakeridentification`) is overseen by BIU. +- online services +- not started yet #### Status @@ -582,23 +571,25 @@ Node *RGB head* (id: `rgbhead`) is overseen by PAL. #### Inputs/outputs + - Input: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) + - Input: `tracking information` (undefined) - - Output: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Output: `person_id_candidate [hri_msgs/IdsMatch]` (undefined) + - Output: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) #### Dependencies -- `sensor_msgs/Image` +- `audio_common_msgs/AudioData` +- `std_msgs/Empty` --- -### multipeoplebodytracker +### maskdetection -Node *Multi-people body tracker* (id: `multipeoplebodytracker`) is overseen by INRIA. +Node *Mask detection* (id: `maskdetection`) is overseen by UNITN. -REPO (Docker): https://gitlab.inria.fr/spring/wp3_av_perception/docker-tracking -This code is primarily developed at INRIA by Luis Gomez Camara. #### Status @@ -606,23 +597,53 @@ This code is primarily developed at INRIA by Luis Gomez Camara. #### Inputs/outputs - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Output: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Output: `/h/f/*/mask [std_msgs/Bool]` (topic) #### Dependencies -- `sensor_msgs/Image` +- `std_msgs/Bool` - `hri_msgs/RegionOfInterestStamped` --- -### rgbdcamera_info +### semanticmapping + +Node *Semantic mapping* (id: `semanticmapping`) is overseen by CVUT. + + + +#### Status + +**This node is currently auto-generated (mock-up)** + +#### Inputs/outputs + + - Input: `dense 3d map` (undefined) + - Input: `/detected_objects [spring_msgs/DetectedObjectArray]` (topic) + + - Output: `scene graph` (undefined) + +#### Dependencies + +- `std_msgs/Empty` +- `spring_msgs/DetectedObjectArray` + + +--- + +### speechanalysis -Node *RGB-D + camera_info* (id: `rgbdcamera_info`) is overseen by PAL. +Node *Speech analysis* (id: `speechanalysis`) is overseen by BIU. +This node does: +- speech echo cancelation, +- speech enhancement, +- speech separation and diarization +REPO: https://gitlab.inria.fr/spring/wp3_av_perception/biu-ros_audio_retreiver #### Status @@ -630,19 +651,24 @@ Node *RGB-D + camera_info* (id: `rgbdcamera_info`) is overseen by PAL. #### Inputs/outputs + - Input: `sound localization` (undefined) + - Input: `/audio/raw_audio [respeaker_ros/RawAudioData]` (topic) + - Input: `/audio/ego_audio [audio_common_msgs/AudioData]` (topic) - - Output: `/camera_torso/color/image_raw [sensor_msgs/Image]` (topic) + - Output: `/audio/postprocess_audio_streams -- cf output in node desc` (undefined) #### Dependencies -- `sensor_msgs/Image` +- `std_msgs/Empty` +- `respeaker_ros/RawAudioData` +- `audio_common_msgs/AudioData` --- -### activityreco +### softbiometrics -Node *Activity reco* (id: `activityreco`) is overseen by UNITN. +Node *Soft biometrics* (id: `softbiometrics`) is overseen by UNITN. @@ -652,21 +678,21 @@ Node *Activity reco* (id: `activityreco`) is overseen by UNITN. #### Inputs/outputs - - Input: `gaze direction` (undefined) - - Input: `TF (bodies)` (undefined) + - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Output: `[?] output` (undefined) + - Output: `/h/f/*/demographics [hri_msgs/AgeAndGender]` (topic) #### Dependencies -- `std_msgs/Empty` +- `hri_msgs/AgeAndGender` +- `hri_msgs/RegionOfInterestStamped` --- -### facedetection +### occupancymap -Node *Face detection* (id: `facedetection`) is overseen by UNITN. +Node *Occupancy map* (id: `occupancymap`) is overseen by CVUT. @@ -676,23 +702,24 @@ Node *Face detection* (id: `facedetection`) is overseen by UNITN. #### Inputs/outputs - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Input: `TF (bodies)` (undefined) + - Input: `dense 3d map` (undefined) - - Output: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Output: `/map_refined [nav_msgs/OccupancyGrid]` (topic) #### Dependencies -- `sensor_msgs/Image` -- `hri_msgs/RegionOfInterestStamped` +- `std_msgs/Empty` +- `nav_msgs/OccupancyGrid` --- -### maskdetection - -Node *Mask detection* (id: `maskdetection`) is overseen by UNITN. +### dialoguearbiter +Node *dialogue arbiter* (id: `dialoguearbiter`) is overseen by HWU. +REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/dialogue/-/tree/spring-dev/dialogue_arbiter #### Status @@ -700,14 +727,16 @@ Node *Mask detection* (id: `maskdetection`) is overseen by UNITN. #### Inputs/outputs - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `interaction messages` (undefined) + - Input: `/h/v/*/speech [std_msgs/String]` (topic) - - Output: `/h/f/*/mask [std_msgs/Bool]` (topic) + - Output: `next utterance` (undefined) + - Output: `DialogueState` (undefined) #### Dependencies -- `hri_msgs/RegionOfInterestStamped` -- `std_msgs/Bool` +- `std_msgs/Empty` +- `std_msgs/String` --- @@ -724,10 +753,10 @@ REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/plan_actions #### Inputs/outputs - - Input: `dialogue state` (undefined) - Input: `TF (persons)` (undefined) - Input: `semantic scene description` (undefined) - Input: `demographics` (undefined) + - Input: `dialogue state` (undefined) - Output: `interaction state` (undefined) - Output: `output` (undefined) @@ -740,11 +769,11 @@ REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/plan_actions --- -### personmanager +### uservisualfocus + +Node *User visual focus* (id: `uservisualfocus`) is overseen by UNITN. -Node *Person manager* (id: `personmanager`) is overseen by INRIA. -PAL to write an initial temporary version, simply forwarding detected faces as persons. #### Status @@ -752,30 +781,25 @@ PAL to write an initial temporary version, simply forwarding detected faces as p #### Inputs/outputs - - Input: `TF (voices)` (undefined) - - Input: `TF (faces)` (undefined) - - Input: `candidate_matchs [hri_msgs/IdsMatch]` (undefined) - - Input: `/h/f/*/demographics [hri_msgs/AgeAndGender]` (topic) + - Input: `attention` (undefined) + - Input: `scene` (undefined) + - Input: `gaze direction` (undefined) + - Input: `depth` (undefined) - - Output: `/h/p/*/face_id [std_msgs/String]` (topic) - - Output: `tf: /person_id` (tf) - - Output: `/h/p/*/voice_id [std_msgs/String]` (topic) - - Output: `/h/p/*/demographics [hri_msgs/AgeAndGender]` (topic) - - Output: `/humans/persons/*/body_id [std_msgs/String]` (topic) + - Output: `who's looking at what?` (undefined) + - Output: `/h/i/gaze [hri_msgs/GazesStamped]` (topic) #### Dependencies -- `std_msgs/String` -- `tf/transform_broadcaster` - `std_msgs/Empty` -- `hri_msgs/AgeAndGender` +- `hri_msgs/GazesStamped` --- -### nonverbalbehaviours +### speechsynthesis -Node *Non-verbal behaviours* (id: `nonverbalbehaviours`) is overseen by UNITN. +Node *Speech synthesis* (id: `speechsynthesis`) is overseen by PAL. @@ -785,23 +809,20 @@ Node *Non-verbal behaviours* (id: `nonverbalbehaviours`) is overseen by UNITN. #### Inputs/outputs - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) + - Input: `speech [std_msgs/String]` (undefined) - - Output: `/h/f/*/expression [hri_msgs/Expression]` (topic) + - Output: `/tts/feedback` (undefined) #### Dependencies -- `hri_msgs/Expression` -- `hri_msgs/RegionOfInterestStamped` -- `audio_common_msgs/AudioData` +- `std_msgs/Empty` --- -### uservisualfocus +### sceneunderstanding -Node *User visual focus* (id: `uservisualfocus`) is overseen by UNITN. +Node *Scene understanding* (id: `sceneunderstanding`) is overseen by HWU. @@ -811,32 +832,22 @@ Node *User visual focus* (id: `uservisualfocus`) is overseen by UNITN. #### Inputs/outputs - - Input: `gaze direction` (undefined) - - Input: `attention` (undefined) - - Input: `depth` (undefined) - - Input: `scene` (undefined) + - Input: `scene graph` (undefined) - - Output: `who's looking at what?` (undefined) - - Output: `/h/i/gaze [hri_msgs/GazesStamped]` (topic) + - Output: `semantic description` (undefined) #### Dependencies - `std_msgs/Empty` -- `hri_msgs/GazesStamped` --- -### speechanalysis +### rosopenpose -Node *Speech analysis* (id: `speechanalysis`) is overseen by BIU. +Node *ROS openpose* (id: `rosopenpose`) is overseen by INRIA. -This node does: -- speech echo cancelation, -- speech enhancement, -- speech separation and diarization -REPO: https://gitlab.inria.fr/spring/wp3_av_perception/biu-ros_audio_retreiver #### Status @@ -844,17 +855,13 @@ REPO: https://gitlab.inria.fr/spring/wp3_av_perception/biu-ros_audio_retreiver #### Inputs/outputs - - Input: `/audio/raw_audio [respeaker_ros/RawAudioData]` (topic) - - Input: `/audio/ego_audio [audio_common_msgs/AudioData]` (topic) - - Input: `sound localization` (undefined) + - Input: `input` (undefined) - - Output: `/audio/postprocess_audio_streams -- cf output in node desc` (undefined) + - Output: `output` (undefined) #### Dependencies - `std_msgs/Empty` -- `respeaker_ros/RawAudioData` -- `audio_common_msgs/AudioData` --- @@ -880,16 +887,16 @@ Node *Visual SLAM + 3D map* (id: `visualslam3dmap`) is overseen by CVUT. #### Dependencies +- `sensor_msgs/Image` - `std_msgs/Empty` - `tf/transform_broadcaster` -- `sensor_msgs/Image` --- -### semanticmapping +### objectdetectionidentificationlocalisation -Node *Semantic mapping* (id: `semanticmapping`) is overseen by CVUT. +Node *Object detection/identification/localisation* (id: `objectdetectionidentificationlocalisation`) is overseen by CVUT. @@ -899,24 +906,23 @@ Node *Semantic mapping* (id: `semanticmapping`) is overseen by CVUT. #### Inputs/outputs - - Input: `/detected_objects [spring_msgs/DetectedObjectArray]` (topic) - - Input: `dense 3d map` (undefined) + - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `scene graph` (undefined) + - Output: `/detected_objects [spring_msgs/DetectedObjectArray]` (topic) #### Dependencies +- `sensor_msgs/Image` - `spring_msgs/DetectedObjectArray` -- `std_msgs/Empty` --- -### fisheye - -Node *Fisheye* (id: `fisheye`) is overseen by PAL. +### body2dpose +Node *Body 2D pose* (id: `body2dpose`) is overseen by UNITN. +MOCK: Body 2D pose [openpose] is... #### Status @@ -924,21 +930,25 @@ Node *Fisheye* (id: `fisheye`) is overseen by PAL. #### Inputs/outputs + - Input: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Output: `/h/b/*/skeleton2d [hri_msgs/Skeleton2D]` (topic) #### Dependencies +- `hri_msgs/RegionOfInterestStamped` - `sensor_msgs/Image` +- `hri_msgs/Skeleton2D` --- -### fformation - -Node *F-formation* (id: `fformation`) is overseen by UNITN. +### interaction_manager +Node *interaction_manager* (id: `interaction_manager`) is overseen by HWU. +REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/interaction/-/tree/spring_dev/interaction_manager #### Status @@ -946,49 +956,31 @@ Node *F-formation* (id: `fformation`) is overseen by UNITN. #### Inputs/outputs - - Input: `/h/i/gaze [hri_msgs/GazesStamped]` (topic) - - Input: `tf: /person_id` (tf) - - - Output: `/h/i/groups [hri_msgs/GroupsStamped]` (topic) - -#### Dependencies - -- `hri_msgs/GazesStamped` -- `tf/transform_listener` -- `hri_msgs/GroupsStamped` - - ---- - -### respeaker_ros - -Node *respeaker_ros* (id: `respeaker_ros`) is overseen by PAL. - -REPO:git@gitlab.inria.fr:spring/wp7_ari/respeaker_ros.git BIN:respeaker_multichan_node.py - -#### Status - -**Current release: master** - -#### Inputs/outputs - + - Input: `semantic scene description` (undefined) + - Input: `input` (undefined) + - Input: `dialogue state` (undefined) + - Input: `robot state` (undefined) + - Input: `TF` (undefined) + - Input: `/h/p/personID` (undefined) - - Output: `/audio/raw_audio [respeaker_ros/RawAudioData]` (topic) - - Output: `/audio/ego_audio [audio_common_msgs/AudioData]` (topic) + - Output: `who to look at` (undefined) + - Output: `verbal command` (undefined) + - Output: `nav goals` (undefined) + - Output: `active personID` (undefined) + - Output: `gestures` (undefined) #### Dependencies -- `respeaker_ros/RawAudioData` -- `audio_common_msgs/AudioData` +- `std_msgs/Empty` --- -### asr +### personmanager -Node *ASR* (id: `asr`) is overseen by BIU. +Node *Person manager* (id: `personmanager`) is overseen by INRIA. -REPO [TBC]: https://gitlab.inria.fr/spring/wp5_spoken_conversations/asr +PAL to write an initial temporary version, simply forwarding detected faces as persons. #### Status @@ -996,23 +988,32 @@ REPO [TBC]: https://gitlab.inria.fr/spring/wp5_spoken_conversations/asr #### Inputs/outputs - - Input: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) + - Input: `TF (voices)` (undefined) + - Input: `TF (faces)` (undefined) + - Input: `candidate_matchs [hri_msgs/IdsMatch]` (undefined) + - Input: `/h/f/*/demographics [hri_msgs/AgeAndGender]` (topic) - - Output: `/audio/speech_streams -- array of std_msgs/String` (undefined) + - Output: `tf: /person_id` (tf) + - Output: `/humans/persons/*/body_id [std_msgs/String]` (topic) + - Output: `/h/p/*/voice_id [std_msgs/String]` (topic) + - Output: `/h/p/*/demographics [hri_msgs/AgeAndGender]` (topic) + - Output: `/h/p/*/face_id [std_msgs/String]` (topic) #### Dependencies -- `audio_common_msgs/AudioData` +- `tf/transform_broadcaster` +- `std_msgs/String` - `std_msgs/Empty` +- `hri_msgs/AgeAndGender` --- -### body2dpose +### voicespeechmatching + +Node *Voice speech matching* (id: `voicespeechmatching`) is overseen by BIU. -Node *Body 2D pose* (id: `body2dpose`) is overseen by UNITN. -MOCK: Body 2D pose [openpose] is... #### Status @@ -1020,30 +1021,29 @@ MOCK: Body 2D pose [openpose] is... #### Inputs/outputs - - Input: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Input: `/audio/speech_streams [std_msgs/String]` (topic) + - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) - - Output: `/h/b/*/skeleton2d [hri_msgs/Skeleton2D]` (topic) + - Output: `/h/v/*/speech [std_msgs/String]` (topic) #### Dependencies -- `hri_msgs/Skeleton2D` -- `hri_msgs/RegionOfInterestStamped` -- `sensor_msgs/Image` +- `std_msgs/String` +- `audio_common_msgs/AudioData` -### Non-executable dependency: hri_msgs +### Non-executable dependency: spring_msgs -Module hri_msgs (id: `hri_msgs`) is overseen by Other. +Module spring_msgs (id: `spring_msgs`) is overseen by Other. -REPO: git@gitlab:ros4hri/hri_msgs.git +REPO:git@gitlab.inria.fr:spring/wp7_ari/spring_msgs.git NOT EXECUTABLE #### Status -**Current release: 0.1.1** +**Current release: 0.0.2** #### Dependencies @@ -1051,17 +1051,17 @@ NOT EXECUTABLE -### Non-executable dependency: spring_msgs +### Non-executable dependency: hri_msgs -Module spring_msgs (id: `spring_msgs`) is overseen by Other. +Module hri_msgs (id: `hri_msgs`) is overseen by Other. -REPO:git@gitlab.inria.fr:spring/wp7_ari/spring_msgs.git +REPO: git@gitlab:ros4hri/hri_msgs.git NOT EXECUTABLE #### Status -**Current release: 0.0.2** +**Current release: 0.1.1** #### Dependencies