diff --git a/spring-architecture.md b/spring-architecture.md index 6a09adc77450991773ab093a1dce4981d9415dc5..1a11d052be3aa5c500ce4dd1b4eb7c191724211c 100644 --- a/spring-architecture.md +++ b/spring-architecture.md @@ -1,6 +1,6 @@ # SPRING architecture -**Version:** 0.3.2 +**Version:** 0.5.0 EU H2020 SPRING architecture @@ -8,46 +8,51 @@ EU H2020 SPRING architecture | **Node** | **id** | **Partner** | **Status** | |----------|--------|-------------|------------| -| [ASR](#asr) | asr | BIU | mock-up | -| [Depth estimation from monocular](#depthestimationfrommonocular) | depthestimationfrommonocular | UNITN | mock-up | -| [Multi-people body tracker](#multipeoplebodytracker) | multipeoplebodytracker | INRIA | mock-up | -| [hri_fullbody](#hri_fullbody) | hri_fullbody | PAL | mock-up | -| [Sound source localisation](#soundsourcelocalisation) | soundsourcelocalisation | BIU | mock-up | -| [RGB head](#rgbhead) | rgbhead | PAL | mock-up | -| [Activity reco](#activityreco) | activityreco | UNITN | mock-up | -| [Face detection](#facedetection) | facedetection | UNITN | mock-up | -| [ spring_msgs](#spring_msgs) | spring_msgs | Other | released (version 0.0.2) (dependency) | -| [Robot GUI](#robotgui) | robotgui | ERM | mock-up | -| [respeaker_ros](#respeaker_ros) | respeaker_ros | PAL | released (version master) | | [Person re-identification](#personreidentification) | personreidentification | UNITN | mock-up | -| [ORB SLAM](#orbslam) | orbslam | PAL | mock-up | +| [Non-verbal behaviours](#nonverbalbehaviours) | nonverbalbehaviours | UNITN | mock-up | +| [torso_rgbd_camera](#torso_rgbd_camera) | torso_rgbd_camera | PAL | mock-up | +| [ InLoc Server](#inlocserver) | inlocserver | CVUT | mock-up | +| [ spring_msgs](#spring_msgs) | spring_msgs | PAL | released (version 0.0.2) | +| [User attention estimation](#userattentionestimation) | userattentionestimation | UNITN | mock-up | +| [ interaction_manager_msgs](#interaction_manager_msgs) | interaction_manager_msgs | HWU | released (version spring_dev) (dependency) | +| [interaction_manager](#interaction_manager) | interaction_manager | HWU | released (version spring_dev) | +| [ hri_msgs](#hri_msgs) | hri_msgs | PAL | released (version 0.1.1) (dependency) | +| [Object detection/identification/localisation](#objectdetectionidentificationlocalisation) | objectdetectionidentificationlocalisation | CVUT | mock-up | +| [Voice speech matching](#voicespeechmatching) | voicespeechmatching | BIU | mock-up | +| [Speaker identification](#speakeridentification) | speakeridentification | BIU | mock-up | +| [mask_detector](#mask_detector) | mask_detector | UNITN | released (version master) | +| [hri_person_manager](#hri_person_manager) | hri_person_manager | PAL | released (version master) | +| [soft_biometrics_estimator](#soft_biometrics_estimator) | soft_biometrics_estimator | UNITN | released (version master) | +| [ROS mediapipe/openpose](#rosmediapipeopenpose) | rosmediapipeopenpose | INRIA | mock-up | | [People 3D tracker](#people3dtracker) | people3dtracker | INRIA | mock-up | -| [Fisheye](#fisheye) | fisheye | PAL | mock-up | | [F-formation](#fformation) | fformation | UNITN | mock-up | +| [ robot_behaviour_msgs](#robot_behaviour_msgs) | robot_behaviour_msgs | HWU | released (version spring_dev) (dependency) | +| [respeaker_ros](#respeaker_ros) | respeaker_ros | PAL | released (version master) | +| [fisheye](#fisheye) | fisheye | PAL | mock-up | +| [dialogue arbiter](#dialoguearbiter) | dialoguearbiter | HWU | released (version spring_dev) | +| [Face detection](#facedetection) | facedetection | UNITN | mock-up | +| [Depth estimation from monocular](#depthestimationfrommonocular) | depthestimationfrommonocular | UNITN | mock-up | | [Robot functional layer](#robotfunctionallayer) | robotfunctionallayer | PAL | mock-up | -| [robot_behavior](#robot_behavior) | robot_behavior | INRIA | mock-up | -| [User attention estimation](#userattentionestimation) | userattentionestimation | UNITN | mock-up | -| [RGB-D + camera_info](#rgbdcamera_info) | rgbdcamera_info | PAL | mock-up | -| [Non-verbal behaviours](#nonverbalbehaviours) | nonverbalbehaviours | UNITN | mock-up | -| [Speaker identification](#speakeridentification) | speakeridentification | BIU | mock-up | -| [Mask detection](#maskdetection) | maskdetection | UNITN | mock-up | +| [ audio_msgs](#audio_msgs) | audio_msgs | HWU | released (version spring_dev) (dependency) | +| [FairMOT Multi-people body tracker](#fairmotmultipeoplebodytracker) | fairmotmultipeoplebodytracker | INRIA | released (version devel) | +| [MoDE](#mode) | mode | BIU | released (version BIU_dev) | +| [Speech synthesis](#speechsynthesis) | speechsynthesis | PAL | mock-up | +| [ social_scene_msgs](#social_scene_msgs) | social_scene_msgs | HWU | released (version spring_dev) (dependency) | +| [User visual focus](#uservisualfocus) | uservisualfocus | UNITN | mock-up | +| [ORB SLAM](#orbslam) | orbslam | PAL | mock-up | +| [robot_behavior](#robot_behavior) | robot_behavior | INRIA | released (version devel) | | [Semantic mapping](#semanticmapping) | semanticmapping | CVUT | mock-up | -| [Speech analysis](#speechanalysis) | speechanalysis | BIU | mock-up | -| [Soft biometrics](#softbiometrics) | softbiometrics | UNITN | mock-up | +| [social_scene_context_understanding](#social_scene_context_understanding) | social_scene_context_understanding | HWU | released (version spring_dev) | +| [Robot GUI](#robotgui) | robotgui | ERM | mock-up | +| [ros_openpose](#ros_openpose) | ros_openpose | UNITN | released (version 0.0.1) | +| [Activity reco](#activityreco) | activityreco | UNITN | mock-up | +| [ wp4_msgs](#wp4_msgs) | wp4_msgs | UNITN | released (version master) (dependency) | +| [robot_behaviour_plan_actions](#robot_behaviour_plan_actions) | robot_behaviour_plan_actions | HWU | released (version spring_dev) | +| [Multi-person audio tracking](#multipersonaudiotracking) | multipersonaudiotracking | BIU | mock-up | +| [InLoc-ROS](#inlocros) | inlocros | CVUT | mock-up | +| [google_asr](#google_asr) | google_asr | BIU | released (version BIU_dev) | | [Occupancy map](#occupancymap) | occupancymap | CVUT | mock-up | -| [dialogue arbiter](#dialoguearbiter) | dialoguearbiter | HWU | mock-up | -| [plan_actions](#plan_actions) | plan_actions | HWU | mock-up | -| [User visual focus](#uservisualfocus) | uservisualfocus | UNITN | mock-up | -| [ hri_msgs](#hri_msgs) | hri_msgs | Other | released (version 0.1.1) (dependency) | -| [Speech synthesis](#speechsynthesis) | speechsynthesis | PAL | mock-up | -| [Scene understanding](#sceneunderstanding) | sceneunderstanding | HWU | mock-up | -| [ROS openpose](#rosopenpose) | rosopenpose | INRIA | mock-up | -| [Visual SLAM + 3D map](#visualslam3dmap) | visualslam3dmap | CVUT | mock-up | -| [Object detection/identification/localisation](#objectdetectionidentificationlocalisation) | objectdetectionidentificationlocalisation | CVUT | mock-up | -| [Body 2D pose](#body2dpose) | body2dpose | UNITN | mock-up | -| [interaction_manager](#interaction_manager) | interaction_manager | HWU | mock-up | -| [Person manager](#personmanager) | personmanager | INRIA | mock-up | -| [Voice speech matching](#voicespeechmatching) | voicespeechmatching | BIU | mock-up | +| [raspicam](#raspicam) | raspicam | PAL | mock-up | ## Detailed description @@ -55,11 +60,11 @@ EU H2020 SPRING architecture --- -### asr +### personreidentification -Node *ASR* (id: `asr`) is overseen by BIU. +Node *Person re-identification* (id: `personreidentification`) is overseen by UNITN. -REPO [TBC]: https://gitlab.inria.fr/spring/wp5_spoken_conversations/asr +MOCK: Person re-identification is... #### Status @@ -67,21 +72,21 @@ REPO [TBC]: https://gitlab.inria.fr/spring/wp5_spoken_conversations/asr #### Inputs/outputs - - Input: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) + - Input: `/h/f/*/roi [sensor_msgs/RegionOfInterest]` (topic) - - Output: `/audio/speech_streams -- array of std_msgs/String` (undefined) + - Output: `person_id_candidate [hri_msgs/IdsMatch]` (undefined) #### Dependencies -- `audio_common_msgs/AudioData` - `std_msgs/Empty` +- `sensor_msgs/RegionOfInterest` --- -### depthestimationfrommonocular +### nonverbalbehaviours -Node *Depth estimation from monocular* (id: `depthestimationfrommonocular`) is overseen by UNITN. +Node *Non-verbal behaviours* (id: `nonverbalbehaviours`) is overseen by UNITN. @@ -91,24 +96,25 @@ Node *Depth estimation from monocular* (id: `depthestimationfrommonocular`) is o #### Inputs/outputs - - Input: `input` (undefined) + - Input: `/h/f/*/roi [hri_msgs/RegionOfInterest]` (topic) + - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) - - Output: `depth` (undefined) + - Output: `/h/f/*/expression [hri_msgs/Expression]` (topic) #### Dependencies -- `std_msgs/Empty` +- `hri_msgs/RegionOfInterest` +- `hri_msgs/Expression` +- `audio_common_msgs/AudioData` --- -### multipeoplebodytracker +### torso_rgbd_camera -Node *Multi-people body tracker* (id: `multipeoplebodytracker`) is overseen by INRIA. +Node *torso_rgbd_camera* (id: `torso_rgbd_camera`) is overseen by PAL. -REPO (Docker): https://gitlab.inria.fr/spring/wp3_av_perception/docker-tracking -This code is primarily developed at INRIA by Luis Gomez Camara. #### Status @@ -116,21 +122,19 @@ This code is primarily developed at INRIA by Luis Gomez Camara. #### Inputs/outputs - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Output: `/torso_rgbd_camera/color/image_raw [sensor_msgs/Image]` (topic) #### Dependencies - `sensor_msgs/Image` -- `hri_msgs/RegionOfInterestStamped` --- -### hri_fullbody +### inlocserver -Node *hri_fullbody* (id: `hri_fullbody`) is overseen by PAL. +Node * InLoc Server* (id: `inlocserver`) is overseen by CVUT. @@ -140,54 +144,37 @@ Node *hri_fullbody* (id: `hri_fullbody`) is overseen by PAL. #### Inputs/outputs - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Input: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Output: `tf: /full_body` (tf) - - Output: `/h/b/*/skeleton2d [hri_msgs/Skeleton2D]` (topic) - - Output: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) #### Dependencies -- `tf/transform_broadcaster` -- `hri_msgs/Skeleton2D` -- `sensor_msgs/Image` -- `hri_msgs/RegionOfInterestStamped` --- -### soundsourcelocalisation - -Node *Sound source localisation* (id: `soundsourcelocalisation`) is overseen by BIU. +### spring_msgs +Node * spring_msgs* (id: `spring_msgs`) is overseen by PAL. +REPO:git@gitlab.inria.fr:spring/wp7_ari/spring_msgs.git NOT EXECUTABLE #### Status -**This node is currently auto-generated (mock-up)** +**Current release: 0.0.2** #### Inputs/outputs - - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) - - Output: `source angle` (undefined) - - Output: `tracking information` (undefined) - - Output: `tf: /voice_id` (tf) - - Output: `output` (undefined) #### Dependencies -- `std_msgs/Empty` -- `tf/transform_broadcaster` -- `audio_common_msgs/AudioData` --- -### rgbhead +### userattentionestimation -Node *RGB head* (id: `rgbhead`) is overseen by PAL. +Node *User attention estimation* (id: `userattentionestimation`) is overseen by UNITN. @@ -197,43 +184,57 @@ Node *RGB head* (id: `rgbhead`) is overseen by PAL. #### Inputs/outputs + - Input: `TF (faces)` (undefined) + - Input: `/h/f/*/roi [hri_msgs/RegionOfInterest]` (topic) - - Output: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Output: `tf: /face_id_gaze` (tf) + - Output: `x,y + attention heatmap` (undefined) #### Dependencies -- `sensor_msgs/Image` +- `tf/transform_broadcaster` +- `std_msgs/Empty` +- `hri_msgs/RegionOfInterest` --- -### activityreco - -Node *Activity reco* (id: `activityreco`) is overseen by UNITN. +### interaction_manager +Node *interaction_manager* (id: `interaction_manager`) is overseen by HWU. +REPO:git@gitlab.inria.fr:spring/wp5_spoken_conversations/interaction.git +SUBFOLDER:interaction_manager #### Status -**This node is currently auto-generated (mock-up)** +**Current release: spring_dev** #### Inputs/outputs - - Input: `gaze direction` (undefined) - - Input: `TF (bodies)` (undefined) + - Input: `semantic scene description` (undefined) + - Input: `/h/p/*/softbiometrics [hri_msgs/Softbiometrics]` (topic) + - Input: `TF` (undefined) + - Input: `robot state` (undefined) + - Input: `dialogue state` (undefined) - - Output: `[?] output` (undefined) + - Output: `verbal command` (undefined) + - Output: `nav goals` (undefined) + - Output: `who to look at` (undefined) + - Output: `active personID` (undefined) + - Output: `gestures` (undefined) #### Dependencies - `std_msgs/Empty` +- `hri_msgs/Softbiometrics` --- -### facedetection +### objectdetectionidentificationlocalisation -Node *Face detection* (id: `facedetection`) is overseen by UNITN. +Node *Object detection/identification/localisation* (id: `objectdetectionidentificationlocalisation`) is overseen by CVUT. @@ -245,19 +246,19 @@ Node *Face detection* (id: `facedetection`) is overseen by UNITN. - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Output: `/detected_objects [spring_msgs/DetectedObjectArray]` (topic) #### Dependencies +- `spring_msgs/DetectedObjectArray` - `sensor_msgs/Image` -- `hri_msgs/RegionOfInterestStamped` --- -### robotgui +### voicespeechmatching -Node *Robot GUI* (id: `robotgui`) is overseen by ERM. +Node *Voice speech matching* (id: `voicespeechmatching`) is overseen by BIU. @@ -267,74 +268,139 @@ Node *Robot GUI* (id: `robotgui`) is overseen by ERM. #### Inputs/outputs - - Input: `additional support material` (undefined) - - Input: `/tts/feedback` (undefined) - - Input: `speech output` (undefined) - - Input: `/h/v/*/speech [std_msgs/String]` (topic) + - Input: `/audio/speech_streams [std_msgs/String]` (topic) + - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) + - Output: `/h/v/*/speech [hri_msgs/LiveSpeech]` (topic) #### Dependencies -- `std_msgs/Empty` +- `hri_msgs/LiveSpeech` - `std_msgs/String` +- `audio_common_msgs/AudioData` --- -### respeaker_ros +### speakeridentification -Node *respeaker_ros* (id: `respeaker_ros`) is overseen by PAL. +Node *Speaker identification* (id: `speakeridentification`) is overseen by BIU. -REPO:git@gitlab.inria.fr:spring/wp7_ari/respeaker_ros.git -BIN:respeaker_multichan_node.py +- online services +- not started yet #### Status -**Current release: master** +**This node is currently auto-generated (mock-up)** #### Inputs/outputs + - Input: `tracking information` (undefined) + - Input: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) - - Output: `/audio/raw_audio [respeaker_ros/RawAudioData]` (topic) - - Output: `/audio/ego_audio [audio_common_msgs/AudioData]` (topic) + - Output: `person_id_candidate [hri_msgs/IdsMatch]` (undefined) + - Output: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) #### Dependencies -- `respeaker_ros/RawAudioData` +- `std_msgs/Empty` - `audio_common_msgs/AudioData` --- -### personreidentification +### mask_detector -Node *Person re-identification* (id: `personreidentification`) is overseen by UNITN. +Node *mask_detector* (id: `mask_detector`) is overseen by UNITN. +Detects presence of a facial mask +REPO:git@gitlab.inria.fr:spring/wp4_behavior/wp4_behavior_understanding.git +SUBFOLDER:wp4_people_characteristics +BIN:mask_detector.py + +#### Status +**Current release: master** + +#### Inputs/outputs + + - Input: `/h/f/*/roi [sensor_msgs/RegionOfInterest]` (topic) + + - Output: `/h/f/*/mask [std_msgs/Bool]` (topic) + +#### Dependencies + +- `sensor_msgs/RegionOfInterest` +- `std_msgs/Bool` + + +--- + +### hri_person_manager + +Node *hri_person_manager* (id: `hri_person_manager`) is overseen by PAL. + +REPO:git@gitlab.inria.fr:spring/wp7_ari/hri_person_manager.git #### Status -**This node is currently auto-generated (mock-up)** +**Current release: master** #### Inputs/outputs - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `TF (faces)` (undefined) + - Input: `/h/f/*/softbiometrics [hri_msgs/Softbiometrics]` (topic) + - Input: `candidate_matches [hri_msgs/IdsMatch]` (undefined) + - Input: `TF (voices)` (undefined) - - Output: `person_id_candidate [hri_msgs/IdsMatch]` (undefined) + - Output: `/h/p/*/voice_id [std_msgs/String]` (topic) + - Output: `/h/p/*/softbiometrics [hri_msgs/Softbiometrics]` (topic) + - Output: `/h/p/*/face_id [std_msgs/String]` (topic) + - Output: `/humans/persons/*/body_id [std_msgs/String]` (topic) + - Output: `tf: /person_id` (tf) #### Dependencies +- `std_msgs/String` - `std_msgs/Empty` -- `hri_msgs/RegionOfInterestStamped` +- `hri_msgs/Softbiometrics` +- `tf/transform_broadcaster` --- -### orbslam +### soft_biometrics_estimator -Node *ORB SLAM* (id: `orbslam`) is overseen by PAL. +Node *soft_biometrics_estimator* (id: `soft_biometrics_estimator`) is overseen by UNITN. + +Detects age/gender +REPO:git@gitlab.inria.fr:spring/wp4_behavior/wp4_behavior_understanding.git +SUBFOLDER:wp4_people_characteristics +BIN:soft_biometrics_estimator.py + +#### Status + +**Current release: master** +#### Inputs/outputs + - Input: `/h/f/*/roi [hri_msgs/RegionOfInterest]` (topic) + + - Output: `/h/f/*/softbiometrics [hri_msgs/Softbiometrics]` (topic) + +#### Dependencies + +- `hri_msgs/RegionOfInterest` +- `hri_msgs/Softbiometrics` + + +--- + +### rosmediapipeopenpose + +Node *ROS mediapipe/openpose* (id: `rosmediapipeopenpose`) is overseen by INRIA. + +MOCK: ROS mediapipe/openpose is... #### Status @@ -342,16 +408,13 @@ Node *ORB SLAM* (id: `orbslam`) is overseen by PAL. #### Inputs/outputs - - Input: `/camera_torso/color/image_raw [sensor_msgs/Image]` (topic) + - Input: `input` (undefined) - - Output: `tf: /odom` (tf) - - Output: `/map [nav_msgs/OccupancyGrid]` (topic) + - Output: `output` (undefined) #### Dependencies -- `tf/transform_broadcaster` -- `sensor_msgs/Image` -- `nav_msgs/OccupancyGrid` +- `std_msgs/Empty` --- @@ -360,7 +423,7 @@ Node *ORB SLAM* (id: `orbslam`) is overseen by PAL. Node *People 3D tracker* (id: `people3dtracker`) is overseen by INRIA. - +MOCK: People 3D tracker [position/orientation] is... #### Status @@ -368,9 +431,9 @@ Node *People 3D tracker* (id: `people3dtracker`) is overseen by INRIA. #### Inputs/outputs - - Input: `feet position` (undefined) - - Input: `people RoIs` (undefined) - Input: `ground plane` (undefined) + - Input: `people RoIs` (undefined) + - Input: `feet position` (undefined) - Output: `tf: /body_id` (tf) @@ -382,9 +445,9 @@ Node *People 3D tracker* (id: `people3dtracker`) is overseen by INRIA. --- -### fisheye +### fformation -Node *Fisheye* (id: `fisheye`) is overseen by PAL. +Node *F-formation* (id: `fformation`) is overseen by UNITN. @@ -394,45 +457,47 @@ Node *Fisheye* (id: `fisheye`) is overseen by PAL. #### Inputs/outputs + - Input: `/h/i/gaze [hri_msgs/Gaze]` (topic) + - Input: `tf: /person_id` (tf) - - Output: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Output: `/h/i/groups [hri_msgs/Group]` (topic) #### Dependencies -- `sensor_msgs/Image` +- `hri_msgs/Group` +- `hri_msgs/Gaze` +- `tf/transform_listener` --- -### fformation - -Node *F-formation* (id: `fformation`) is overseen by UNITN. +### respeaker_ros +Node *respeaker_ros* (id: `respeaker_ros`) is overseen by PAL. +REPO:git@gitlab.inria.fr:spring/wp7_ari/respeaker_ros.git BIN:respeaker_multichan_node.py #### Status -**This node is currently auto-generated (mock-up)** +**Current release: master** #### Inputs/outputs - - Input: `tf: /person_id` (tf) - - Input: `/h/i/gaze [hri_msgs/GazesStamped]` (topic) - - Output: `/h/i/groups [hri_msgs/GroupsStamped]` (topic) + - Output: `/audio/ego_audio [audio_common_msgs/AudioData]` (topic) + - Output: `/audio/raw_audio [respeaker_ros/RawAudioData]` (topic) #### Dependencies -- `tf/transform_listener` -- `hri_msgs/GazesStamped` -- `hri_msgs/GroupsStamped` +- `audio_common_msgs/AudioData` +- `respeaker_ros/RawAudioData` --- -### robotfunctionallayer +### fisheye -Node *Robot functional layer* (id: `robotfunctionallayer`) is overseen by PAL. +Node *fisheye* (id: `fisheye`) is overseen by PAL. @@ -442,23 +507,48 @@ Node *Robot functional layer* (id: `robotfunctionallayer`) is overseen by PAL. #### Inputs/outputs - - Input: `input` (undefined) + - Output: `/torso_front_camera/color/image_raw [sensor_msgs/Image]` (topic) + +#### Dependencies + +- `sensor_msgs/Image` + + +--- + +### dialoguearbiter + +Node *dialogue arbiter* (id: `dialoguearbiter`) is overseen by HWU. + +REPO:git@gitlab.inria.fr:spring/wp5_spoken_conversations/dialogue.git +SUBFOLDER:dialogue_arbiter + +#### Status + +**Current release: spring_dev** + +#### Inputs/outputs + + - Input: `interaction messages` (undefined) + - Input: `/h/v/*/speech [hri_msgs/LiveSpeech]` (topic) + + - Output: `next utterance` (undefined) + - Output: `DialogueState` (undefined) #### Dependencies - `std_msgs/Empty` +- `hri_msgs/LiveSpeech` --- -### robot_behavior +### facedetection -Node *robot_behavior* (id: `robot_behavior`) is overseen by INRIA. +Node *Face detection* (id: `facedetection`) is overseen by UNITN. -REPO: https://gitlab.inria.fr/spring/wp6_robot_behavior/robot_behavior/-/tree/devel/src/robot_behavior -The code is primarily developed at INRIA by Timothée Wintz. #### Status @@ -466,26 +556,23 @@ The code is primarily developed at INRIA by Timothée Wintz. #### Inputs/outputs - - Input: `look at` (undefined) - - Input: `following/nav goals` (undefined) - - Input: `occupancy map` (undefined) - - Input: `status` (undefined) - - Input: `/h/i/groups [hri_msgs/GroupsStamped]` (topic) + - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `status` (undefined) - - Output: `low-level actions` (undefined) + - Output: `/h/f/*/cropped [sensor_msg/Image]` (topic) + - Output: `/h/f/*/roi [sensor_msgs/RegionOfInterest]` (topic) #### Dependencies -- `std_msgs/Empty` -- `hri_msgs/GroupsStamped` +- `sensor_msg/Image` +- `sensor_msgs/RegionOfInterest` +- `sensor_msgs/Image` --- -### userattentionestimation +### depthestimationfrommonocular -Node *User attention estimation* (id: `userattentionestimation`) is overseen by UNITN. +Node *Depth estimation from monocular* (id: `depthestimationfrommonocular`) is overseen by UNITN. @@ -495,24 +582,20 @@ Node *User attention estimation* (id: `userattentionestimation`) is overseen by #### Inputs/outputs - - Input: `TF (faces)` (undefined) - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `input` (undefined) - - Output: `tf: /face_id_gaze` (tf) - - Output: `x,y + attention heatmap` (undefined) + - Output: `depth` (undefined) #### Dependencies -- `tf/transform_broadcaster` - `std_msgs/Empty` -- `hri_msgs/RegionOfInterestStamped` --- -### rgbdcamera_info +### robotfunctionallayer -Node *RGB-D + camera_info* (id: `rgbdcamera_info`) is overseen by PAL. +Node *Robot functional layer* (id: `robotfunctionallayer`) is overseen by PAL. @@ -522,72 +605,78 @@ Node *RGB-D + camera_info* (id: `rgbdcamera_info`) is overseen by PAL. #### Inputs/outputs + - Input: `input` (undefined) - - Output: `/camera_torso/color/image_raw [sensor_msgs/Image]` (topic) #### Dependencies -- `sensor_msgs/Image` +- `std_msgs/Empty` --- -### nonverbalbehaviours - -Node *Non-verbal behaviours* (id: `nonverbalbehaviours`) is overseen by UNITN. +### fairmotmultipeoplebodytracker +Node *FairMOT Multi-people body tracker* (id: `fairmotmultipeoplebodytracker`) is overseen by INRIA. +This code is primarily developed at INRIA by Luis Gomez Camara. +REPO: https://gitlab.inria.fr/spring/wp3_av_perception/multi-person_visual_tracker/ #### Status -**This node is currently auto-generated (mock-up)** +**Current release: devel** #### Inputs/outputs - - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `/h/f/*/expression [hri_msgs/Expression]` (topic) + - Output: `/h/b/*/cropped [sensor_msg/Image]` (topic) + - Output: `/h/b/*/roi [sensor_msgs/RegionOfInterest` (undefined) #### Dependencies -- `hri_msgs/Expression` -- `audio_common_msgs/AudioData` -- `hri_msgs/RegionOfInterestStamped` +- `sensor_msg/Image` +- `sensor_msgs/Image` +- `std_msgs/Empty` --- -### speakeridentification +### mode -Node *Speaker identification* (id: `speakeridentification`) is overseen by BIU. +Node *MoDE* (id: `mode`) is overseen by BIU. -- online services -- not started yet +This node does: +- speech echo cancelation, +- speech enhancement, +- speech separation and diarization +REPO:https://gitlab.inria.fr/spring/wp3_av_perception/speech-enhancement +SUBFOLDER:audio_processing #### Status -**This node is currently auto-generated (mock-up)** +**Current release: BIU_dev** #### Inputs/outputs - - Input: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) - - Input: `tracking information` (undefined) + - Input: `sound localization` (undefined) + - Input: `/audio/ego_audio [audio_common_msgs/AudioData]` (topic) + - Input: `/audio/raw_audio [respeaker_ros/RawAudioData]` (topic) - - Output: `person_id_candidate [hri_msgs/IdsMatch]` (undefined) - - Output: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) + - Output: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) #### Dependencies - `audio_common_msgs/AudioData` - `std_msgs/Empty` +- `respeaker_ros/RawAudioData` --- -### maskdetection +### speechsynthesis -Node *Mask detection* (id: `maskdetection`) is overseen by UNITN. +Node *Speech synthesis* (id: `speechsynthesis`) is overseen by PAL. @@ -597,21 +686,20 @@ Node *Mask detection* (id: `maskdetection`) is overseen by UNITN. #### Inputs/outputs - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `speech [std_msgs/String]` (undefined) - - Output: `/h/f/*/mask [std_msgs/Bool]` (topic) + - Output: `/tts/feedback` (undefined) #### Dependencies -- `std_msgs/Bool` -- `hri_msgs/RegionOfInterestStamped` +- `std_msgs/Empty` --- -### semanticmapping +### uservisualfocus -Node *Semantic mapping* (id: `semanticmapping`) is overseen by CVUT. +Node *User visual focus* (id: `uservisualfocus`) is overseen by UNITN. @@ -621,29 +709,27 @@ Node *Semantic mapping* (id: `semanticmapping`) is overseen by CVUT. #### Inputs/outputs - - Input: `dense 3d map` (undefined) - - Input: `/detected_objects [spring_msgs/DetectedObjectArray]` (topic) + - Input: `scene` (undefined) + - Input: `attention` (undefined) + - Input: `depth` (undefined) + - Input: `gaze direction` (undefined) - - Output: `scene graph` (undefined) + - Output: `who's looking at what?` (undefined) + - Output: `/h/i/gaze [hri_msgs/Gaze]` (topic) #### Dependencies - `std_msgs/Empty` -- `spring_msgs/DetectedObjectArray` +- `hri_msgs/Gaze` --- -### speechanalysis +### orbslam -Node *Speech analysis* (id: `speechanalysis`) is overseen by BIU. +Node *ORB SLAM* (id: `orbslam`) is overseen by PAL. -This node does: -- speech echo cancelation, -- speech enhancement, -- speech separation and diarization -REPO: https://gitlab.inria.fr/spring/wp3_av_perception/biu-ros_audio_retreiver #### Status @@ -651,48 +737,54 @@ REPO: https://gitlab.inria.fr/spring/wp3_av_perception/biu-ros_audio_retreiver #### Inputs/outputs - - Input: `sound localization` (undefined) - - Input: `/audio/raw_audio [respeaker_ros/RawAudioData]` (topic) - - Input: `/audio/ego_audio [audio_common_msgs/AudioData]` (topic) + - Input: `/camera_torso/color/image_raw [sensor_msgs/Image]` (topic) - - Output: `/audio/postprocess_audio_streams -- cf output in node desc` (undefined) + - Output: `/map [nav_msgs/OccupancyGrid]` (topic) + - Output: `tf: /odom` (tf) #### Dependencies -- `std_msgs/Empty` -- `respeaker_ros/RawAudioData` -- `audio_common_msgs/AudioData` +- `nav_msgs/OccupancyGrid` +- `tf/transform_broadcaster` +- `sensor_msgs/Image` --- -### softbiometrics - -Node *Soft biometrics* (id: `softbiometrics`) is overseen by UNITN. +### robot_behavior +Node *robot_behavior* (id: `robot_behavior`) is overseen by INRIA. +The code is primarily developed at INRIA by Timothée Wintz. +REPO: https://gitlab.inria.fr/spring/wp6_robot_behavior/robot_behavior +SUBFOLDER:src/robot_behavior #### Status -**This node is currently auto-generated (mock-up)** +**Current release: devel** #### Inputs/outputs - - Input: `/h/f/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) + - Input: `following/nav goals` (undefined) + - Input: `status` (undefined) + - Input: `/h/i/groups [hri_msgs/Group]` (topic) + - Input: `look at` (undefined) + - Input: `occupancy map` (undefined) - - Output: `/h/f/*/demographics [hri_msgs/AgeAndGender]` (topic) + - Output: `status` (undefined) + - Output: `low-level actions` (undefined) #### Dependencies -- `hri_msgs/AgeAndGender` -- `hri_msgs/RegionOfInterestStamped` +- `std_msgs/Empty` +- `hri_msgs/Group` --- -### occupancymap +### semanticmapping -Node *Occupancy map* (id: `occupancymap`) is overseen by CVUT. +Node *Semantic mapping* (id: `semanticmapping`) is overseen by CVUT. @@ -702,50 +794,48 @@ Node *Occupancy map* (id: `occupancymap`) is overseen by CVUT. #### Inputs/outputs - - Input: `TF (bodies)` (undefined) - Input: `dense 3d map` (undefined) + - Input: `/detected_objects [spring_msgs/DetectedObjectArray]` (topic) - - Output: `/map_refined [nav_msgs/OccupancyGrid]` (topic) + - Output: `scene graph` (undefined) #### Dependencies - `std_msgs/Empty` -- `nav_msgs/OccupancyGrid` +- `spring_msgs/DetectedObjectArray` --- -### dialoguearbiter +### social_scene_context_understanding -Node *dialogue arbiter* (id: `dialoguearbiter`) is overseen by HWU. +Node *social_scene_context_understanding* (id: `social_scene_context_understanding`) is overseen by HWU. -REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/dialogue/-/tree/spring-dev/dialogue_arbiter +REPO:git@gitlab.inria.fr:spring/wp5_spoken_conversations/interaction.git +SUBFOLDER:social_scene_context_understanding #### Status -**This node is currently auto-generated (mock-up)** +**Current release: spring_dev** #### Inputs/outputs - - Input: `interaction messages` (undefined) - - Input: `/h/v/*/speech [std_msgs/String]` (topic) + - Input: `scene graph` (undefined) - - Output: `next utterance` (undefined) - - Output: `DialogueState` (undefined) + - Output: `semantic description` (undefined) #### Dependencies - `std_msgs/Empty` -- `std_msgs/String` --- -### plan_actions +### robotgui + +Node *Robot GUI* (id: `robotgui`) is overseen by ERM. -Node *plan_actions* (id: `plan_actions`) is overseen by HWU. -REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/plan_actions #### Status @@ -753,53 +843,48 @@ REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/plan_actions #### Inputs/outputs - - Input: `TF (persons)` (undefined) - - Input: `semantic scene description` (undefined) - - Input: `demographics` (undefined) - - Input: `dialogue state` (undefined) + - Input: `speech output` (undefined) + - Input: `additional support material` (undefined) + - Input: `/h/v/*/speech [hri_msgs/LiveSpeech]` (topic) + - Input: `/tts/feedback` (undefined) - - Output: `interaction state` (undefined) - - Output: `output` (undefined) - - Output: `nav goals` (undefined) #### Dependencies - `std_msgs/Empty` +- `hri_msgs/LiveSpeech` --- -### uservisualfocus - -Node *User visual focus* (id: `uservisualfocus`) is overseen by UNITN. +### ros_openpose +Node *ros_openpose* (id: `ros_openpose`) is overseen by UNITN. +REPO:git@gitlab.inria.fr:spring/wp4_behavior/wp4_behavior_understanding.git +SUBFOLDER:ros_openpose #### Status -**This node is currently auto-generated (mock-up)** +**Current release: 0.0.1** #### Inputs/outputs - - Input: `attention` (undefined) - - Input: `scene` (undefined) - - Input: `gaze direction` (undefined) - - Input: `depth` (undefined) + - Input: `/h/b/*/cropped [sensor_msg/Image]` (topic) - - Output: `who's looking at what?` (undefined) - - Output: `/h/i/gaze [hri_msgs/GazesStamped]` (topic) + - Output: `/h/b/*/skeleton2d [hri_msgs/Skeleton2D]` (topic) #### Dependencies -- `std_msgs/Empty` -- `hri_msgs/GazesStamped` +- `sensor_msg/Image` +- `hri_msgs/Skeleton2D` --- -### speechsynthesis +### activityreco -Node *Speech synthesis* (id: `speechsynthesis`) is overseen by PAL. +Node *Activity reco* (id: `activityreco`) is overseen by UNITN. @@ -809,9 +894,10 @@ Node *Speech synthesis* (id: `speechsynthesis`) is overseen by PAL. #### Inputs/outputs - - Input: `speech [std_msgs/String]` (undefined) + - Input: `TF (bodies)` (undefined) + - Input: `gaze direction` (undefined) - - Output: `/tts/feedback` (undefined) + - Output: `[?] output` (undefined) #### Dependencies @@ -820,32 +906,39 @@ Node *Speech synthesis* (id: `speechsynthesis`) is overseen by PAL. --- -### sceneunderstanding - -Node *Scene understanding* (id: `sceneunderstanding`) is overseen by HWU. +### robot_behaviour_plan_actions +Node *robot_behaviour_plan_actions* (id: `robot_behaviour_plan_actions`) is overseen by HWU. +REPO:git@gitlab.inria.fr:spring/wp5_spoken_conversations/plan_actions.git +SUBFOLDER:robot_behaviour_plan_actions #### Status -**This node is currently auto-generated (mock-up)** +**Current release: spring_dev** #### Inputs/outputs - - Input: `scene graph` (undefined) + - Input: `TF (persons)` (undefined) + - Input: `semantic scene description` (undefined) + - Input: `dialogue state` (undefined) + - Input: `/h/p/*/softbiometrics [hri_msgs/Softbiometrics]` (topic) - - Output: `semantic description` (undefined) + - Output: `nav goals` (undefined) + - Output: `interaction state` (undefined) + - Output: `output` (undefined) #### Dependencies - `std_msgs/Empty` +- `hri_msgs/Softbiometrics` --- -### rosopenpose +### multipersonaudiotracking -Node *ROS openpose* (id: `rosopenpose`) is overseen by INRIA. +Node *Multi-person audio tracking* (id: `multipersonaudiotracking`) is overseen by BIU. @@ -855,20 +948,25 @@ Node *ROS openpose* (id: `rosopenpose`) is overseen by INRIA. #### Inputs/outputs - - Input: `input` (undefined) + - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) + - Output: `tracking information` (undefined) - Output: `output` (undefined) + - Output: `source angle` (undefined) + - Output: `tf: /voice_id` (tf) #### Dependencies +- `audio_common_msgs/AudioData` - `std_msgs/Empty` +- `tf/transform_broadcaster` --- -### visualslam3dmap +### inlocros -Node *Visual SLAM + 3D map* (id: `visualslam3dmap`) is overseen by CVUT. +Node *InLoc-ROS* (id: `inlocros`) is overseen by CVUT. @@ -878,8 +976,8 @@ Node *Visual SLAM + 3D map* (id: `visualslam3dmap`) is overseen by CVUT. #### Inputs/outputs - - Input: `/camera_head/color/image_head [sensor_msgs/Image]` (topic) - Input: `/camera_torso/color/image_torso [sensor_msgs/Image]` (topic) + - Input: `/camera_head/color/image_head [sensor_msgs/Image]` (topic) - Input: `localisation prior` (undefined) - Output: `dense 3d map` (undefined) @@ -887,42 +985,43 @@ Node *Visual SLAM + 3D map* (id: `visualslam3dmap`) is overseen by CVUT. #### Dependencies -- `sensor_msgs/Image` - `std_msgs/Empty` - `tf/transform_broadcaster` +- `sensor_msgs/Image` --- -### objectdetectionidentificationlocalisation - -Node *Object detection/identification/localisation* (id: `objectdetectionidentificationlocalisation`) is overseen by CVUT. +### google_asr +Node *google_asr* (id: `google_asr`) is overseen by BIU. +REPO:https://gitlab.inria.fr/spring/wp5_spoken_conversations/asr +SUBFOLDER:google_asr/google_asr #### Status -**This node is currently auto-generated (mock-up)** +**Current release: BIU_dev** #### Inputs/outputs - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Input: `/audio/postprocess_audio_streams [audio_common_msgs/AudioData]` (topic) - - Output: `/detected_objects [spring_msgs/DetectedObjectArray]` (topic) + - Output: `/audio/speech_streams -- array of hri_msgs/LiveSpeech` (undefined) #### Dependencies -- `sensor_msgs/Image` -- `spring_msgs/DetectedObjectArray` +- `audio_common_msgs/AudioData` +- `std_msgs/Empty` --- -### body2dpose +### occupancymap + +Node *Occupancy map* (id: `occupancymap`) is overseen by CVUT. -Node *Body 2D pose* (id: `body2dpose`) is overseen by UNITN. -MOCK: Body 2D pose [openpose] is... #### Status @@ -930,25 +1029,24 @@ MOCK: Body 2D pose [openpose] is... #### Inputs/outputs - - Input: `/h/b/*/roi [hri_msgs/RegionOfInterestStamped]` (topic) - - Input: `/camera_head/color/image_raw [sensor_msgs/Image]` (topic) + - Input: `dense 3d map` (undefined) + - Input: `TF (bodies)` (undefined) - - Output: `/h/b/*/skeleton2d [hri_msgs/Skeleton2D]` (topic) + - Output: `/map_refined [nav_msgs/OccupancyGrid]` (topic) #### Dependencies -- `hri_msgs/RegionOfInterestStamped` -- `sensor_msgs/Image` -- `hri_msgs/Skeleton2D` +- `std_msgs/Empty` +- `nav_msgs/OccupancyGrid` --- -### interaction_manager +### raspicam + +Node *raspicam* (id: `raspicam`) is overseen by PAL. -Node *interaction_manager* (id: `interaction_manager`) is overseen by HWU. -REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/interaction/-/tree/spring_dev/interaction_manager #### Status @@ -956,94 +1054,102 @@ REPO: https://gitlab.inria.fr/spring/wp5_spoken_conversations/interaction/-/tree #### Inputs/outputs - - Input: `semantic scene description` (undefined) - - Input: `input` (undefined) - - Input: `dialogue state` (undefined) - - Input: `robot state` (undefined) - - Input: `TF` (undefined) - - Input: `/h/p/personID` (undefined) - - Output: `who to look at` (undefined) - - Output: `verbal command` (undefined) - - Output: `nav goals` (undefined) - - Output: `active personID` (undefined) - - Output: `gestures` (undefined) + - Output: `/head_front_camera/color/image_raw [sensor_msgs/Image]` (topic) #### Dependencies -- `std_msgs/Empty` +- `sensor_msgs/Image` ---- -### personmanager +### Non-executable dependency: interaction_manager_msgs + +Module interaction_manager_msgs (id: `interaction_manager_msgs`) is overseen by HWU. -Node *Person manager* (id: `personmanager`) is overseen by INRIA. +REPO:git@gitlab.inria.fr:spring/wp5_spoken_conversations/interaction.git +SUBFOLDER:interaction_manager_msgs +NOT EXECUTABLE -PAL to write an initial temporary version, simply forwarding detected faces as persons. #### Status -**This node is currently auto-generated (mock-up)** +**Current release: spring_dev** -#### Inputs/outputs +#### Dependencies - - Input: `TF (voices)` (undefined) - - Input: `TF (faces)` (undefined) - - Input: `candidate_matchs [hri_msgs/IdsMatch]` (undefined) - - Input: `/h/f/*/demographics [hri_msgs/AgeAndGender]` (topic) - - Output: `tf: /person_id` (tf) - - Output: `/humans/persons/*/body_id [std_msgs/String]` (topic) - - Output: `/h/p/*/voice_id [std_msgs/String]` (topic) - - Output: `/h/p/*/demographics [hri_msgs/AgeAndGender]` (topic) - - Output: `/h/p/*/face_id [std_msgs/String]` (topic) + + + +### Non-executable dependency: hri_msgs + +Module hri_msgs (id: `hri_msgs`) is overseen by PAL. + +REPO: git@gitlab:ros4hri/hri_msgs.git +NOT EXECUTABLE + + +#### Status + +**Current release: 0.1.1** #### Dependencies -- `tf/transform_broadcaster` -- `std_msgs/String` -- `std_msgs/Empty` -- `hri_msgs/AgeAndGender` ---- -### voicespeechmatching -Node *Voice speech matching* (id: `voicespeechmatching`) is overseen by BIU. +### Non-executable dependency: robot_behaviour_msgs +Module robot_behaviour_msgs (id: `robot_behaviour_msgs`) is overseen by HWU. + +REPO:git@gitlab.inria.fr:spring/wp5_spoken_conversations/plan_actions.git +SUBFOLDER:robot_behaviour_msgs +NOT EXECUTABLE #### Status -**This node is currently auto-generated (mock-up)** +**Current release: spring_dev** + +#### Dependencies + + -#### Inputs/outputs - - Input: `/audio/speech_streams [std_msgs/String]` (topic) - - Input: `/h/v/*/audio [audio_common_msgs/AudioData]` (topic) - - Output: `/h/v/*/speech [std_msgs/String]` (topic) +### Non-executable dependency: audio_msgs + +Module audio_msgs (id: `audio_msgs`) is overseen by HWU. + +REPO:git@gitlab.inria.fr:spring/wp5_spoken_conversations/asr.git +SUBFOLDER:audio_msgs +NOT EXECUTABLE + + +#### Status + +**Current release: spring_dev** #### Dependencies -- `std_msgs/String` -- `audio_common_msgs/AudioData` -### Non-executable dependency: spring_msgs -Module spring_msgs (id: `spring_msgs`) is overseen by Other. +### Non-executable dependency: social_scene_msgs + +Module social_scene_msgs (id: `social_scene_msgs`) is overseen by HWU. -REPO:git@gitlab.inria.fr:spring/wp7_ari/spring_msgs.git +REPO:git@gitlab.inria.fr:spring/wp5_spoken_conversations/interaction.git +SUBFOLDER:social_scene_msgs NOT EXECUTABLE #### Status -**Current release: 0.0.2** +**Current release: spring_dev** #### Dependencies @@ -1051,17 +1157,18 @@ NOT EXECUTABLE -### Non-executable dependency: hri_msgs +### Non-executable dependency: wp4_msgs -Module hri_msgs (id: `hri_msgs`) is overseen by Other. +Module wp4_msgs (id: `wp4_msgs`) is overseen by UNITN. -REPO: git@gitlab:ros4hri/hri_msgs.git +REPO:git@gitlab.inria.fr:spring/wp4_behavior/wp4_behavior_understanding.git +SUBFOLDER:wp4_msgs NOT EXECUTABLE #### Status -**Current release: 0.1.1** +**Current release: master** #### Dependencies