In our latest series of studies, we are thrilled to highlight the advancements and innovations occurring in the realm of human-robot collaboration (HRC). Central to our exploration is the development of real-time Human-Computer Interaction (HCI) systems that elevate the synergy between humans and robots to unprecedented levels.
In this project, we take a notable stride forward, proposing an HCI system informed by eye gaze, leveraging a combination of cutting-edge techniques such as Dlib 68-point landmark detection and Mask R-CNN for exemplary accuracy in tool and part segmentation and eye gaze recognition. This commitment to precision is echoed in our gesture communication studies, where we employ dynamic gesture recognition systems built on the synergy of Motion History Image (MHI) and Convolutional Neural Networks (CNN) to facilitate seamless interaction between human workers and industrial robots.
As we navigate further, we unfold multi-modal HRC systems that harmonize speech and gestures, epitomizing the collaborative future through real-time, advanced interaction capabilities and multi-threading architectures. This theme of real-time collaboration reverberates in our subsequent endeavors, which focus on fostering instantaneous responses through dynamic gestures, introducing robust systems that amalgamate speed with intelligence, ensuring harmony in industrial settings.
Venturing beyond conventional boundaries, we integrate attention-based approaches in Human Activity Recognition (HAR) via wearable devices, presenting a groundbreaking strategy that promises to redefine activity tracking in our daily lives. The series also witnesses an invigorated approach to repetitive counting, offering a solution that stands resilient against the prevalent issues in existing methods.
Finally, we take a leap into the weakly-supervised space of temporal action localization, presenting ACM-BANets to meticulously address the challenges of false positives and negatives, promising a future of more accurate and comprehensive action recognition systems.
Join us as we embark on this fascinating journey, shaping a future where human-robot collaborations transcend existing barriers, ushering in an era of harmony, efficiency, and unprecedented potential.
Recent Publications
2023
Chen, Haodong., Zendehdel, Niloofar, Leu, M.C. and Zhaozheng Yin. “Real-time Human-Computer Interaction Using Eye Gazes.” , in Proceedings of the ASME 51st SME North American Manufacturing Research Conference. American Society of Mechanical Engineers. 2023
Chen, Haodong, Ming C. Leu, Md Moniruzzaman, Zhaozheng Yin, Solmaz Hajmohammadi, and Zhuoqing Chang. “Advancements in Repetitive Action Counting: Joint-Based PoseRAC Model With Improved Performance.” arXiv preprint arXiv:2308.08632 (2023).
2022
Chen, Haodong, Ming C. Leu, and Zhaozheng Yin. “Real-time multi-modal human–robot collaboration using gestures and speech.” Journal of Manufacturing Science and Engineering 144, no. 10 (2022): 101007.
2021
Tao, Wenjin, Haodong Chen, Md Moniruzzaman, Ming C. Leu, Zhaozheng Yi, and Ruwen Qin. “Attention-based sensor fusion for human activity recognition using IMU signals.” arXiv preprint arXiv:2112.11224 (2021).
2020
Chen, Haodong, Wenjin Tao, Ming C. Leu, and Zhaozheng Yin. “Dynamic gesture design and recognition for human-robot collaboration with convolutional neural networks.” In International Symposium on Flexible Automation, vol. 83617, p. V001T09A001. American Society of Mechanical Engineers, 2020.
Chen, Haodong, Ming C. Leu, Wenjin Tao, and Zhaozheng Yin. “Design of a real-time human-robot collaboration system using dynamic gestures.” In ASME International Mechanical Engineering Congress and Exposition, vol. 84492, p. V02BT02A051. American Society of Mechanical Engineers, 2020.
Moniruzzaman, Md, Zhaozheng Yin, Zhihai He, Ruwen Qin, and Ming C. Leu. “Action completeness modeling with background aware networks for weakly-supervised temporal action localization.” In Proceedings of the 28th ACM international conference on multimedia, pp. 2166-2174. 2020.