Towards Controlling Mobile Robot Using Upper Human Body Gesture Based on Convolutional Neural Network

Muhammad Fuad, Faikul Umam, Sri Wahyuni, Nuniek Fahriani, Ilham Nurwahyudi, Mochammad Ilham Darwaman, Fahmi Maulana

Abstract


Human-Robot Interaction (HRI) has challenges in investigation of a nonverbal and natural interaction. This study contributes to developing a gesture recognition system capable of recognizing the entire human upper body for HRI, which has never been done in previous research. Preprocessing is applied to improve image quality, reduce noise and highlight important features of each image, including color segmentation, thresholding and resizing. The hue, saturation, value (HSV) color segmentation is executed by utilizing blue color backdrop and additional lighting to deal with illumination issue. Then thresholding is performed to get a black and white image to distinguish between background and foreground. The resizing is completed to adjust the image to match the size expected by the model. The preprocessed data image is used as input for gesture recognition based on Convolutional Neural Network (CNN). This study recorded five gestures from five research subjects in difference gender and body posture with total of 450 images which divided into 380 and 70 images for training and testing respectively. Experiments that performed in an indoor environment showed that CNN achieved 92% of accuracy in the gesture recognition. It has lower level of accuracy compare to AlexNet model but with faster training computation time of 9 seconds. This result was obtained by testing the system over various distances. The optimal distance for a camera setting from user to interact with mobile robot by using gesture was 2.5 m. For future research, the proposed method will be improved and implemented for mobile robot motion control.

Keywords


Gesture; Upper Human Body; Convolutional Neural Network; Mobile Robot Control.

Full Text:

PDF

References


M. A. Goodrich and A. C. Schultz, Human-robot interaction: a survey. in Foundations and trends in human-computer interaction, no. 1,3. Boston, Mass.: Now, 2007.

Barteck, Christoph, Balpaeme, Tony, Eyssel, Frederike, Kanda, Takayuki, Keijsers, Meyer, and Sabanovic, Selma, “Human–Robot Interaction: An Introduction,” Camb. Univ. Press, 2020, doi: 10.1017/9781108676649.

E. S. Neo, K. Yokoi, S. Kajita, and K. Tanie, “Whole-Body Motion Generation Integrating Operator’s Intention and Robot’s Autonomy in Controlling Humanoid Robots,” IEEE Trans. Robot., vol. 23, no. 4, pp. 763–775, 2007, doi: 10.1109/TRO.2007.903818.

S. K. Cho, H. Z. Jin, J. M. Lee, and B. Yao, “Teleoperation of a Mobile Robot Using a Force-Reflection Joystick With Sensing Mechanism of Rotating Magnetic Field,” IEEEASME Trans. Mechatron., vol. 15, no. 1, pp. 17–26, 2010, doi: 10.1109/TMECH.2009.2013848.

S. K. Agrawal, X. Chen, C. Ragonesi, and J. C. Galloway, “Training Toddlers Seated on Mobile Robots to Steer Using Force-Feedback Joystick,” IEEE Trans. Haptics, vol. 5, no. 4, pp. 376–383, 2012, doi: 10.1109/TOH.2011.67.

Y. Kim, T. Oyabu, G. Obinata, and K. Hase, “Operability of Joystick-Type Steering Device Considering Human Arm Impedance Characteristics,” IEEE Trans. Syst. Man Cybern. - Part Syst. Hum., vol. 42, no. 2, pp. 295–306, 2012, doi: 10.1109/TSMCA.2011.2162501.

D. Scaradozzi, L. Sorbi, S. Zingaretti, M. Biagiola, and E. Omerdic, “Development and integration of a novel IP66 Force Feedback Joystick for offshore operations,” in 22nd Mediterranean Conference on Control and Automation, pp. 664–669, 2014, doi: 10.1109/MED.2014.6961449.

H. Zhang and S. K. Agrawal, “An Active Neck Brace Controlled by a Joystick to Assist Head Motion,” IEEE Robot. Autom. Lett., vol. 3, no. 1, pp. 37–43, 2018, doi: 10.1109/LRA.2017.2728858.

K. A. Radhika, B. L. Raksha, B. R. Sujatha, U. Pruthviraj, and K. V. Gangadharan, “IoT Based Joystick Controlled Pibot Using Socket Communication,” in 2018 IEEE Distributed Computing, VLSI, Electrical Circuits and Robotics (DISCOVER), Mangalore (Mangaluru), pp. 121–125, 2018, doi: 10.1109/DISCOVER.2018.8674130.

M. Mohammadi, H. Knoche, M. Gaihede, B. Bentsen, and L. N. S. Andreasen Struijk, “A high-resolution tongue-based joystick to enable robot control for individuals with severe disabilities,” in 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR), pp. 1043–1048, 2019, doi: 10.1109/ICORR.2019.8779434.

R. Rahman, M. S. Rahman, and J. R. Bhuiyan, “Joystick controlled industrial robotic system with robotic arm,” in 2019 IEEE International Conference on Robotics, Automation, Artificial-intelligence and Internet-of-Things (RAAICON), pp. 31–34, 2019, doi: 10.1109/RAAICON48939.2019.18.

Z. Wang et al., “Joystick Car Drive System and its Application to Self-driving Microbus,” in IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society, pp. 632–637, 2020, doi: 10.1109/IECON43393.2020.9254364.

H. Zhang, B.-C. Chang, Y.-J. Rue, and S. K. Agrawal, “Using the Motion of the Head-Neck as a Joystick for Orientation Control,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 27, no. 2, pp. 236–243, 2019, doi: 10.1109/TNSRE.2019.2894517.

M. Mohammadi, H. Knoche, and L. N. S. A. Struijk, “Continuous Tongue Robot Mapping for Paralyzed Individuals Improves the Functional Performance of Tongue-Based Robotic Assistance,” IEEE Trans. Biomed. Eng., vol. 68, no. 8, pp. 2552–2562, 2021, doi: 10.1109/TBME.2021.3055250.

J. C. Yepes, J. J. Yepes, J. R. Martinez, and V. Z. Perez, “Implementation of an Android based teleoperation application for controlling a KUKA-KR6 robot by using sensor fusion,” in 2013 Pan American Health Care Exchanges (PAHCE), pp. 1–5, 2013, doi: 10.1109/PAHCE.2013.6568286.

P. Rouanet, P.-Y. Oudeyer, F. Danieau, and D. Filliat, “The Impact of Human–Robot Interfaces on the Learning of Visual Objects,” IEEE Trans. Robot., vol. 29, no. 2, pp. 525–541, 2013, doi: 10.1109/TRO.2012.2228134.

J. Nadvornik and P. Smutny, “Remote control robot using Android mobile device,” in Proceedings of the 2014 15th International Carpathian Control Conference (ICCC), pp. 373–378, 2014, doi: 10.1109/CarpathianCC.2014.6843630.

G. Mohanarajah, V. Usenko, M. Singh, R. D’Andrea, and M. Waibel, “Cloud-Based Collaborative 3D Mapping in Real-Time With Low-Cost Robots,” IEEE Trans. Autom. Sci. Eng., vol. 12, no. 2, pp. 423–431, 2015, doi: 10.1109/TASE.2015.2408456.

S. P. Donohoe, S. A. Velinsky, and T. A. Lasky, “Mechatronic Implementation of a Force Optimal Underconstrained Planar Cable Robot,” IEEEASME Trans. Mechatron., vol. 21, no. 1, pp. 69–78, 2016, doi: 10.1109/TMECH.2015.2431192.

G. Loianno, C. Brunner, G. McGrath, and V. Kumar, “Estimation, Control, and Planning for Aggressive Flight With a Small Quadrotor With a Single Camera and IMU,” IEEE Robot. Autom. Lett., vol. 2, no. 2, pp. 404–411, 2017, doi: 10.1109/LRA.2016.2633290.

K.-T. Song, S.-Y. Jiang, and M.-H. Lin, “Interactive Teleoperation of a Mobile Manipulator Using a Shared-Control Approach,” IEEE Trans. Hum.-Mach. Syst., vol. 46, no. 6, pp. 834–845, 2016, doi: 10.1109/THMS.2016.2586760.

Y. Wang et al., “Monitoring Aquatic Debris Using Smartphone-Based Robots,” IEEE Trans. Mob. Comput., vol. 15, no. 6, pp. 1412–1426, 2016, doi: 10.1109/TMC.2015.2460240.

B. Gokcen, F. Baygul, F. Cakmak, E. Uslu, M. F. Amasyali, and S. Yavuz, “Android application for simultaneously control of multiple land robots which have different drive strategy,” in 2017 International Conference on Computer Science and Engineering (UBMK), pp. 724–728, 2017,doi: 10.1109/UBMK.2017.8093513.

K.-R. Kim, S.-H. Jeong, W.-Y. Kim, Y. Jeon, K.-S. Kim, and J.-H. Hong, “Design of small mobile robot remotely controlled by an android operating system via bluetooth and NFC communication,” in 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), pp. 913–915, 2017, doi: 10.1109/URAI.2017.7992864.

G. Zheng, S. Bi, H. Min, K. Yang, and Y. Zhang, “Design and Implementation of Chinese Speech Robot Control System Based on Android Embedded Platform,” in 2017 IEEE 7th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pp. 498–503, 2017, doi: 10.1109/CYBER.2017.8446468.

J. Lim, G. S. Tewolde, J. Kwon, and S. Choi, “Design and Implementation of a Network Robotic Framework Using a Smartphone-Based Platform,” IEEE Access, vol. 7, pp. 59357–59368, 2019, doi: 10.1109/ACCESS.2019.2915344.

M. Alwateer, S. W. Loke, and N. Fernando, “Enabling Drone Services: Drone Crowdsourcing and Drone Scripting,” IEEE Access, vol. 7, pp. 110035–110049, 2019, doi: 10.1109/ACCESS.2019.2933234.

R. K. Megalingam et al., “Amaran: An Unmanned Robotic Coconut Tree Climber and Harvester,” IEEEASME Trans. Mechatron., pp. 1–1, 2020, doi: 10.1109/TMECH.2020.3014293.

L. Wu, R. Alqasemi, and R. Dubey, “Development of Smartphone-Based Human-Robot Interfaces for Individuals With Disabilities,” IEEE Robot. Autom. Lett., vol. 5, no. 4, pp. 5835–5841, 2020, doi: 10.1109/LRA.2020.3010453.

W. Dudek and T. Winiarski, “Scheduling of a Robot’s Tasks With the TaskER Framework,” IEEE Access, vol. 8, pp. 161449–161471, 2020, doi: 10.1109/ACCESS.2020.3020265.

D. Kiryanov, R. Lavrenov, R. Safin, M. Svinin, and E. Magid, “Mobile application for controlling multiple robots,” in 2021 IEEE 16th Conference on Industrial Electronics and Applications (ICIEA), pp. 1913–1917, 2021, doi: 10.1109/ICIEA51954.2021.9516091.

Y.-B. Lin, H. Luo, C.-C. Liao, and Y.-F. Huang, “PuppetTalk: Conversation Between Glove Puppetry and Internet of Things,” IEEE Access, vol. 9, pp. 6786–6797, 2021, doi: 10.1109/ACCESS.2020.3048697.

M. Fuad and D. Purwanto, “Wall-Following Using a Kinect Sensor for Corridor Coverage Navigation,” Journal of Theoretical and Applied Information Technology, vol. 70, no. 1, pp. 106 - 111, 2014.

M. Fuad, “RGB-D Image Based Landmark Identification for Corridor Coverage Navigation,” Journal of Theoretical and Applied Information Technology, vol. 79, no. 3, pp. 437 - 443, 2015.

M. Fuad, T. Agustinah, D. Purwanto, T. A. Sardjono, and R. Dikairono, “Robot Orientation Estimation Based on Single-Frame of Fish-eye Image,” J. Phys. Conf. Ser., vol. 1569, no. 2, p. 022092, 2020, doi: 10.1088/1742-6596/1569/2/022092.

M. Fuad, “Translational Motion Estimation Using Kinect,” J. Phys. Conf. Ser., vol. 1569, p. 032067, 2020, doi: 10.1088/1742-6596/1569/3/032067.

M. Fuad, “Skeleton based gesture to control manipulator,” in 2015 International Conference on Advanced Mechatronics, Intelligent Manufacture, and Industrial Automation (ICAMIMIA), pp. 96–101, 2015, doi: 10.1109/ICAMIMIA.2015.7508010.

Y. Liao, P. Xiong, W. Min, W. Min, and J. Lu, “Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks,” IEEE Access, vol. 7, pp. 38044–38054, 2019, doi: 10.1109/ACCESS.2019.2904749.

W. Fang, Y. Ding, F. Zhang, and J. Sheng, “Gesture Recognition Based on CNN and DCGAN for Calculation and Text Output,” IEEE Access, vol. 7, pp. 28230–28237, 2019, doi: 10.1109/ACCESS.2019.2901930.

A. D. Berenguer, M. C. Oveneke, H.-U.-R. Khalid, M. Alioscha-Perez, A. Bourdoux, and H. Sahli, “GestureVLAD: Combining Unsupervised Features Representation and Spatio-Temporal Aggregation for Doppler-Radar Gesture Recognition,” IEEE Access, vol. 7, pp. 137122–137135, 2019, doi: 10.1109/ACCESS.2019.2942305.

Z. Wang et al., “Hand Gesture Recognition Based on Active Ultrasonic Sensing of Smartphone: A Survey,” IEEE Access, vol. 7, pp. 111897–111922, 2019, doi: 10.1109/ACCESS.2019.2933987.

K. Cheng, N. Ye, R. Malekian, and R. Wang, “In-Air Gesture Interaction: Real Time Hand Posture Recognition Using Passive RFID Tags,” IEEE Access, vol. 7, pp. 94460–94472, 2019, doi: 10.1109/ACCESS.2019.2928318.

C. Liu, Y. Li, D. Ao, and H. Tian, “Spectrum-Based Hand Gesture Recognition Using Millimeter-Wave Radar Parameter Measurements,” IEEE Access, vol. 7, pp. 79147–79158, 2019, doi: 10.1109/ACCESS.2019.2923122.

Y. Wang, S. Wang, M. Zhou, Q. Jiang, and Z. Tian, “TS-I3D Based Hand Gesture Recognition Method With Radar Sensor,” IEEE Access, vol. 7, pp. 22902–22913, 2019, doi: 10.1109/ACCESS.2019.2897060.

T. Zhang, T. Song, D. Chen, T. Zhang, and J. Zhuang, “WiGrus: A Wifi-Based Gesture Recognition System Using Software-Defined Radio,” IEEE Access, vol. 7, pp. 131102–131113, 2019, doi: 10.1109/ACCESS.2019.2940386.

Y. Wang, A. Ren, M. Zhou, W. Wang, and X. Yang, “A Novel Detection and Recognition Method for Continuous Hand Gesture Using FMCW Radar,” IEEE Access, vol. 8, pp. 167264–167275, 2020, doi: 10.1109/ACCESS.2020.3023187.

M. Lee and J. Bae, “Deep Learning Based Real-Time Recognition of Dynamic Finger Gestures Using a Data Glove,” IEEE Access, vol. 8, pp. 219923–219933, 2020, doi: 10.1109/ACCESS.2020.3039401.

H. Heydarian, P. V. Rouast, M. T. P. Adam, T. Burrows, C. E. Collins, and M. E. Rollo, “Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions,” IEEE Access, vol. 8, pp. 164936–164949, 2020, doi: 10.1109/ACCESS.2020.3022042.

M. Al-Hammadi et al., “Deep Learning-Based Approach for Sign Language Gesture Recognition With Efficient Hand Gesture Representation,” IEEE Access, vol. 8, pp. 192527–192542, 2020, doi: 10.1109/ACCESS.2020.3032140.

S. Aly and W. Aly, “DeepArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition,” IEEE Access, vol. 8, pp. 83199–83212, 2020, doi: 10.1109/ACCESS.2020.2990699.

Y. Guo, S. Yang, N. Li, and X. Jiang, “Device-Free Localization Scheme With Time-Varying Gestures Using Block Compressive Sensing,” IEEE Access, vol. 8, pp. 88951–88960, 2020, doi: 10.1109/ACCESS.2020.2993576.

Y. Che and Y. Qi, “Embedding Gesture Prior to Joint Shape Optimization Based Real-Time 3D Hand Tracking,” IEEE Access, vol. 8, pp. 34204–34214, 2020, doi: 10.1109/ACCESS.2020.2974551.

J. Y. Oh, J.-H. Park, and J.-M. Park, “FingerTouch: Touch Interaction Using a Fingernail-Mounted Sensor on a Head-Mounted Display for Augmented Reality,” IEEE Access, vol. 8, pp. 101192–101208, 2020, doi: 10.1109/ACCESS.2020.2997972.

Z. Ni and Q. Li, “Fusion Learning Model for Mobile Face Safe Detection and Facial Gesture Analysis,” IEEE Access, vol. 8, pp. 61043–61050, 2020, doi: 10.1109/ACCESS.2019.2948714.

S. M. Aslam and S. Samreen, “Gesture Recognition Algorithm for Visually Blind Touch Interaction Optimization Using Crow Search Method,” IEEE Access, vol. 8, pp. 127560–127568, 2020, doi: 10.1109/ACCESS.2020.3006443.

G. Luo, P. Yang, M. Chen, and P. Li, “HCI on the Table: Robust Gesture Recognition Using Acoustic Sensing in Your Hand,” IEEE Access, vol. 8, pp. 31481–31498, 2020, doi: 10.1109/ACCESS.2020.2973305.

E. A. Ibrahim, M. Geilen, M. Li, and J. P. De Gyvez, “Multi-Angle Fusion for Low-Cost Near-Field Ultrasonic in-Air Gesture Recognition,” IEEE Access, vol. 8, pp. 191204–191218, 2020, doi: 10.1109/ACCESS.2020.3031677.

P. V. Rouast, H. Heydarian, M. T. P. Adam, and M. E. Rollo, “OREBA: A Dataset for Objectively Recognizing Eating Behavior and Associated Intake,” IEEE Access, vol. 8, pp. 181955–181963, 2020, doi: 10.1109/ACCESS.2020.3026965.

F. Ma, F. Song, Y. Liu, and J. Niu, “Quantitative Analysis on the Interaction Fatigue of Natural Gestures,” IEEE Access, vol. 8, pp. 190797–190811, 2020, doi: 10.1109/ACCESS.2020.3031967.

R. Ji, “Research on Basketball Shooting Action Based on Image Feature Extraction and Machine Learning,” IEEE Access, vol. 8, pp. 138743–138751, 2020, doi: 10.1109/ACCESS.2020.3012456.

C. Wu et al., “sEMG Measurement Position and Feature Optimization Strategy for Gesture Recognition Based on ANOVA and Neural Networks,” IEEE Access, vol. 8, pp. 56290–56299, 2020, doi: 10.1109/ACCESS.2020.2982405.

H. Bi, J. Zhang, and Y. Chen, “SmartGe: Identifying Pen-Holding Gesture With Smartwatch,” IEEE Access, vol. 8, pp. 28820–28830, 2020, doi: 10.1109/ACCESS.2020.2967770.

F. S. Khan, M. N. H. Mohd, D. M. Soomro, S. Bagchi, and M. D. Khan, “3D Hand Gestures Segmentation and Optimized Classification Using Deep Learning,” IEEE Access, vol. 9, pp. 131614–131624, 2021, doi: 10.1109/ACCESS.2021.3114871.

A. Macintosh, N. Vignais, E. Desailly, E. Biddiss, and V. Vigneron, “A Classification and Calibration Procedure for Gesture Specific Home-Based Therapy Exercise in Young People With Cerebral Palsy,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 29, pp. 144–155, 2021, doi: 10.1109/TNSRE.2020.3038370.

U. Farooq, M. S. Mohd Rahim, N. S. Khan, S. Rasheed, and A. Abid, “A Crowdsourcing-Based Framework for the Development and Validation of Machine Readable Parallel Corpus for Sign Languages,” IEEE Access, vol. 9, pp. 91788–91806, 2021, doi: 10.1109/ACCESS.2021.3091433.

K.-C. Lin and R.-J. Wai, “A Feasible Fall Evaluation System via Artificial Intelligence Gesture Detection of Gait and Balance for Sub-Healthy Community- Dwelling Older Adults in Taiwan,” IEEE Access, vol. 9, pp. 146404–146413, 2021, doi: 10.1109/ACCESS.2021.3123297.

N. Mohamed, M. B. Mustafa, and N. Jomhari, “A Review of the Hand Gesture Recognition System: Current Progress and Future Directions,” IEEE Access, vol. 9, pp. 157422–157436, 2021, doi: 10.1109/ACCESS.2021.3129650.

M. A. Bencherif et al., “Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data,” IEEE Access, vol. 9, pp. 59612–59627, 2021, doi: 10.1109/ACCESS.2021.3069714.

S. A. Raurale, J. McAllister, and J. M. D. Rincon, “EMG Biometric Systems Based on Different Wrist-Hand Movements,” IEEE Access, vol. 9, pp. 12256–12266, 2021, doi: 10.1109/ACCESS.2021.3050704.

E. Rahimian, S. Zabihi, A. Asif, D. Farina, S. F. Atashzar, and A. Mohammadi, “FS-HGR: Few-Shot Learning for Hand Gesture Recognition via Electromyography,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 29, pp. 1004–1015, 2021, doi: 10.1109/TNSRE.2021.3077413.

P. Nallabolu, L. Zhang, H. Hong, and C. Li, “Human Presence Sensing and Gesture Recognition for Smart Home Applications With Moving and Stationary Clutter Suppression Using a 60-GHz Digital Beamforming FMCW Radar,” IEEE Access, vol. 9, pp. 72857–72866, 2021, doi: 10.1109/ACCESS.2021.3080655.

D. F. Q. Melo, B. M. C. Silva, N. Pombo, and L. Xu, “Internet of Things Assisted Monitoring Using Ultrasound-Based Gesture Recognition Contactless System,” IEEE Access, vol. 9, pp. 90185–90194, 2021, doi: 10.1109/ACCESS.2021.3089940.

M. Altmann, P. Ott, N. C. Stache, and C. Waldschmidt, “Multi-Modal Cross Learning for an FMCW Radar Assisted by Thermal and RGB Cameras to Monitor Gestures and Cooking Processes,” IEEE Access, vol. 9, pp. 22295–22303, 2021, doi: 10.1109/ACCESS.2021.3056878.

A. R. Elshenaway and S. K. Guirguis, “On-Air Hand-Drawn Doodles for IoT Devices Authentication During COVID-19,” IEEE Access, vol. 9, pp. 161723–161744, 2021, doi: 10.1109/ACCESS.2021.3131551.

X. Jiang et al., “Open Access Dataset, Toolbox and Benchmark Processing Results of High-Density Surface Electromyogram Recordings,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 29, pp. 1035–1046, 2021, doi: 10.1109/TNSRE.2021.3082551.

M. Chmurski, M. Zubert, K. Bierzynski, and A. Santra, “Analysis of Edge-Optimized Deep Learning Classifiers for Radar-Based Gesture Recognition,” IEEE Access, vol. 9, pp. 74406–74421, 2021, doi: 10.1109/ACCESS.2021.3081353.

A. Pradhan, J. He, and N. Jiang, “Performance Optimization of Surface Electromyography Based Biometric Sensing System for Both Verification and Identification,” IEEE Sens. J., vol. 21, no. 19, pp. 21718–21729, 2021, doi: 10.1109/JSEN.2021.3079428.

N. Pan, “Research on Music Wireless Control Based on Motion Tracking Sensor and Internet of Things,” IEEE Access, vol. 9, pp. 48803–48810, 2021, doi: 10.1109/ACCESS.2021.3064565.

P. V. Rouast and M. T. P. Adam, “Single-Stage Intake Gesture Detection Using CTC Loss and Extended Prefix Beam Search,” IEEE J. Biomed. Health Inform., vol. 25, no. 7, pp. 2733–2743, 2021, doi: 10.1109/JBHI.2020.3046613.

Z. A. Kahar, P. S. Sulaiman, F. Khalid, and A. Azman, “Skeleton Joints Moment (SJM): A Hand Gesture Dimensionality Reduction for Central Nervous System Interaction,” IEEE Access, vol. 9, pp. 146640–146652, 2021, doi: 10.1109/ACCESS.2021.3123570.

M.-C. Su, J.-H. Chen, A. M. Arifai, S.-Y. Tsai, and H.-H. Wei, “Smart Living: An Interactive Control System for Household Appliances,” IEEE Access, vol. 9, pp. 14897–14904, 2021, doi: 10.1109/ACCESS.2021.3051253.

Z. Chen, J. Yang, and H. Xie, “Surface-Electromyography-Based Gesture Recognition Using a Multistream Fusion Strategy,” IEEE Access, vol. 9, pp. 50583–50592, 2021, doi: 10.1109/ACCESS.2021.3059499.

H. Zhou, C. Tawk, and G. Alici, “A 3D Printed Soft Robotic Hand With Embedded Soft Sensors for Direct Transition Between Hand Gestures and Improved Grasping Quality and Diversity,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 550–558, 2022, doi: 10.1109/TNSRE.2022.3156116.

R. Hu, X. Chen, H. Zhang, X. Zhang, and X. Chen, “A Novel Myoelectric Control Scheme Supporting Synchronous Gesture Recognition and Muscle Force Estimation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 1127–1137, 2022, doi: 10.1109/TNSRE.2022.3166764.

Q. Zengyu et al., “A Simultaneous Gesture Classification and Force Estimation Strategy Based on Wearable A-Mode Ultrasound and Cascade Model,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 2301–2311, 2022, doi: 10.1109/TNSRE.2022.3196926.

X. Song, S. S. Van De Ven, L. Liu, F. J. Wouda, H. Wang, and P. B. Shull, “Activities of Daily Living-Based Rehabilitation System for Arm and Hand Motor Function Retraining After Stroke,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 621–631, 2022, doi: 10.1109/TNSRE.2022.3156387.

S. B. Abdullahi and K. Chamnongthai, “American Sign Language Words Recognition Using Spatio-Temporal Prosodic and Angle Features: A Sequential Learning Approach,” IEEE Access, vol. 10, pp. 15911–15923, 2022, doi: 10.1109/ACCESS.2022.3148132.

A. Ali et al., “End-to-End Dynamic Gesture Recognition Using MmWave Radar,” IEEE Access, vol. 10, pp. 88692–88706, 2022, doi: 10.1109/ACCESS.2022.3199411.

J. Li, S. Ray, V. Rajanna, and T. Hammond, “Evaluating the Performance of Machine Learning Algorithms in Gaze Gesture Recognition Systems,” IEEE Access, vol. 10, pp. 1020–1035, 2022, doi: 10.1109/ACCESS.2021.3136153.

J.-W. Choi, C.-W. Park, and J.-H. Kim, “FMCW Radar-Based Real-Time Hand Gesture Recognition System Capable of Out-of-Distribution Detection,” IEEE Access, vol. 10, pp. 87425–87434, 2022, doi: 10.1109/ACCESS.2022.3200757.

T. Wang, Y. Zhao, and Q. Wang, “Hand Gesture Recognition With Flexible Capacitive Wristband Using Triplet Network in Inter-Day Applications,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 2876–2885, 2022, doi: 10.1109/TNSRE.2022.3212705.

Q. M. Areeb, Maryam, M. Nadeem, R. Alroobaea, and F. Anwer, “Helping Hearing-Impaired in Emergency Situations: A Deep Learning-Based Approach,” IEEE Access, vol. 10, pp. 8502–8517, 2022, doi: 10.1109/ACCESS.2022.3142918.

C. Canuto, E. O. Freire, L. Molina, E. A. N. Carvalho, and S. N. Givigi, “Intuitiveness Level: Frustration-Based Methodology for Human–Robot Interaction Gesture Elicitation,” IEEE Access, vol. 10, pp. 17145–17154, 2022, doi: 10.1109/ACCESS.2022.3146838.

J. E. Lara, L. K. Cheng, O. Rohrle, and N. Paskaranandavadivel, “Muscle-Specific High-Density Electromyography Arrays for Hand Gesture Classification,” IEEE Trans. Biomed. Eng., vol. 69, no. 5, pp. 1758–1766, 2022, doi: 10.1109/TBME.2021.3131297.

J. Li, J. Meng, H. Gong, and Z. Fan, “Research on Continuous Dynamic Gesture Recognition of Chinese Sign Language Based on Multi-Mode Fusion,” IEEE Access, vol. 10, pp. 106946–106957, 2022, doi: 10.1109/ACCESS.2022.3212064.

J. Xu, H. Wang, J. Zhang, and L. Cai, “Robust Hand Gesture Recognition Based on RGB-D Data for Natural Human–Computer Interaction,” IEEE Access, vol. 10, pp. 54549–54562, 2022, doi: 10.1109/ACCESS.2022.3176717.

B. Zhu, D. Zhang, Y. Chu, Y. Gu, and X. Zhao, “SeNic: An Open Source Dataset for sEMG-Based Gesture Recognition in Non-Ideal Conditions,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 1252–1260, 2022, doi: 10.1109/TNSRE.2022.3173708.

D. R. Kothadiya, C. M. Bhatt, T. Saba, A. Rehman, and S. A. Bahaj, “SIGNFORMER: DeepVision Transformer for Sign Language Recognition,” IEEE Access, vol. 11, pp. 4730–4739, 2023, doi: 10.1109/ACCESS.2022.3231130.

B. Fang et al., “Simultaneous sEMG Recognition of Gestures and Force Levels for Interaction With Prosthetic Hand,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 2426–2436, 2022, doi: 10.1109/TNSRE.2022.3199809.

Z. Pan, X. Qian, and H. Li, “Speaker Extraction With Co-Speech Gestures Cue,” IEEE Signal Process. Lett., vol. 29, pp. 1467–1471, 2022, doi: 10.1109/LSP.2022.3175130.

Z. Lu, S. Cai, B. Chen, Z. Liu, L. Guo, and L. Yao, “Wearable Real-Time Gesture Recognition Scheme Based on A-Mode Ultrasound,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 30, pp. 2623–2629, 2022, doi: 10.1109/TNSRE.2022.3205026.

F. Sufyan, S. Sagar, Z. Ashraf, S. Nayel, M. S. Chishti, and A. Banerjee, “A Novel and Lightweight Real-Time Continuous Motion Gesture Recognition Algorithm for Smartphones,” IEEE Access, vol. 11, pp. 42725–42737, 2023, doi: 10.1109/ACCESS.2023.3255402.

S. Khetavath et al., “An Intelligent Heuristic Manta-Ray Foraging Optimization and Adaptive Extreme Learning Machine for Hand Gesture Image Recognition,” Big Data Min. Anal., vol. 6, no. 3, pp. 321–335, 2023, doi: 10.26599/BDMA.2022.9020036.

B. Hampiholi, C. Jarvers, W. Mader, and H. Neumann, “Convolutional Transformer Fusion Blocks for Multi-Modal Gesture Recognition,” IEEE Access, vol. 11, pp. 34094–34103, 2023, doi: 10.1109/ACCESS.2023.3263812.

Q. Li, Z. Luo, R. Qi, and J. Zheng, “DeepTPA-Net: A Deep Triple Attention Network for sEMG-Based Hand Gesture Recognition,” IEEE Access, vol. 11, pp. 96797–96807, 2023, doi: 10.1109/ACCESS.2023.3312219.

S. Wan, L. Yang, K. Ding, and D. Qiu, “Dynamic Gesture Recognition Based on Three-Stream Coordinate Attention Network and Knowledge Distillation,” IEEE Access, vol. 11, pp. 50547–50559, 2023, doi: 10.1109/ACCESS.2023.3278100.

A. S. M. Miah, Md. A. M. Hasan, and J. Shin, “Dynamic Hand Gesture Recognition Using Multi-Branch Attention Based Graph and General Deep Learning Model,” IEEE Access, vol. 11, pp. 4703–4716, 2023, doi: 10.1109/ACCESS.2023.3235368.

Y. Ghasemi, H. Jeong, K.-B. Park, S. H. Choi, and J. Y. Lee, “Evaluating User Interactions in Wearable Extended Reality: Modeling, Online Remote Survey, and In-Lab Experimental Methods,” IEEE Access, vol. 11, pp. 77856–77872, 2023, doi: 10.1109/ACCESS.2023.3298598.

Y. Pang, Y. Gong, and X. Hao, “Hand Gesture Recognition Based on Point Cloud Sequences and Inverse Kinematics Model,” IEEE Access, vol. 11, pp. 44082–44091, 2023, doi: 10.1109/ACCESS.2023.3272746.

H. Ansar et al., “Hand Gesture Recognition for Characters Understanding Using Convex Hull Landmarks and Geometric Features,” IEEE Access, vol. 11, pp. 82065–82078, 2023, doi: 10.1109/ACCESS.2023.3300712.

H.-Q. Nguyen et al., “Hand Gesture Recognition From Wrist-Worn Camera for Human–Machine Interaction,” IEEE Access, vol. 11, pp. 53262–53274, 2023, doi: 10.1109/ACCESS.2023.3279845.

W. E. Villegas-Ch, J. García-Ortiz, and S. Sánchez-Viteri, “Identification of Emotions From Facial Gestures in a Teaching Environment With the Use of Machine Learning Techniques,” IEEE Access, vol. 11, pp. 38010–38022, 2023, doi: 10.1109/ACCESS.2023.3267007.

S. B. Abdullahi and K. Chamnongthai, “IDF-Sign: Addressing Inconsistent Depth Features for Dynamic Sign Word Recognition,” IEEE Access, vol. 11, pp. 88511–88526, 2023, doi: 10.1109/ACCESS.2023.3305255.

K. Wang, Y. Chen, Y. Zhang, X. Yang, and C. Hu, “Iterative Self-Training Based Domain Adaptation for Cross-User sEMG Gesture Recognition,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 31, pp. 2974–2987, 2023, doi: 10.1109/TNSRE.2023.3293334.

Y. Liu, Q. Qiao, S. Shi, X. Wang, M. Yang, and X. Gao, “Keyframe Extraction and Process Recognition Method for Assembly Operation Based on Density Clustering,” IEEE Access, vol. 11, pp. 13564–13573, 2023, doi: 10.1109/ACCESS.2023.3243083.

A. Raza, A. M. Qadri, I. Akhtar, N. A. Samee, and M. Alabdulhafith, “LogRF: An Approach to Human Pose Estimation Using Skeleton Landmarks for Physiotherapy Fitness Exercise Correction,” IEEE Access, vol. 11, pp. 107930–107939, 2023, doi: 10.1109/ACCESS.2023.3320144.

J. Ma, X. Ren, H. Li, W. Li, V. Y. Tsviatkou, and A. A. Boriskevich, “Noise-Against Skeleton Extraction Framework and Application on Hand Gesture Recognition,” IEEE Access, vol. 11, pp. 9547–9559, 2023, doi: 10.1109/ACCESS.2023.3240313.

A. Nimbekar, Y. V. S. Dinesh, A. Gautam, V. Hunsigida, A. R. Nali, and A. Acharyya, “Reconfigurable VLSI Design Architecture for Deep Learning Established Forelimb and Hindlimb Gesture Recognition for Rehabilitation Application,” IEEE Access, vol. 11, pp. 70061–70070, 2023, doi: 10.1109/ACCESS.2023.3293422.

Y. Lin, R. Palaniappan, P. De Wilde, and L. Li, “Robust Long-Term Hand Grasp Recognition With Raw Electromyographic Signals Using Multidimensional Uncertainty-Aware Models,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 31, pp. 962–971, 2023, doi: 10.1109/TNSRE.2023.3236982.

E. F. Asl, S. Ebadollahi, R. Vahidnia, and A. Jalali, “Statistical Database of Human Motion Recognition Using Wearable IoT—A Review,” IEEE Sens. J., vol. 23, no. 14, pp. 15253–15304, 2023, doi: 10.1109/JSEN.2023.3282171.

S. Zhang and S. Zhang, “A Novel Human-3DTV Interaction System Based on Free Hand Gestures and a Touch-Based Virtual Interface,” IEEE Access, vol. 7, pp. 165961–165973, 2019, doi: 10.1109/ACCESS.2019.2953798.

E. S. Goh, M. S. Sunar, and A. W. Ismail, “3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review,” IEEE Access, vol. 7, pp. 40581–40601, 2019, doi: 10.1109/ACCESS.2019.2906394.

S. Jacob, V. G. Menon, F. Al-Turjman, V. P. G., and L. Mostarda, “Artificial Muscle Intelligence System With Deep Learning for Post-Stroke Assistance and Rehabilitation,” IEEE Access, vol. 7, pp. 133463–133473, 2019, doi: 10.1109/ACCESS.2019.2941491.

H. Brock, J. Ponce Chulani, L. Merino, D. Szapiro, and R. Gomez, “Developing a Lightweight Rock-Paper-Scissors Framework for Human-Robot Collaborative Gaming,” IEEE Access, vol. 8, pp. 202958–202968, 2020, doi: 10.1109/ACCESS.2020.3033550.

A. Cisnal, J. Perez-Turiel, J.-C. Fraile, D. Sierra, and E. De La Fuente, “RobHand: A Hand Exoskeleton With Real-Time EMG-Driven Embedded Control. Quantifying Hand Gesture Recognition Delays for Bilateral Rehabilitation,” IEEE Access, vol. 9, pp. 137809–137823, 2021, doi: 10.1109/ACCESS.2021.3118281.

J. A. Mahmud, B. C. Das, J. Shin, K. Md. Hasib, R. Sadik, and M. F. Mridha, “3D Gesture Recognition and Adaptation for Human–Robot Interaction,” IEEE Access, vol. 10, pp. 116485–116513, 2022, doi: 10.1109/ACCESS.2022.3218679.

A. Rachmad, M. Fuad, and E. M. S. Rochman, “Convolutional Neural Network-Based Classification Model of Corn Leaf Disease,” Math. Model. Eng. Probl., vol. 10, no. 2, pp. 530–536, 2023, doi: 10.18280/mmep.100220.

X. Guan, J. Huang, and T. Tang, “Robot vision application on embedded vision implementation with digital signal processor,” International Journal of Advanced Robotic Systems, vol. 17, no. 1, 2020, doi:10.1177/1729881419900437.

B. Zhang, C. Quan and F. Ren, "Study on CNN in the recognition of emotion in audio and images," 2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS), pp. 1-5, 2016, doi: 10.1109/ICIS.2016.7550778.

J. Chang, J. Xiao, J. Chai, and Z. Zhou, “An Improved Faster R-CNN Algorithm for Gesture Recognition in Human-Robot Interaction,” in 2019 Chinese Automation Congress (CAC), pp. 5761–5764, 2019, doi: 10.1109/CAC48633.2019.8997339.




DOI: https://doi.org/10.18196/jrc.v4i6.20399

Refbacks

  • There are currently no refbacks.


Copyright (c) 2023 Muhammad Fuad, Faikul Umam, Sri Wahyuni, Nuniek Fahriani, Ilham Nurwahyudi, Mochammad Ilham Darwaman, Fahmi Maulana

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

 

Journal of Robotics and Control (JRC)

P-ISSN: 2715-5056 || E-ISSN: 2715-5072
Organized by Peneliti Teknologi Teknik Indonesia
Published by Universitas Muhammadiyah Yogyakarta in collaboration with Peneliti Teknologi Teknik Indonesia, Indonesia and the Department of Electrical Engineering
Website: http://journal.umy.ac.id/index.php/jrc
Email: jrcofumy@gmail.com

Kuliah Teknik Elektro Terbaik