Path Following and Avoiding Obstacle for Mobile Robot Under Dynamic Environments Using Reinforcement Learning

Le Duc Hanh, Vo Duy Cong

Abstract


Obstacle avoidance for mobile robot to reach the desired target from a start location is one of the most interesting research topics. However, until now, few works discuss about working of mobile robot in the dynamic and continuously changing environment. So, this issue is still the research challenge for mobile robots. Traditional algorithm for obstacle avoidance in the dynamic, complex environment had many drawbacks. As known that Q-learning, the type of reinforcement learning, has been successfully applied in computer games. However, it is still rarely used in real world applications. This research presents an effectively method for real time dynamic obstacle avoidance based on Q-learning in the real world by using three-wheeled mobile robot. The position of obstacles including many static and dynamic obstacles and the mobile robot are recognized by fixed camera installed above the working space. The input for the robot is the 2D data from the camera. The output is an action for the robot (velocities, linear and angular parameters). Firstly, the simulation is performed for Q-learning algorithm then based on trained data, The Q-table value is implemented to the real mobile robot to perform the task in the real scene. The results are compared with intelligent control method for both static and dynamic obstacles cases. Through implement experiments, the results show that, after training in dynamic environments and testing in a new environment, the mobile robot is able to reach the target position successfully and have better performance comparing with fuzzy controller.

Keywords


Path Following; Avoiding Obstacle; Mobile Robot; Reinforcement Learning; Q-learning.

Full Text:

PDF

References


A. Mora, A. Prados, A. Mendez, R. Barber, and S. Garrido, “Sensor Fusion for Social Navigation on a Mobile Robot Based on Fast Marching Square and Gaussian Mixture Model,” Sensors, vol. 22, no. 22, p. 8728, 2022.

R. M. Sousa, D. B-. Aranibar, J. D-. Amado, R.E. P-. Escarcina, and R. M. P Trindade, “A New Approach for Including Social Conventions into Social Robots Navigation by Using Polygonal Triangulation and Group Asymmetric Gaussian Functions,” Sensors, vol. 22, no. 12, p. 4602, 2022.

L. C. Santos, A. S. Aguiar, F. N. Santos, A. Valente, and M. Petry, “Occupancy Grid and Topological Maps Extraction from Satellite Images for Path Planning in Agricultural Robots,” Robotics, vol. 9, no. 4, p. 77, 2020.

Y. Gao, C. Bai, R. Fu, and Q. Quan, “A non-potential orthogonal vector field method for more efficient robot navigation and control,” Robotics and Autonomous Systems, vol. 159, p. 104291, 2023.

M. Santilli, P. Mukherjee, R. K. Williams, and A. Gasparri, “Multirobot Field of View Control with Adaptive Decentralization,” IEEE Transactions on Robotics, vol. 38, no. 4, pp. 2131-2150, 2022.

Y. Zhao, T. Wang and W. Bi, “Consensus Protocol for Multiagent Systems with Undirected Topologies and Binary-Valued Communications,” in IEEE Transactions on Automatic Control, vol. 64, no. 1, pp. 206-221, 2019.

D. Gadjov and L. Pavel, “A Passivity-Based Approach to Nash Equilibrium Seeking Over Networks,” in IEEE Transactions on Automatic Control, vol. 64, no. 3, pp. 1077-1092, 2019.

Y. Hua, X. Dong, Q. Li, and Z. Ren, “Distributed Fault-Tolerant Time-Varying Formation Control for Second-Order Multi-Agent Systems with Actuator Failures and Directed Topologies,” in IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 65, no. 6, pp. 774-778, 2018.

A. Carrio, J. Tordesillas, S. Vemprala, S. Saripalli, P. Campoy and J. P. How, “Onboard Detection and Localization of Drones Using Depth Maps,” in IEEE Access, vol. 8, pp. 30480-30490, 2020.

H. Seo, G. Cho, S. -J. Kim, J. -H. Chun, and J. Choi, “Multievent Histogramming TDC With Pre–Post Weighted Histogramming Filter for CMOS LiDAR Sensors,” in IEEE Sensors Journal, vol. 22, no. 23, pp. 22785-22798, 2022.

Z. Tahir, A. H. Qureshi, Y. Ayaz, and R. Nawaz, “Potentially guided bidirectionalized RRT* for fast optimal path planning in cluttered environments,” Robotics and Autonomous Systems, vol. 108, pp. 13–27, 2018.

K. M. I. Khalilullah, S. Ota, T. Yasuda, and M. Jindai, “Road area detection method based on DBNN for robot navigation using single camera in outdoor environments,” Industrial Robot, vol. 45, no. 2, pp. 275-286, 2018.

G. Lei, R. Yao, Y. Zhao, and Y. Zheng, “Detection and Modeling of Unstructured Roads in Forest Areas Based on Visual-2D Lidar Data Fusion,” Forests, vol. 12, no. 7, p. 820, 2021.

F. Amorós, L. Payá, W. Mayol-Cuevas, L. M. Jiménez, and O. Reinoso, “Holistic Descriptors of Omnidirectional Color Images and Their Performance in Estimation of Position and Orientation,” in IEEE Access, vol. 8, pp. 81822-81848, 2020.

A. W. L. Yao and H. C. Chen, “An Intelligent Color Image Recognition and Mobile Control System for Robotic Arm,” International Journal of Robotics and Control Systems, vol. 2, no. 1, pp. 97-104, 2022.

I. Hassani, I. Ergui, and C. Rekik, “Turning Point and Free Segments Strategies for Navigation of Wheeled Mobile Robot,” International Journal of Robotics and Control Systems, vol. 2, no. 1, pp. 172-186, 2022.

G. Farid et al., “Modified A-Star (A*) Approach to Plan the Motion of a Quadrotor UAV in Three-Dimensional Obstacle-Cluttered Environment,” Appl. Sci., vol. 12, no. 12, p. 5791, 2022.

T. -Y. Lin, K. -R. Wu, Y. -S. Chen, and Y. -S. Shen, “Collision-Free Motion Algorithms for Sensors Automated Deployment to Enable a Smart Environmental Sensing-Net,” in IEEE Transactions on Automation Science and Engineering, vol. 19, no. 4, pp. 3853-3870, 2022.

K. Li, Q. Hu, and J. Liu, “Path planning of mobile robot based on improved multiobjective genetic algorithm,” Wireless Communications and Mobile Computing, vol. 2021, pp. 1-12, 2021.

K. Hao, J. Zhao, B. Wang, Y. Liu, and C. Wang, “The application of an adaptive genetic algorithm based on collision detection in path planning of mobile robots,” Computational Intelligence and Neuroscience, vol. 2021, pp. 1-20, 2021.

I. Noreen, A. Khan, K. Asghar, and Z. Habib, “A path-planning performance comparison of RRT*-AB with MEA* in a 2-dimensional environment,” Symmetry, vol. 11, no. 7, pp. 945-960, 2019.

J. Wang, W. Chi, C. Li, C. Wang, and M. Q. -H. Meng, “Neural RRT*: Learning-Based Optimal Path Planning,” in IEEE Transactions on Automation Science and Engineering, vol. 17, no. 4, pp. 1748-1758, 2020.

Y. Bai, G. Li, and N. Li, “Motion Planning and Tracking Control of Autonomous Vehicle Based on Improved A∗ Algorithm,” Journal of Advanced Transportation, vol. 2022, pp.1-14, 2022.

H. Min, X. Xiong, P. Wang, and Y. Yu, “Autonomous driving path planning algorithm based on improved algorithm in unstructured environment,” Proceedings of the Institution of Mechanical Engineers - Part D: Journal of Automobile Engineering, vol. 235, no. 2-3, pp. 513–526, 2021.

B. Fu et al., “An improved A* algorithm for the industrial robot path planning with high success rate and short length,” Robotics and Autonomous Systems, vol. 106, pp. 26–37, 2018.

N. B. A. Latip, R. Omar, and S. K. Debnath, “Optimal path planning using equilateral spaces-oriented visibility graph method,” Intl. J. Electr. Comput. Eng., vol. 7, no. 6, pp. 3046–3051, 2017.

L. Janson, E. Schmerling, A. Clark, and M. Pavone, “Fast marching tree: A fast marching sampling-based method for optimal motion planning in many dimensions,” The International journal of robotics research, vol. 34, no. 7, pp. 883–921, 2015.

O. Montiel, U. Orozco-Rosas, and R. Seplveda, “Path planning for mobile robots using bacterial potential field for avoiding static and dynamic obstacles,” Expert Syst. Appl., vol. 42, no.12, pp. 5177–5191, 2015.

A.H. Karami and M. Hasanzadeh, “An adaptive genetic algorithm for robot motion planning in 2D complex environments,” Computers & Electrical Engineering, vol. 43, pp. 317-329, 2015.

F. Kamil, T.S. Hong, W. Khaksar, M.Y. Moghrabiah, N. Zulkifli, and S.A. Ahmad, “New robot navigation algorithm for arbitrary unknown dynamic environments based on future prediction and priority behavior,” Expert Syst. Appl., vol. 86, pp. 274–291, 2017.

M. Alajlan, I. Chaari, A. Koubaa, H. Bennaceur, A. Ammar, and H. Youssef, “Global robot path planning using GA for large grid maps: Modelling performance and experimentation,” International Journal of Robotics and Automation, vol. 31, no. 6, pp. 484-495, 2016.

Y. Zhao, X. Liu, G. Wang, S. Wu, and S. Han, “Dynamic Resource Reservation Based Collision and Deadlock Prevention for Multi-AGVs,” IEEE Access, vol. 8, pp. 82120-82130, 2020.

L. Yue and H. Fan, “Dynamic Scheduling and Path Planning of Automated Guided Vehicles in Automatic Container Terminal,” in IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 11, pp. 2005-2019, 2022.

H. Xiao, X. Wu, D. Qin, and J. Zhai, “A Collision and Deadlock Prevention Method with Traffic Sequence Optimization Strategy for UGN-Based AGVS,” in IEEE Access, vol. 8, pp. 209452-209470, 2020.

M. Faisal, R. Hedjar, M. Al-Sulaiman, and K. Al-Mutib, “Fuzzy Logic Navigation and Obstacle Avoidance by a Mobile Robot in an Unknown Dynamic Environment,” International Journal of Advanced Robotic Systems, vol. 10, no. 1, 2013.

A. Shitsukane, W. Cheruiyot, C. Otieno and M. Mvurya, “Fuzzy Logic Sensor Fusion for Obstacle Avoidance Mobile Robot,” 2018 IST-Africa Week Conference (IST-Africa), pp. 1-8, 2018.

S. -Y. Chiang, “Vision-based obstacle avoidance system with fuzzy logic for humanoid robots,” The Knowledge Engineering Review, vol. 32, no. E9, 2017.

S. Ayub, N. Singh, M. Z. Hussain, M. Ashraf, D. K. Singh, and A. Haldorai, “Hybrid approach to implement multi-robotic navigation system using neural network, fuzzy logic, and bio-inspired optimization methodologies,” Computational Intelligence, pp. 1- 15, 2022.

K. Farah and M. Y. Mohammed, “Multilayer Decision-Based Fuzzy Logic Model to Navigate Mobile Robot in Unknown Dynamic Environments”, Fuzzy Information and Engineering, vol. 14, no. 1, pp. 51-73, 2021.

H. Muhammad et al., “Robust mobile robot navigation in cluttered environments based on hybrid adaptive neuro-fuzzy inference and sensor fusion,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 10, pp. 9060-9070, 2022.

F. Á. Szili, J. Botzheim, and B. Nagy, “Bacterial Evolutionary Algorithm-Trained Interpolative Fuzzy System for Mobile Robot Navigation”, Electronics, vol. 11, no. 11, p. 1734, 2022.

S. Zubair, S. ABU, R. Ruzairi, A. Andi, and H. Mohd, “Non-Verbal Human-Robot Interaction Using Neural Network for The Application of Service Robot,” IIUM Engineering Journal, vol. 24, no. 1, pp. 301-318, 2023.

U.A. Syed, F. Kunwar, and M. Iqbal, “Guided autowave pulse coupled neural network (GAPCNN) based real time path planning and an obstacle avoidance scheme for mobile robots,” Robotics and Autonomous Systems, vol. 62, no. 4, pp. 474-486, 2014.

K. -H. Chi and M. -F. R. Lee, “Obstacle avoidance in mobile robot using Neural Network,” 2011 International Conference on Consumer Electronics, Communications and Networks (CECNet), pp. 5082-5085, 2011.

A. Medina-Santiago, J. L. Camas-Anzueto, J. A. Vazquez-Feijoo, H. R. Hernández-de León, R. Mota-Grajales, “Neural Control System in Obstacle Avoidance in Mobile Robots Using Ultrasonic Sensors,” Journal of Applied Research and Technology, vol. 12, no. 1, pp. 104-110, 2014.

A. N. A. Rafai, N. Adzhar, and N. I. Jaini, “A Review on Path Planning and Obstacle Avoidance Algorithms for Autonomous Mobile Robots,” Journal of Robotics, vol. 2022, pp. 1-14, 2022.

F. Qu, W. Yu, K. Xiao, C. Liu, and W. Liu, “Trajectory Generation and Optimization Using the Mutual Learning and Adaptive Ant Colony Algorithm in Uneven Environments”, Applied Sciences, vol. 12, no. 9, p. 4629, 2022. Doi: 10.3390/app12094629.

A. Muhammad, M. A. H. Ali, and I. H. Shanono, “Path Planning Methods for Mobile Robots: A systematic and Bibliometric Review”, ELEKTRIKA- Journal of Electrical Engineering, vol. 19, no. 3, pp. 14-34, 2020.

D. Zhu, C. Tian, B. Sun, and C. Luo, “Complete Coverage Path Planning of Autonomous Underwater Vehicle Based on GBNN Algorithm,” Journal of Intelligent & Robotic Systems, vol. 94, no. 237–249, 2019.

C. Sun, W. He, and J. Hong, “Neural Network Control of a Flexible Robotic Manipulator Using the Lumped Spring-Mass Model,” in IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 47, no. 8, pp. 1863-1874, 2017.

A. Kumar, P. B. Kumar, and D. R. Parhi, “Intelligent Navigation of Humanoids in Cluttered Environments Using Regression Analysis and Genetic Algorithm,” Arabian Journal for Science and Engineering, vol. 43, pp. 7655–7678, 2018.

A. K. Abbas, Y. A. Mashhadany, M. J. Hameed, and S. Algburi, “Review of Intelligent Control Systems with Robotics,” Indonesian Journal of Electrical Engineering and Informatics (IJEEI), vol. 10, no. 4, pp. 734-753, 2022.

M. Naeem, S. T. H. Rizvi, and A. Coronato, “A Gentle Introduction to Reinforcement Learning and its Application in Different Fields,” in IEEE Access, vol. 8, pp. 209320-209344, 2020.

M. Naeem, A. Coronato, Z. Ullah, S. Bashir, and G. Paragliola, “Optimal User Scheduling in Multi Antenna System Using Multi Agent Reinforcement Learning,” Sensors, vol. 22, no. 21, p. 8278, 2022.

W. Kumwilaisak, S. Phikulngoen, J. Piriyataravet, N. Thatphithakkul, and C. Hansakunbuntheung, “Adaptive Call Center Workforce Management with Deep Neural Network and Reinforcement Learning,” in IEEE Access, vol. 10, pp. 35712-35724, 2022.

A. Alwarafy, M. Abdallah, B. S. Çiftler, A. Al-Fuqaha, and M. Hamdi, “The Frontiers of Deep Reinforcement Learning for Resource Management in Future Wireless HetNets: Techniques, Challenges, and Research Directions,” in IEEE Open Journal of the Communications Society, vol. 3, pp. 322-365, 2022.

B. Jang, M. Kim, G. Harerimana, and J. W. Kim, “Q-Learning Algorithms: A Comprehensive Classification and Applications,” in IEEE Access, vol. 7, pp. 133653-133667, 2019.

N. Sutisna, A. M. R. Ilmy, I. Syafalni, R. Mulyawan, and T. Adiono, “FARANE-Q: Fast Parallel and Pipeline Q-Learning Accelerator for Configurable Reinforcement Learning SoC,” in IEEE Access, vol. 11, pp. 144-161, 2023.

N. Sutisna, Z. N. Arifuzzaki, I. Syafalni, R. Mulyawan, and T. Adiono, “Architecture Design of Q-Learning Accelerator for Intelligent Traffic Control System,” 2022 International Symposium on Electronics and Smart Devices (ISESD), pp. 1-6, 2022.

A. Chandrakar and P. Paliwal, “An Intelligent Mechanism for Utility and Active Customers in Demand Response Using Single and Double Q Learning Approach,” Smart Energy and Advancement in Power Technologies, vol. 926, pp. 397-413, 2023.

J. Raajan, P.V. Srihari, J. P. Satya, B. Bhikkaji, and R. Pasumarthy, “Real Time Path Planning of Robot using Deep Reinforcement Learning,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 15602-15607, 2020.

R. Cimurs, J. H. Lee, and I. H. Suh, “Goal-Oriented Obstacle Avoidance with Deep Reinforcement Learning in Continuous Action Space,” Electronics, vol. 9, no. 3, p. 411, 2020.

L. Huang, H. Qu, M. Fu, and W. Deng, “Reinforcement Learning for Mobile Robot Obstacle Avoidance Under Dynamic Environments,” PRICAI 2018: Trends in Artificial Intelligence, vol. 11012, 2018.

A. D. Pambudi, T. Agustinah, and R. Effendi, “Reinforcement Point and Fuzzy Input Design of Fuzzy Q-Learning for Mobile Robot Navigation System,” 2019 International Conference of Artificial Intelligence and Information Technology (ICAIIT), pp. 186-191, 2019.

S. Wen, X. Hu, Z. Li, H.K. Lam, F. Sun, and B. Fang, “NAO robot obstacle avoidance based on fuzzy Q-learning,” Industrial Robot, vol. 47, no. 6, pp. 801-811, 2020.

D. Lj debeliković, V. B. Bajić, T. N. Erić, and S. A. Milinković, “A Lyapunov analysis of stability robustness for discrete linear descriptor systems,” IMA Journal of Mathematical Control and Information, vol. 15, no. 1, pp. 53–62, 1998.

J. Chang and G. Kwon, “Analysis of Hyper-parameters in Solving Sokoban using Q-learning,” The Journal of Korean Institute of Information Technology, vol. 20, no. 11, pp. 67-72, 2022.

S. Suratkar et al, “An adaptive honeypot using Q-Learning with severity analyzer,” Journal of Ambient Intelligence and Humanized Computing, vol. 13, no. 10, pp. 4865-4876, 2022.

K. Cheena, T. Amgoth, and G. Shankar, “Deep Q-probabilistic algorithm based rock hyraxes swarm optimization for channel allocation in CRSN smart grids,” Wireless Networks, vol. 28, no. 6, pp. 2553-2565, 2022.

A. M. Rahmani et al., “An energy-aware and Q-learning-based area coverage for oil pipeline monitoring systems using sensors and Internet of Things,” Scientific Reports, vol. 12, no. 1, 2022.

J. Lansky et al., “A Q-learning-based routing scheme for smart air quality monitoring system using flying ad hoc networks,” Scientific Reports, vol. 12, no. 1, 2022.

C. M. Horváth, J. Botzheim, T. Thomessen, and P. Korondi, “Bead geometry modeling on uneven base metal surface by fuzzy systems for multi-pass welding,” Expert Systems with Applications, vol. 186, pp. 115356, 2021.

M. Al-Mallah, M. Ali, and M. Al-Khawaldeh, “Obstacles Avoidance for Mobile Robot Using Type-2 Fuzzy Logic Controller,” Robotics, vol. 11, no. 6, p. 130, 2022.

M. Khairudin, R. Refalda, S. Yatmono, H. S. Pramono, A. K. Triatmaja, and A. Shah, “The Mobile Robot Control in Obstacle Avoidance Using Fuzzy Logic Controller,” Indones. J. Sci. Technol., vol. 5, no. 3, pp. 334–353, 2020.

S. Ayub, N. Singh, M. Z. Hussain, M. Ashraf, D. K. Singh, and A. Haldorai, “Hybrid approach to implement multi-robotic navigation system using neural network, fuzzy logic, and bio-inspired optimization methodologies,” Computational Intelligence, pp. 1-15, 2022.

N. Zagradjanin, A. Rodic, D. Pamucar, and B. Pavkovic, “Cloud-Based Multi-Robot Path Planning in Complex and Crowded Environment Using Fuzzy Logic and Online Learning,” Information Technology and Control, vol. 50, no. 2, pp. 357-374, 2021.

V. Sangeetha et al., “A Fuzzy Gain-Based Dynamic Ant Colony Optimization for Path Planning in Dynamic Environments,” Symmetry, vol. 13, no. 2, p. 280, 2021.




DOI: https://doi.org/10.18196/jrc.v4i2.17368

Refbacks

  • There are currently no refbacks.


Copyright (c) 2023 Cong Duy Vo

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

 

Journal of Robotics and Control (JRC)

P-ISSN: 2715-5056 || E-ISSN: 2715-5072
Organized by Peneliti Teknologi Teknik Indonesia
Published by Universitas Muhammadiyah Yogyakarta in collaboration with Peneliti Teknologi Teknik Indonesia, Indonesia and the Department of Electrical Engineering
Website: http://journal.umy.ac.id/index.php/jrc
Email: jrcofumy@gmail.com

Kuliah Teknik Elektro Terbaik