Machine Learning Paradigms for UAV Path Planning: Review and Challenges
DOI:
https://doi.org/10.18196/jrc.v6i1.24097Keywords:
UAV, Path Planning, Path Optimization, Machine Learning, Autonomous Navigation, Supervised Learning, Reinforcement Learning, Unsupervised Learning, Deep LearningAbstract
Path planning is a crucial step in robotic navigation to satisfy: tasks safety, efficiency requirements and adapt to the complexity of environments. Path planning problem is particularly critical for Unmanned Aerial Vehicles (UAV), being increasingly involved within important tasks in diverse military and civil fields such as: inspection, search and rescue and communication, taking advantage of their high flexibility, maneuverability and cost-effective solutions. This continuous growth made the solution of UAV path planning problem an interesting research topic in recent years. In this scope, machine learning algorithms were a promising tool due to their continuous data-driven selfimprovement to adapt with the high dynamicity of environments where conventional programming fails. This paper provides a review on recent developments in machine learning-based UAV path planning issued from credible databases like: IEEE, Elsevier, Springer Links and MDPI. The main contribution of this paper is to delve through these recent works providing a taxonomy of algorithms into the fundamental paradigms: supervised, unsupervised and reinforcement, evaluating their efficiency and limitations under distinct scenarios. Despite the relative generalization of deep reinforcement learning to different environments, this study highlighted some active challenges about computational cost and real-time applications that remain open.
References
J. Fan and M. A. Saadeghvaziri, “Applications of drones in infrastructures: Challenges and opportunities,” International Journal of Mechanical and Mechatronics Engineering, vol. 13, no. 10, pp. 649–655, 2019, doi: 10.5281/zenodo.3566281.
H. Shakhatreh et al., “Unmanned Aerial Vehicles (UAVs): A Survey on Civil Applications and Key Research Challenges,” in IEEE Access, vol. 7, pp. 48572-48634, 2019, doi: 10.1109/ACCESS.2019.2909530.
P. Matyas and N. Mate, “Brief history of uav development,” Repulestudomanyi Kozlemenyek, vol. 31, no. 1, pp. 155–166, 2019, doi: 10.32560/rk.2019.1.13.
G. Muchiri and S. Kimathi, “A review of applications and potential applications of uav,” in Proceedings of the Sustainable Research and Innovation Conference, pp. 280–283, 2022.
A. Rejeb, A. Abdollahi, K. Rejeb, and H. Treiblmaier, “Drones in agriculture: A review and bibliometric analysis,” Computers and Electronics in Agriculture, vol. 198, 2022, doi: 10.1016/j.compag.2022.107017.
J. Fan and M. A. Saadeghvaziri, “Applications of drones in infrastructures: Challenges and opportunities,” International Journal of Mechanical and Mechatronics Engineering, vol. 13, no. 10, pp. 649–655, 2019, doi: 10.5281/zenodo.3566281.
S. M. S. Mohd Daud et al., “Applications of drone in disaster management: A scoping review,” Science & Justice, vol. 62, no. 1, pp. 30–42, 2022, doi: 10.1016/j.scijus.2021.11.002.
A. Puente-Castro, D. Rivero, A. Pazos, and E. Fernandez-Blanco, “A review of artificial intelligence applied to path planning in UAV swarms,” vol. 34, no. 1, pp. 153–170, 2022, doi: 10.1007/s00521-021-06569-4.
A. Gasparetto, P. Boscariol, A. Lanzutti, and R. Vidoni, “Path planning and trajectory planning algorithms: A general overview,” Motion and Operation Planning of Robotic Systems: Background and Practical Approaches, pp. 3–27, 2015, doi: 10.1007/978-3-319-14705-5_1.
X. Li, J. Tupayachi, A. Sharmin, and M. Martinez Ferguson, “Drone-aided delivery methods, challenge, and the future: A methodological review,” Drones, vol. 7, no. 3, 2023, doi: 10.3390/drones7030191.
R. Masroor, M. Naeem, and W. Ejaz, “Resource management in uav-assisted wireless networks: An optimization perspective,” Ad Hoc Networks, vol. 121, 2021, doi: 10.1016/j.adhoc.2021.102596.
K. Okumura, F. Bonnet, Y. Tamura and X. Defago, “Offline ´ Time-Independent Multiagent Path Planning,” in IEEE Transactions on Robotics, vol. 39, no. 4, pp. 2720-2737, 2023, doi: 10.1109/TRO.2023.3258690.
H. Sun, W. Zhang, R. Yu and Y. Zhang, “Motion Planning for Mobile Robots—Focusing on Deep Reinforcement Learning: A Systematic Review,” in IEEE Access, vol. 9, pp. 69061-69081, 2021, doi: 10.1109/ACCESS.2021.3076530.
M. Krichen, “Strengthening the security of smart contracts through the power of artificial intelligence,” Computers, vol. 12, no. 5, 2023, doi: 10.3390/computers12050107.
R. Choi, A. Coyner, J. Kalpathy-Cramer, M. Chiang, and J. Campbell, “Introduction to machine learning, neural networks, and deep learning,” Translational vision science & technology, vol. 9, no. 2, 2020, doi: 10.1167/tvst.9.2.14.
A. Al-Kaff, D. Mart´ın, F. Garc´ıa, A. de la Escalera, and J. M. Armingol, “Survey of computer vision algorithms and applications for unmanned aerial vehicles,” Expert Systems with Applications, vol. 92, pp. 447–463, 2018, doi: 10.1016/j.eswa.2017.09.033.
V. S. Ajith and K. Jolly, “Unmanned aerial systems in search and rescue applications with their path planning: a review,” Journal of Physics: Conference Series, vol. 2115, no. 1, 2021, doi: 10.1088/1742-6596/2115/1/012020.
M. Hooshyar and Y.-M. Huang, “Meta-heuristic algorithms in uav path planning optimization: A systematic review (2018–2022),” Drones, vol. 7, no. 12, 2023, doi: 10.3390/drones7120687.
H. S. Yahia and A. S. Mohammed, “Path planning optimization in unmanned aerial vehicles using meta-heuristic algorithms: a systematic review,” Environmental Monitoring and Assessment, vol. 195, no. 30, 2023, doi: 10.1007/s10661-022-10590-y.
A. Ait Saadi, A. Soukane, Y. Meraihi, A. Benmessaoud Gabis, S. Mirjalili, and A. Ramdane-Cherif, “UAV path planning using optimization approaches: A survey,” Archives of Computational Methods in Engineering, vol. 29, pp. 4233–4284, 2022, doi: 10.1007/s11831-022-09742-7.
Y. Zhang, W. Zhao, J. Wang, and Y. Yuan, “Recent progress, challenges and future prospects of applied deep reinforcement learning : A practical perspective in path planning,” Neurocomputing, vol. 608, 2024, doi: 10.1016/j.neucom.2024.128423.
M. Al-Shareeda, M. Ali, and S. Manickam, “Unmanned aerial vehicle: a review and future directions,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 30, no. 2, pp. 778–786, 2023.
M. Akhloufi, S. Arola, and A. Bonnet, “Drones chasing drones: Reinforcement learning and deep search area proposal,” drone, vol. 3, no. 3, 2019, doi: 10.3390/drones3030058.
A. Otto, N. Agatz, J. Campbell, B. Golden, and E. Pesch, “Optimization approaches for civil applications of unmanned aerial vehicles (uavs) or aerial drones: A survey,” Networks, vol. 72, no. 4, pp. 411–458, 2018, doi: 10.1002/net.21818.
C. Lee, S. Kim, and B. Chu, “A survey: Flight mechanism and mechanical structure of the UAV,” vol. 22, no. 4, pp. 719–743, 2021, doi: 10.1007/s12541-021-00489-y.
M. Hassanalian and A. Abdelkefi, “Classifications, applications, and design challenges of drones: A review,” Progress in Aerospace Sciences, vol. 91, pp. 99–131, 2017, doi: 10.1016/j.paerosci.2017.04.003.
A. Watts, V. Ambrosia, and E. Hinkley, “Unmanned aircraft systems in remote sensing and scientific research: Classification and considerations of use,” remote sensing, vol. 4, no. 6, pp. 1671– 1692, 2012, doi: 10.3390/rs4061671.
H.-y. Zhang, W.-m. Lin, and A.-x. Chen, “Path planning for the mobile robot: A review,” Symmetry, vol. 10, no. 10, 2018, doi: 10.3390/sym10100450.
M. Radmanesh, M. Kumar, P. H. Guentert, and M. Sarim, “Overview of path-planning and obstacle avoidance algorithms for uavs: A comparative study,” Unmanned systems, vol. 6, no. 02, pp. 95–118, 2018, doi: 10.1142/S2301385018400022.
O. Souissi, R. Benatitallah, D. Duvivier, A. Artiba, N. Belanger and P. Feyzeau, “Path planning: A 2013 survey,” Proceedings of 2013 International Conference on Industrial Engineering and Systems Management (IESM), pp. 1-8, 2013.
B. Han, T. Qu, X. Tong, J. Jiang, S. Zlatanova, H. Wang, and C. Cheng, “Grid-optimized uav indoor path planning algorithms in a complex environment,” International Journal of Applied Earth Observation and Geoinformation, vol. 111, 2022, doi: 10.1016/j.jag.2022.102857.
Y. U. Tursinovich, “Geometric modeling of threedimensional space and body,” Eurasian Journal of Physics, Chemistry and Mathematics, vol. 5, no. 4, pp. 85–88, 2022.
C. Zhang, H. Liu and Y. Tang, “Quantitative Evaluation of Voronoi Graph Search Algorithm in UAV Path Planning,” 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS), pp. 563-567, 2018, doi: 10.1109/ICSESS.2018.8663950.
M. Aleksandrov, S. Zlatanova, and D. J. Heslop, “Voxelisation algorithms and data structures: A review,” Sensors, vol. 21, no. 24, 2021, doi: 10.3390/s21248241.
X. Wu et al., “A Non-Rigid Hierarchical Discrete Grid Structure and Its Application to UAVs Conflict Detection and Path Planning,” in IEEE Transactions on Aerospace and Electronic Systems, vol. 58, no. 6, pp. 5393-5411, 2022, doi: 10.1109/TAES.2022.3170323.
J. Meyer, A. Sendobry, S. Kohlbrecher, U. Klingauf, and O. von Stryk, “Comprehensive simulation of quadrotor uavs using ros and gazebo,” in Simulation, Modeling, and Programming for Autonomous Robots, vol. 7628, pp. 400–411, 2012, doi: 10.1007/978-3-642-34327-8_36.
S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: Highfidelity visual and physical simulation for autonomous vehicles,” in Field and Service Robotics, vol. 5, pp. 621–635, 2018, doi: 10.1007/978-3-319-67361-5_40.
M. Reda, A. Onsy, A. Y. Haikal, and A. Ghanbari, “Path planning algorithms in the autonomous driving system: A comprehensive review,” Robotics and Autonomous Systems, vol. 174, 2024, doi: 10.1016/j.robot.2024.104630.
Z. Tang and H. Ma, “An overview of path planning algorithms,” in IOP Conference Series: Earth and Environmental Science, vol. 804, no. 2, 2024, doi: 10.1088/1755-1315/804/2/022024.
J. S. Zelek and M. D. Levine, “Local-global concurrent path planning and execution,” in IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, vol. 30, no. 6, pp. 865-870, 2000, doi: 10.1109/3468.895924.
L. Liu, X. Wang, X. Yang, H. Liu, J. Li, and P. Wang, “Path planning techniques for mobile robots: Review and prospect,” Expert Systems with Applications, vol. 227, 2023, doi: 10.1016/j.eswa.2023.120254.
S. Qiu, J. Dai, and D. Zhao, “Path planning of an unmanned aerial vehicle based on a multi-strategy improved pelican optimization algorithm,” Biomimetics, vol. 9, no. 10, 2024, doi: 10.3390/biomimetics9100647.
Y. Shen, Y. Zhu, H. Kang, X. Sun, Q. Chen, and D. Wang, “Uav path planning based on multi-stage constraint optimization,” Drones, vol. 5, no. 4, 2021, doi: 10.3390/drones5040144.
G. Rebala, A. Ravi, and S. Churiwala, “Machine learning definition and basics,” in An Introduction to Machine Learning, pp. 1–17, 2019, doi: 10.1007/978-3-030-15729-61.
J. Alzubi, A. Nayyar, and A. Kumar, “Machine learning from theory to algorithms: An overview,” Journal of Physics: Conference Series, vol. 1142, no. 1, 2018, doi: 10.1088/1742-6596/1142/1/012012.
J. Peng, E. Jury, P. Donnes, and C. Ciurtin, “Machine ¨ learning techniques for personalised medicine approaches in immune-mediated chronic inflammatory diseases: Applications and challenges,” Frontiers in Pharmacology, vol. 12, 2021, doi: 10.3389/fphar.2021.720694.
D. Sarkar, R. Bali, and T. Sharma, “Machine learning basics,” in Practical Machine Learning with Python: A Problem-Solver’s Guide to Building Real-World Intelligent Systems, pp. 3–65, 2018, doi: 10.1007/978-1-4842-3207-1_1.
V. Nasteski, “An overview of the supervised machine learning methods,” vol. 4, pp. 51–62, 2017, doi: 10.20544/HORIZONS.B.04.1.17.P05.
P. Y C a, V. Pulabaigari, and B. E. Reddy, “Semi-supervised learning: a brief review,” vol. 7, pp. 81–85, 2018, doi: 10.14419/ijet.v7i1.8.9977.
S. Naeem, A. Ali, S. Anam, and M. Ahmed, “An unsupervised machine learning algorithms: Comprehensive review,” International Journal of Computing and Digital Systems, vol. 13, no. 1, pp. 911–921, 2023, doi: 10.12785/ijcds/130172.
F. AlMahamid and K. Grolinger, “Reinforcement Learning Algorithms: An Overview and Classification,” 2021 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 1-7, 2021, doi: 10.1109/CCECE53047.2021.9569056.
S. Patil and S. Patil, “Linear with polynomial regression: Overview,” International Journal of Applied Research, vol. 7, no. 8, pp. 273–275, 2021, doi: 10.22271/allresearch.2021.v7.i8d.8876.
L. Meng, L. Yang, S. Ren, G. Tang, L. Zhang, F. Yang, and W. Yang, “An approach of linear regression-based UAV GPS spoofing detection,” vol. 2021, pp. 1–16, 2021, doi: 10.1155/2021/5517500.
P. Kumar, C. Sahu, D. Parhi, K. Pandey, and A. Chhotray, “Static and dynamic path planning of humanoids using an advanced regression controller,” International Journal of Science & Technology, vol. 26, no. 1, pp. 375–393, 2019, doi: 10.24200/sci.2018.5064.1071.
H. Qu, K. Xing, and T. Alexander, “An improved genetic algorithm with co-evolutionary strategy for global path planning of multiple mobile robots,” Neurocomputing, vol. 120, pp. 509–517, 2013, doi: 10.1016/j.neucom.2013.04.020.
K. M. Koo et al., “A uav path planning method using polynomial regression for remote sensor data collection,” in Advances in Computer Science and Ubiquitous Computing, vol. 536, pp. 428– 433, 2020, doi: 10.1007/978-981-13-9341-9_74.
M. Boulares and A. Barnawi, “A novel uav path planning algorithm to search for floating objects on the ocean surface based on object’s trajectory prediction by regression,” Robotics and Autonomous Systems, vol. 135, 2021, doi: 10.1016/j.robot.2020.103673.
E. Yel and N. Bezzo, “GP-based Runtime Planning, Learning, and Recovery for Safe UAV Operations under Unforeseen Disturbances,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2173-2180, 2020, doi: 10.1109/IROS45743.2020.9341641.
M. A. Cano Lengua and E. A. Papa Quiroz, “A Systematic Literature Review on Support Vector Machines Applied to Classification,” 2020 IEEE Engineering International Research Conference (EIRCON), pp. 1-4, 2020, doi: 10.1109/EIRCON51178.2020.9254028.
M. Al-Naeem, M. M. Hafizur Rahman, A. Banerjee, and A. Sufian, “Support vector machine-based energy efficient management of uav locations for aerial monitoring of crops over large agriculture lands,” Sustainability, vol. 15, no. 8, 2023, doi: 10.3390/su15086421.
Y. Chen, W. Zu, G. Fan, and H. Chang, “Unmanned aircraft vehicle path planning based on svm algorithm,” in Foundations and Practical Applications of Cognitive Systems and Information Processing, vol. 215, pp. 705–714, doi: 10.1007/978-3-642-37835-5_61.
N. Morales, J. Toledo, and L. Acosta, “Path planning using a multiclass support vector machine,” Applied Soft Computing, vol. 43, pp. 498–509, 2016, doi: 10.1016/j.asoc.2016.02.037.
I. S. Asti, T. Agustinah and A. Santoso, “Obstacle Avoidance with Energy Efficiency and Distance Deviation Using KNN Algorithm for Quadcopter,” 2020 International Seminar on Intelligent Technology and Its Applications (ISITIA), pp. 285-291, 2020, doi: 10.1109/ISITIA49792.2020.9163788.
A. Pandey, V. S. Panwar, M. E. Hasan, and D. R. Parhi, “V-REPbased navigation of automated wheeled robot between obstacles using PSO-tuned feedforward neural network,” Journal of Computational Design and Engineering, vol. 7, no. 4, pp. 427–434, 2020, doi: 10.1093/jcde/qwaa035.
G. Sanna, S. Godio and G. Guglieri, “Neural Network Based Algorithm for Multi-UAV Coverage Path Planning,” 2021 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 1210- 1217, 2021, doi: 10.1109/ICUAS51884.2021.9476864.
Y. J. Choi, T. Rahim, I. N. A. Ramatryana and S. Y. Shin, “Improved CNN-Based Path Planning for Stairs Climbing in Autonomous UAV with LiDAR Sensor,” 2021 International Conference on Electronics, Information, and Communication (ICEIC), pp. 1-7, 2021, doi: 10.1109/ICEIC51217.2021.9369805.
Y. Liu, Z. Zheng, F. Qin, X. Zhang, and H. Yao, “A residual convolutional neural network based approach for real-time path planning,” Knowledge-Based Systems, vol. 242, 2022, doi: 10.1016/j.knosys.2022.108400.
X. Dai, Y. Mao, T. Huang, N. Qin, D. Huang, and Y. Li, “Automatic obstacle avoidance of quadrotor uav via cnn-based learning,” Neurocomputing, vol. 402, pp. 346–358, 2020, doi: 10.1016/j.neucom.2020.04.020.
D. Sartori, D. Zou, L. Pei and W. Yu, “CNN-based path planning on a map,” 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1331-1338, 2021, doi: 10.1109/ROBIO54168.2021.9739331.
R. S. Nair and P. Supriya, “Robotic Path Planning Using Recurrent Neural Networks,” 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1-5, 2020, doi: 10.1109/ICCCNT49239.2020.9225479.
X. Yue and W. Zhang, “UAV Path Planning Based on KMeans Algorithm and Simulated Annealing Algorithm,” 2018 37th Chinese Control Conference (CCC), pp. 2290-2295, 2018, doi: 10.23919/ChiCC.2018.8483993.
J. Li, W. Li, and W. Zhang, “Path planning of UAV navigation mark inspection using a k-means clustering ACA,” Journal of Marine Science and Technology, vol. 31, no. 3, 2023, doi: 10.51400/2709-6998.2705.
S. Kim and J. Park, “Path planning with multiple UAVs considering the sensing range and improved k-means clustering in WSNs,” aerospace, vol. 10, no. 11, 2023, doi: 10.3390/aerospace10110939.
V. K. Chawra and G. P. Gupta, “Multiple UAV Path-Planning for Data Collection in Cluster-based Wireless Sensor Network,” 2020 First International Conference on Power, Control and Computing Technologies (ICPC2T), pp. 194-198, 2020, doi: 10.1109/ICPC2T48082.2020.9071449.
Y. Li, X. Meng, F. Ye, T. Jiang and Y. Li, “Path Planning Based on Clustering and Improved ACO in UAV-assisted Wireless Sensor Network,” 2020 IEEE USNC-CNC-URSI North American Radio Science Meeting (Joint with AP-S Symposium), pp. 57-58, 2020, doi: 10.23919/USNC/URSI49741.2020.9321638.
Y. Ma, H. Zhang, Y. Zhang, R. Gao, Z. Xu and J. Yang, “Coordinated Optimization Algorithm Combining GA with Cluster for Multi-UAVs to Multi-tasks Task Assignment and Path Planning,” 2019 IEEE 15th International Conference on Control and Automation (ICCA), pp. 1026-1031, 2019, doi: 10.1109/ICCA.2019.8899987.
P. Suseno and T. Wardana, “Unmanned air vehicle path planning for maritime surveillance using cluster-base method,” Aviation, vol. 25, no. 3, pp. 211–219, 2021, doi: 10.3846/aviation.2021.14216.
J. Chen, C. Du, Y. Zhang, P. Han and W. Wei, “A ClusteringBased Coverage Path Planning Method for Autonomous Heterogeneous UAVs,” in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 12, pp. 25546-25556, 2022, doi: 10.1109/TITS.2021.3066240.
P. Xiao, N. Li, F. Xie, H. Ni, M. Zhang, and B. Wang, “Clusteringbased multi-region coverage-path planning of heterogeneous uavs,” Drones, vol. 7, no. 11, 2023, doi: 10.3390/drones7110664.
J. Dai, Q. Hu, X. Liu, Y. Zhang, and J. Zhu, “Cluster head selection method of multiple uavs under covid-19 situation,” Computer Communications, vol. 196, pp. 141–147, 2022, doi: 10.1016/j.comcom.2022.09.026.
J. Faigl, P. Va´na, R. P ˇ eni ˇ cka, and M. Saska, “Unsupervised ˇ learning-based flexible framework for surveillance planning with aerial vehicles,” Journal of Field Robotics, vol. 36, no. 1, pp. 270–301, 2019, doi: 10.1002/rob.21823.
T. Kishimoto, H. Woo, R. Komatsu, Y. Tamura, H. Tomita, K. Shimazoe, A. Yamashita, and H. Asama, “Path planning for localization of radiation sources based on principal component analysis,” Applied Sciences, vol. 11, no. 10, 2021, doi: 10.3390/app11104707.
B. Ristic, M. Morelande, and A. Gunatilaka, “Information driven search for point sources of gamma radiation,” Signal Processing, vol. 90, no. 4, pp. 1225–1239, 2010, doi: 10.1016/j.sigpro.2009.10.006.
S. Kim and J. Park, “Path planning with multiple uavs considering the sensing range and improved k-means clustering in wsns,” Aerospace, vol. 10, no. 11, 2023, doi: 10.3390/aerospace10110939.
B. Jang, M. Kim, G. Harerimana and J. W. Kim, “Q-Learning Algorithms: A Comprehensive Classification and Applications,” in IEEE Access, vol. 7, pp. 133653-133667, 2019, doi: 10.1109/ACCESS.2019.2941229.
Y. Gao, Y. Li and Z. Guo, “A Q-learning based UAV Path Planning Method with Awareness of Risk Avoidance,” 2021 China Automation Congress (CAC), pp. 669-673, 2021, doi: 10.1109/CAC53003.2021.9728342.
X. Xie, T. Wang, Z. Zhu and S. Yang, “Q-Learning for Path Planning of a UAV under Energy Consumption Constraints,” 2023 2nd International Symposium on Control Engineering and Robotics (ISCER), pp. 171-180, 2023, doi: 10.1109/ISCER58777.2023.00036.
C. Yan and X. Xiang, “A Path Planning Algorithm for UAV Based on Improved Q-Learning,” 2018 2nd International Conference on Robotics and Automation Sciences (ICRAS), pp. 1-5, 2018, doi: 10.1109/ICRAS.2018.8443226.
H. Boming, L. Wei, M. Fuzeng and F. Huahao, “Research for UAV Path Planning Method Based on Guided Sarsa Algorithm,” 2022 IEEE 2nd International Conference on Software Engineering and Artificial Intelligence (SEAI), pp. 220-224, 2022, doi: 10.1109/SEAI55746.2022.9832224.
X. Huo, T. Zhang, Y. Wang, and W. Liu, “Dyna-q algorithm for path planning of quadrotor uavs,” in Methods and Applications for Modeling and Simulation of Complex Systems, vol. 946, pp. 349–360, 2018, doi: 10.1007/978-981-13-2853-4_27.
Z. Cui and Y. Wang, “UAV Path Planning Based on Multi-Layer Reinforcement Learning Technique,” in IEEE Access, vol. 9, pp. 59486-59497, 2021, doi: 10.1109/ACCESS.2021.3073704.
B. Jang, M. Kim, G. Harerimana and J. W. Kim, “Q-Learning Algorithms: A Comprehensive Classification and Applications,” in IEEE Access, vol. 7, pp. 133653-133667, 2019, doi: 10.1109/ACCESS.2019.2941229.
H. Anas, W. H. Ong, and O. A. Malik, “Comparison of deep qlearning, q-learning and sarsa reinforced learning for robot local navigation,” in Robot Intelligence Technology and Applications 6, vol. 429, pp. 443–454, 2022, doi: 10.1007/978-3-030-97672-9_40.
W. Luo, Q. Tang, C. Fu, P. Eberhard, Y. Shi, and Q. Tang, “Deepsarsa based multi-uav path planning and obstacle avoidance in a dynamic environment,” in Advances in Swarm Intelligence, 2018, doi: 10.1007/978-3-319-93818-910.
Y. Wang, C. Jiang, T. Ren, Z. Yin, L. Liu, L. Jiang, G. Gu, X. Wu, and W. Ren, “Uav path planning based on ddqn for mountain rescue,” in Intelligent Robotics and Applications, vol. 13458, pp. 509–516, 2022, doi: 10.1007/978-3-031-13841-6_46.
C. Yan, X. Xiang, and C. Wang, “Towards real-time path planning through deep reinforcement learning for a UAV in dynamic environments,” Journal of Intelligent & Robotic Systems, vol. 98, no. 2, pp. 297–309, 2020, doi: 10.1007/s10846-019-01073-3.
Y. Chao, R. Dillmann, A. Roennau, and Z. Xiong, “E-dqnbased path planning method for drones in airsim simulator under unknown environment,” Biomimetics, vol. 9, no. 4, 2024, doi: 10.3390/biomimetics9040238.
W. Wang et al., “A novel uav path planning method based on layered per-ddqn,” in The Proceedings of the 2021 Asia-Pacific International Symposium on Aerospace Technology (APISAT 2021), Volume 2, vol. 913, pp. 693–702, 2023, doi: 10.1007/978-981-19-2635-8_51.
R. Xie, Z. Meng, L. Wang, H. Li, K. Wang and Z. Wu, “Unmanned Aerial Vehicle Path Planning Algorithm Based on Deep Reinforcement Learning in Large-Scale and Dynamic Environments,” in IEEE Access, vol. 9, pp. 24884-24900, 2021, doi: 10.1109/ACCESS.2021.3057485.
F. Wang, X. Zhu, Z. Zhou, and Y. Tang, “Deep-reinforcement-learning-based uav autonomous navigation and collision avoidance in unknown environments,” Chinese Journal of Aeronautics, vol. 37, no. 3, pp. 237–257, 2024, doi: 10.1016/j.cja.2023.09.033.
M. Boulares, A. Fehri, and M. Jemni, “Uav path planning algorithm based on deep q-learning to search for a floating lost target in the ocean,” Robotics and Autonomous Systems, vol. 179, 2024, doi: 10.1016/j.robot.2024.104730.
A. K. Shakya, G. Pillai, and S. Chakrabarty, “Reinforcement learning algorithms: A brief survey,” Expert Systems with Applications, vol. 231, 2023, doi: 10.1016/j.eswa.2023.120495.
H. Kumar, A. Koppel, and A. Ribeiro, “On the sample complexity of actor-critic method for reinforcement learning with function approximation,” vol. 112, no. 7, pp. 2433–2467, 2023, doi: 10.1007/s10994-023-06303-2.
X. Han, J. Wang, Q. Zhang, X. Qin and M. Sun, “Multi-UAV Automatic Dynamic Obstacle Avoidance with Experience-shared A2C,” 2019 International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), pp. 330- 335, 2019, doi: 10.1109/WiMOB.2019.8923344.
G. A. Jimenez, A. de la Escalera Hueso, and M. J. Gomez-Silva, “Reinforcement learning algorithms for autonomous mission accomplishment by unmanned aerial vehicles: A comparative view with dqn, sarsa and a2c,” Sensors, vol. 23, no. 21, 2023, doi: 10.3390/s23219013.
T. Zhao, M. Wang, Q. Zhao, X. Zheng, and H. Gao, “A path-planning method based on improved soft actor-critic algorithm for mobile robots,” Biomimetics, vol. 8, no. 6, 2023, doi: 10.3390/biomimetics8060481.
Y. Zhou, J. Shu, H. Hao, H. Song, and X. Lai, “UAV 3d online track planning based on improved SAC algorithm,” vol. 46, no. 12, 2024, doi: 10.1007/s40430-023-04570-7.
S. Tian, Y. Li, X. Zhang, L. Zheng, L. Cheng, W. She, and W. Xie, “Fast uav path planning in urban environments based on three-step experience buffer sampling ddpg,” Digital Communications and Networks, vol. 10, no. 4, pp. 813–826, 2023, doi: 10.1016/j.dcan.2023.02.016.
Z. Hu, X. Gao, K. Wan, Y. Zhai, and Q. Wang, “Relevant experience learning: A deep reinforcement learning method for uav autonomous motion planning in complex unknown environments,” Chinese Journal of Aeronautics, vol. 34, no. 12, pp. 187–204, 2021, doi: 10.1016/j.cja.2020.12.027.
Z. Wang, S. X. Ng and M. EI-Hajjar, “Deep Reinforcement Learning Assisted UAV Path Planning Relying on Cumulative Reward Mode and Region Segmentation,” in IEEE Open Journal of Vehicular Technology, vol. 5, pp. 737-751, 2024, doi: 10.1109/OJVT.2024.3402129.
X. Luo, Q. Wang, H. Gong and C. Tang, “UAV Path Planning Based on the Average TD3 Algorithm With Prioritized Experience Replay,” in IEEE Access, vol. 12, pp. 38017-38029, 2024, doi: 10.1109/ACCESS.2024.3375083.
J. Fan, Z. Wang, J. Ren, Y. Lu and Y. Liu, “UAV online path planning technology based on deep reinforcement learning,” 2020 Chinese Automation Congress (CAC), pp. 5382-5386, 2020, doi: 10.1109/CAC51589.2020.9327752.
J. Li, W. Li, and W. Zhang, “Path planning of uav navigation mark inspection using a k-means clustering aca,” Journal of Marine Science and Technology, vol. 31, no. 3, 2023, doi: 10.51400/2709-6998.2705.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Anis MAHMOUD BACHA
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
This journal is based on the work at https://journal.umy.ac.id/index.php/jrc under license from Creative Commons Attribution-ShareAlike 4.0 International License. You are free to:
- Share – copy and redistribute the material in any medium or format.
- Adapt – remix, transform, and build upon the material for any purpose, even comercially.
The licensor cannot revoke these freedoms as long as you follow the license terms, which include the following:
- Attribution. You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- ShareAlike. If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
- No additional restrictions. You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
• Creative Commons Attribution-ShareAlike (CC BY-SA)
JRC is licensed under an International License
