Reinforcement Learning for Multi-Task Manipulation in Robotic Arm Systems Operating in Dynamic Environments
DOI:
https://doi.org/10.18196/jrc.v6i5.27780Keywords:
Artificial Intelligence (AI), Reinforcement Learning (RL), Goal-Directed Manipulation, Variable Object Positions, Robotic Arm ControlAbstract
The development of integrating Reinforcement Learning (RL) in robots seems to provide solutions to a variety of complex manipulation tasks in uncertain dynamic environments. The limitation of the research is that the given research permits a robotic arm to learn and perform several manipulation tasks in an autonomously-observed manner, using a model-free RL approach. The key improvement of the current work is an ability to train an agent to perform various actions in a shared space that is needed to perform very different manipulation actions. The method is implemented with the help of a three-dimensional simulator that is done by using a robotic arm, items in the workspace (table, objects), and the time-varying location of targets. The robotic system undergoes training in six different manipulation actions including Reach, Push, Slide, Pick and Place, Stack and Flip. With reward shaping based on the tasks, RL architecture learns to execute each task effectively in working with the environment. The success rates in each of the manipulation tasks during experiment time have demonstrated successful completion of the tasks as opposed to before training, displaying their adaptability and accuracy. Also, this framework has generalization ability as it changes its object positions and the dynamics used. These results help justify the possibility of reinforcement learning as a tool to train robots on flexible, goal directed manipulation tasks and so avoid manual programming. Future work may extend this approach to real-world robotic platforms with sensory feedback integration.
References
L. Chen, P. Chen, and Z. Lin, “Artificial Intelligence in Education: A Review,” IEEE Access, vol. 8, pp. 75264–75278, 2020, doi: 10.1109/access.2020.2988510.
L. Vandewinckele et al., “Overview of artificial intelligence-based applications in radiotherapy: Recommendations for implementation and quality assurance,” Radiotherapy and Oncology, vol. 153, pp. 55–66, Dec. 2020, doi: 10.1016/j.radonc.2020.09.008.
I. H. Sarker, “AI-Based Modeling: Techniques, Applications and Research Issues Towards Automation, Intelligent and Smart Systems,” SN Computer Science, vol. 3, no. 2, Feb. 2022, doi: 10.1007/s42979-022-01043-x.
P. Wang, “On Defining Artificial Intelligence,” Journal of Artificial General Intelligence, vol. 10, no. 2, pp. 1–37, Jan. 2019, doi: 10.2478/jagi-2019-0002.
R. Liu, F. Nageotte, P. Zanne, M. de Mathelin, and B. Dresp-Langley, “Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Review,” Robotics, vol. 10, no. 1, p. 22, Jan. 2021, doi: 10.3390/robotics10010022.
G. Siemens et al., “Human and artificial cognition,” Computers and Education: Artificial Intelligence, vol. 3, p. 100107, 2022, doi: 10.1016/j.caeai.2022.100107.
C. Zhang and Y. Lu, “Study on artificial intelligence: The state of the art and future prospects,” Journal of Industrial Information Integration, vol. 23, p. 100224, Sep. 2021, doi: 10.1016/j.jii.2021.100224.
B. Singh, R. Kumar, and V. P. Singh, “Reinforcement learning in robotic applications: a comprehensive survey,” Artificial Intelligence Review, vol. 55, no. 2, pp. 945–990, Apr. 2021, doi: 10.1007/s10462-021-09997-9.
F. Vicentini, “Collaborative Robotics: A Survey,” Journal of Mechanical Design, vol. 143, no. 4, Oct. 2020, doi: 10.1115/1.4046238.
Y. Li and O. Hilliges, Eds., Artificial Intelligence for Human Computer Interaction: A Modern Approach. Springer International Publishing, 2021, doi: 10.1007/978-3-030-82681-9.
S. Kumar, A. K. Verma, and A. Mirza, “Digitalisation, Artificial Intelligence, IoT, and Industry 4.0 and Digital Society,” Digital Transformation, Artificial Intelligence and Society, pp. 35–57, 2024, doi: 10.1007/978-981-97-5656-8_3.
S. P. Yadav, D. P. Mahato, and N. T. D. Linh, Distributed Artificial Intelligence. CRC Press, 2020, doi: 10.1201/9781003038467.
V. V. Krishna, “A I and contemporary challenges: The good, bad and the scary,” Journal of Open Innovation: Technology, Market, and Complexity, vol. 10, no. 1, p. 100178, Mar. 2024, doi: 10.1016/j.joitmc.2023.100178.
W. Wang and K. Siau, “Artificial Intelligence, Machine Learning, Automation, Robotics, Future of Work and Future of Humanity,” Journal of Database Management, vol. 30, no. 1, pp. 61–79, Jan. 2019, doi: 10.4018/jdm.2019010104.
M. Soori, B. Arezoo, and R. Dastres, “Artificial intelligence, machine learning and deep learning in advanced robotics, a review,” Cognitive Robotics, vol. 3, pp. 54–70, 2023, doi: 10.1016/j.cogr.2023.04.001.
A. K. Tyagi, T. F. Fernandez, S. Mishra, and S. Kumari, “Intelligent Automation Systems at the Core of Industry 4.0,” Intelligent Systems Design and Applications, pp. 1–18, 2021, doi: 10.1007/978-3-030-71187-0_1.
T. V. N. Rao, A. Gaddam, M. Kurni, and K. Saritha, “Reliance on Artificial Intelligence, Machine Learning and Deep Learning in the Era of Industry 4.0,” Smart Healthcare System Design, pp. 281–299, Jun. 2021, doi: 10.1002/9781119792253.ch12.
M. Bashabsheh, “Simulation of An Automatic System of Robotics for Artificial Animated Being Manufacturing Using AnyLogic Simulation Software,” International Journal of Electrical and Electronics Engineering, vol. 11, no. 5, pp. 129–137, May 2024, doi: 10.14445/23488379/ijeee-v11i5p112.
M. Bashabsheh, “Comprehensive and Simulated Modeling of a Centralized Transport Robot Control System,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 5, 2024, doi: 10.14569/ijacsa.2024.0150552.
B. Ni and A. Obashi, “Robotics technology and firm-level employment adjustment in Japan,” Japan and the World Economy, vol. 57, p. 101054, Mar. 2021, doi: 10.1016/j.japwor.2021.101054.
K. Peta, M. Wiśniewski, M. Kotarski, and O. Ciszak, “Comparison of Single-Arm and Dual-Arm Collaborative Robots in Precision Assembly,” Applied Sciences, vol. 15, no. 6, p. 2976, Mar. 2025, doi: 10.3390/app15062976.
W. Kornmaneesang, S.-L. Chen, and S. Boonto, “Contouring Control of an Innovative Manufacturing System Based on Dual-Arm Robot,” IEEE Transactions on Automation Science and Engineering, vol. 19, no. 3, pp. 2042–2053, Jul. 2022, doi: 10.1109/tase.2021.3073423.
N. Lv, J. Liu, and Y. Jia, “Dynamic Modeling and Control of Deformable Linear Objects for Single-Arm and Dual-Arm Robot Manipulations,” IEEE Transactions on Robotics, vol. 38, no. 4, pp. 2341–2353, Aug. 2022, doi: 10.1109/tro.2021.3139838.
M. Santoro et al., “Recent applications of artificial intelligence in radiotherapy: where we are and beyond,” Applied Sciences, vol. 12, no. 7, p. 3223, 2022.
Y. Himeur et al., “AI-big data analytics for building automation and management systems: a survey, actual challenges and future perspectives,” Artificial Intelligence Review, vol. 56, no. 6, pp. 4929–5021, Oct. 2022, doi: 10.1007/s10462-022-10286-2.
J.-A. Johannessen, “Artificial Intelligence, Automation and the Future of Competence at Work,” Routledge, Dec. 2020, doi: 10.4324/9781003121923.
V. Bhardwaj et al., “Automatic Speech Recognition (ASR) Systems for Children: A Systematic Literature Review,” Applied Sciences, vol. 12, no. 9, p. 4419, Apr. 2022, doi: 10.3390/app12094419.
M. Arora and M. Kumar, “AutoFER: PCA and PSO based automatic facial emotion recognition,” Multimedia Tools and Applications, vol. 80, no. 2, pp. 3039–3049, Sep. 2020, doi: 10.1007/s11042-020-09726-4.
S. M. Zinchenko, A. P. Ben, P. S. Nosov, I. S. Popovych, P. P. Mamenko, and V. M. Mateichuk, “Improving the accuracy and reliability of automatic vessel moution control system,” Radio Electronics, Computer Science, Control, no. 2, pp. 183–195, Sep. 2020, doi: 10.15588/1607-3274-2020-2-19.
A. Hameed, A. Ordys, J. Możaryn, and A. Sibilska-Mroziewicz, “Control system design and methods for collaborative robots,” Applied Sciences, vol. 13, no. 1, p. 675, 2023.
Z. Makhataeva and H. Varol, “Augmented Reality for Robotics: A Review,” Robotics, vol. 9, no. 2, p. 21, Apr. 2020, doi: 10.3390/robotics9020021.
T. Zhang and H. Mo, “Reinforcement learning for robot research: A comprehensive review and open issues,” International Journal of Advanced Robotic Systems, vol. 18, no. 3, p. 172988142110073, May 2021, doi: 10.1177/17298814211007305.
Ó. Pérez-Gil et al., “DQN-Based Deep Reinforcement Learning for Autonomous Driving,” Advances in Physical Agents II, pp. 60–76, Nov. 2020, doi: 10.1007/978-3-030-62579-5_5.
Z. Arkouli, G. Kokotinis, G. Michalos, N. Dimitropoulos, and S. Makris, “AI-enhanced cooperating robots for reconfigurable manufacturing of large parts,” IFAC-PapersOnLine, vol. 54, no. 1, pp. 617–622, 2021, doi: 10.1016/j.ifacol.2021.08.072.
J. Li, Y. Chen, X. Zhao, and J. Huang, “An improved DQN path planning algorithm,” The Journal of Supercomputing, vol. 78, no. 1, pp. 616–639, May 2021, doi: 10.1007/s11227-021-03878-2.
J. Wan, X. Li, H.-N. Dai, A. Kusiak, M. Martinez-Garcia, and D. Li, “Artificial-Intelligence-Driven Customized Manufacturing Factory: Key Technologies, Applications, and Challenges,” Proceedings of the IEEE, vol. 109, no. 4, pp. 377–398, Apr. 2021, doi: 10.1109/jproc.2020.3034808.
H. B. Mahajan et al., “Automatic robot Manoeuvres detection using computer vision and deep learning techniques: a perspective of internet of robotics things (IoRT),” Multimedia Tools and Applications, vol. 82, no. 15, pp. 23251–23276, Nov. 2022, doi: 10.1007/s11042-022-14253-5.
N. Emaminejad and R. Akhavian, “Trustworthy AI and robotics: Implications for the AEC industry,” Automation in Construction, vol. 139, p. 104298, Jul. 2022, doi: 10.1016/j.autcon.2022.104298.
J. Wang and A. Chortos, “Control Strategies for Soft Robot Systems,” Advanced Intelligent Systems, vol. 4, no. 5, Feb. 2022, doi: 10.1002/aisy.202100165.
M. Soori, F. K. G. Jough, R. Dastres, and B. Arezoo, “AI-Based Decision Support Systems in Industry 4.0, A Review,” Journal of Economy and Technology, Aug. 2024, doi: 10.1016/j.ject.2024.08.005.
S. H. M. Mehr, M. Craven, A. I. Leonov, G. Keenan, and L. Cronin, “A universal system for digitization and automatic execution of the chemical synthesis literature,” Science, vol. 370, no. 6512, pp. 101–108, Oct. 2020, doi: 10.1126/science.abc2986.
D. Carou, A. Sartal, and J. P. Davim, Eds., Machine Learning and Artificial Intelligence with Industrial Applications. Springer International Publishing, 2022, doi: 10.1007/978-3-030-91006-8
H. Chen et al., “From Automation System to Autonomous System: An Architecture Perspective,” Journal of Marine Science and Engineering, vol. 9, no. 6, p. 645, Jun. 2021, doi: 10.3390/jmse9060645.
F. Folgado, D. Calderón, I. González, and A. Calderón, “Review of Industry 4.0 from the Perspective of Automation and Supervision Systems: Definitions, Architectures and Recent Trends,” Electronics, vol. 13, no. 4, p. 782, Feb. 2024, doi: 10.3390/electronics13040782.
Y. Huang, “Deep Q-Networks,” Deep Reinforcement Learning, pp. 135–160, 2020, doi: 10.1007/978-981-15-4095-0_4.
S. Satheeshbabu, N. K. Uppalapati, T. Fu, and G. Krishnan, “Continuous Control of a Soft Continuum Arm using Deep Reinforcement Learning,” 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), pp. 497–503, May 2020, doi: 10.1109/robosoft48309.2020.9116003.
N. Sharma, J. K. Pandey, and S. Mondal, “A Review of Mobile Robots: Applications and Future Prospect,” International Journal of Precision Engineering and Manufacturing, vol. 24, no. 9, pp. 1695–1706, Aug. 2023, doi: 10.1007/s12541-023-00876-7.
G. Singh and V. K. Banga, “Robots and its types for industrial applications,” Materials Today: Proceedings, vol. 60, pp. 1779–1786, 2022, doi: 10.1016/j.matpr.2021.12.426.
Y. Dai, C. Xiang, Y. Zhang, Y. Jiang, W. Qu, and Q. Zhang, “A Review of Spatial Robotic Arm Trajectory Planning,” Aerospace, vol. 9, no. 7, p. 361, Jul. 2022, doi: 10.3390/aerospace9070361.
N. Rokbani, B. Neji, M. Slim, S. Mirjalili, and R. Ghandour, “A Multi-Objective Modified PSO for Inverse Kinematics of a 5-DOF Robotic Arm,” Applied Sciences, vol. 12, no. 14, p. 7091, Jul. 2022, doi: 10.3390/app12147091.
T. T. Tung, N. Van Tinh, D. T. Phuong Thao, and T. V. Minh, “Development of a prototype 6 degree of freedom robot arm,” Results in Engineering, vol. 18, p. 101049, Jun. 2023, doi: 10.1016/j.rineng.2023.101049.
P. K. Srivastava, M. P. Yadav, and V. R. Joshi, “International Conference on Multidisciplinary Emerging Trends in Engineering and Technology (ICMETET-2024),” International Conference on Multidisciplinary Emerging Trends in Engineering and Technology (ICMETET-2024), May 2024, doi: 10.17492/jpi.isbm.042401.
J. S. Dhatterwal, K. S. Kaswan, and R. Batra. Nature Inspired Robotics. Chapman and Hall/CRC, 2024, doi: 10.1201/9781032624358.
C. Calderón-Cordova, R. Sarango, D. Castillo, and V. Lakshminarayanan, “A Deep Reinforcement Learning Framework for Control of Robotic Manipulators in Simulated Environments,” IEEE Access, vol. 12, pp. 103133–103161, 2024, doi: 10.1109/access.2024.3432741.
S. Joshi, S. Kumra, and F. Sahin, “Robotic Grasping using Deep Reinforcement Learning,” 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pp. 1461–1466, Aug. 2020, doi: 10.1109/case48305.2020.9216986.
D. Liu et al., Using Educational Robots to Enhance Learning. Springer Nature Singapore, 2024, doi: 10.1007/978-981-97-5826-5.
H. Wang et al., “Deep reinforcement learning: a survey,” Frontiers of Information Technology & Electronic Engineering, vol. 21, no. 12, pp. 1726–1744, Oct. 2020, doi: 10.1631/fitee.1900533.
L. A. Munoz Ubando, A. Amigud, and E. Sirazitdinova, “Computer simulation and hands-on labs: A case study of teaching robotics and AI,” International Journal of Mechanical Engineering Education, vol. 53, no. 2, pp. 451–469, Mar. 2024, doi: 10.1177/03064190241240416.
A. Heuillet, F. Couthouis, and N. Díaz-Rodríguez, “Explainability in deep reinforcement learning,” Knowledge-Based Systems, vol. 214, p. 106685, Feb. 2021, doi: 10.1016/j.knosys.2020.106685.
Q. Wu et al., “Position Control of Cable-Driven Robotic Soft Arm Based on Deep Reinforcement Learning,” Information, vol. 11, no. 6, p. 310, Jun. 2020, doi: 10.3390/info11060310.
M. Bashabsheh, “Autonomous Robotic Systems with Artificial Intelligence Technology Using a Deep Q Network-Based Approach for Goal-Oriented 2D Arm Control,” Journal of Robotics and Control (JRC), vol. 5, no. 6, pp. 1872-1887, 2024, doi: 10.18196/jrc.v5i6.23850.
M. Bashabsheh, “A Combined Model for Simulating the Spatial Dynamics of Epidemic Spread: Integrating Stochastic Compartmentalization and Cellular Automata Approach,” International Journal of Mathematical, Engineering and Management Sciences, vol. 10, no. 2, pp. 522–536, Apr. 2025, doi: 10.33889/ijmems.2025.10.2.026.
M. Bashabsheh and M. Alzubi, “Integrated Simulation Model of the Spatial Distribution of Dynamic Systems Using Intelligent Cellular Automaton,” Panamerican Mathematical Journal, vol. 34, no. 3, pp. 29–37, Oct. 2024, doi: 10.52783/pmj.v34.i3.1771.
M. Bashabsheh, “Mathematical model of the spread of COVID-19 using any logic system,” in AIP Conference Proceedings, vol. 2930, no. 1, p. 020012, 2023.
H. Liang, X. Lou, Y. Yang, and C. Choi, “Learning Visual Affordances with Target-Orientated Deep Q-Network to Grasp Objects by Harnessing Environmental Fixtures,” 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2562–2568, May 2021, doi: 10.1109/icra48506.2021.9561737.
J. Wu, H. Yang, L. Yang, Y. Huang, X. He, and C. Lv, “Human-Guided Deep Reinforcement Learning for Optimal Decision Making of Autonomous Vehicles,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 54, no. 11, pp. 6595–6609, Nov. 2024, doi: 10.1109/tsmc.2024.3384992.
A. Jafari-Tabrizi and D. P. Gruber, “Reinforcement-Learning-based Control of an Industrial Robotic Arm for Following a Randomly-Generated 2D-Trajectory,” 2021 IEEE International Conference on Omni-Layer Intelligent Systems (COINS), vol. 518, pp. 1–6, Aug. 2021, doi: 10.1109/coins51742.2021.9524158.
E. G. Ribeiro, R. de Queiroz Mendes, and V. Grassi Jr, “Real-time deep learning approach to visual servo control and grasp detection for autonomous robotic manipulation,” Robotics and Autonomous Systems, vol. 139, p. 103757, 2021.
Y. Jaafra, J. L. Laurent, A. Deruyver, and M. S. Naceur, “Reinforcement learning for neural architecture search: A review,” Image and Vision Computing, vol. 89, pp. 57-66, 2019.
P. Dayan, “Motivated Reinforcement Learning,” Advances in Neural Information Processing Systems, vol. 14, pp. 11–18, Nov. 2002, doi: 10.7551/mitpress/1120.003.0006.
J. Terven, “Deep Reinforcement Learning: A Chronological Overview and Methods,” AI, vol. 6, no. 3, p. 46, Feb. 2025, doi: 10.3390/ai6030046.
S. R. Patil and S. Agashe, “Deep Reinforcement Learning Controller for Conical Tank Process,” 2023 Second IEEE International Conference on Measurement, Instrumentation, Control and Automation (ICMICA), pp. 1–6, May 2024, doi: 10.1109/icmica61068.2024.10732564.
A. Iftikhar, M. A. Ghazanfar, M. Ayub, S. Ali Alahmari, N. Qazi, and J. Wall, “A reinforcement learning recommender system using bi-clustering and Markov Decision Process,” Expert Systems with Applications, vol. 237, p. 121541, Mar. 2024, doi: 10.1016/j.eswa.2023.121541.
D. Han, B. Mulyana, V. Stankovic, and S. Cheng, “A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation,” Sensors, vol. 23, no. 7, p. 3762, Apr. 2023, doi: 10.3390/s23073762.
M. Al‐Gabalawy, “Path planning of robotic arm based on deep reinforcement learning algorithm,” Advanced Control for Applications, vol. 4, no. 1, Mar. 2022, doi: 10.1002/adc2.79.
S. Balhara et al., “A survey on deep reinforcement learning architectures, applications and emerging trends,” IET Communications, Jul. 2022, doi: 10.1049/cmu2.12447.
M. Botvinick, J. X. Wang, W. Dabney, K. J. Miller, and Z. Kurth-Nelson, “Deep Reinforcement Learning and Its Neuroscientific Implications,” Neuron, vol. 107, no. 4, pp. 603–616, Aug. 2020, doi: 10.1016/j.neuron.2020.06.014.
P. Ladosz, L. Weng, M. Kim, and H. Oh, “Exploration in deep reinforcement learning: A survey,” Information Fusion, vol. 85, pp. 1–22, Sep. 2022, doi: 10.1016/j.inffus.2022.03.003.
S. E. Li, “Deep Reinforcement Learning,” Reinforcement Learning for Sequential Decision and Optimal Control, pp. 365–402, 2023, doi: 10.1007/978-981-19-7784-8_10.
S. Carta, A. Ferreira, A. S. Podda, D. Reforgiato Recupero, and A. Sanna, “Multi-DQN: An ensemble of Deep Q-learning agents for stock market forecasting,” Expert Systems with Applications, vol. 164, p. 113820, Feb. 2021, doi: 10.1016/j.eswa.2020.113820.
S. Wang, L. Sun, F. Zha, W. Guo, and P. Wang, “Learning adaptive reaching and pushing skills using contact information,” Frontiers in Neurorobotics, vol. 17, Sep. 2023, doi: 10.3389/fnbot.2023.1271607.
M. J. McDonald and D. Hadfield‑Menell, "Guided Imitation of Task and Motion Planning," in Proc. Conf. Robot Learn. (CoRL), PMLR, pp. 630–640, Jan. 2022.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Murad Bashabsheh
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
This journal is based on the work at https://journal.umy.ac.id/index.php/jrc under license from Creative Commons Attribution-ShareAlike 4.0 International License. You are free to:
- Share – copy and redistribute the material in any medium or format.
- Adapt – remix, transform, and build upon the material for any purpose, even comercially.
The licensor cannot revoke these freedoms as long as you follow the license terms, which include the following:
- Attribution. You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- ShareAlike. If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
- No additional restrictions. You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
• Creative Commons Attribution-ShareAlike (CC BY-SA)
JRC is licensed under an International License
