Addressing Challenges in Dynamic Modeling of Stewart Platform using Reinforcement Learning-Based Control Approach

Hadi Yadavari, Vahid Tavakol Aghaei, Serhat İkizoğlu

Abstract


In this paper, we focus on enhancing the performance of the controller utilized in the Stewart platform by investigating the dynamics of the platform. Dynamic modeling is crucial for control and simulation, yet challenging for parallel robots like the Stewart platform due to closed-loop kinematics. We explore classical methods to solve its inverse dynamical model, but conventional approaches face difficulties, often resulting in simplified and inaccurate models. To overcome this limitation, we propose a novel approach by replacing the classical feedforward inverse dynamic block with a reinforcement learning (RL) agent, which, to our knowledge, has not been tried yet in the context of the Stewart platform control. Our proposed methodology utilizes a hybrid control topology that combines RL with existing classical control topologies and inverse kinematic modeling. We leverage three deep reinforcement learning (DRL) algorithms and two model-based RL algorithms to achieve improved control performance, highlighting the versatility of the proposed approach. By incorporating the learned feedforward control topology into the existing PID controller, we demonstrate enhancements in the overall control performance of the Stewart platform. Notably, our approach eliminates the need for explicit derivation and solving of the inverse dynamic model, overcoming the drawbacks associated with inaccurate and simplified models. Through several simulations and experiments, we validate the effectiveness of our reinforcement learning-based control approach for the dynamic modeling of the Stewart platform. The results highlight the potential of RL techniques in overcoming the challenges associated with dynamic modeling in parallel robot systems, promising improved control performance. This enhances accuracy and reduces the development time of control algorithms in real-world applications. Nonetheless, it requires a simulation step before practical implementations.

Keywords


Stewart Platform; Dynamic Modelling; Reinforcement Learning; Deep Learning; Control.

Full Text:

PDF

References


X. Yang, H. Wu, B. Chen, S. Kang, and S. Cheng, “Dynamic modeling and decoupled control of a flexible stewart platform for vibration isolation,” Journal of Sound and Vibration, vol. 439, pp. 398–412, 2019, doi: 10.1016/J.JSV.2018.10.007.

T. Ono, R. Eto, J. Yamakawa, and H. Murakami, “Analysis and control of a stewart platform as base motion compensators-part i: Kinematics using moving frames,” Nonlinear Dynamics, vol. 107, pp. 51–76, 2022, doi: 10.1007/s11071-021-06767-8.

A. Jishnu, D. K. Chauhan, and P. R. Vundavilli, “Design of neural network-based adaptive inverse dynamics controller for motion control of stewart platform,” International Journal of Computational Methods, vol. 19, no. 08, p. 2142010, 2022, doi: 10.1142/s021987622142010x.

S. O. Abioye, L. O. Oyedele, L. Akanbi, A. Ajayi, J. M. D. Delgado, M. Bilal, O. O. Akinade, and A. Ahmed, “Artificial intelligence in the construction industry: A review of present status, opportunities and future challenges,” Journal of Building Engineering, vol. 44, p. 103299, 2021, doi: 10.1016/j.jobe.2021.103299.

R. S. Sutton and A. G. Barto, “Reinforcement Learning: An Introduction,” 2018, doi: 10.1109/TNN.1998.712192.

Z. Ding, Y. Huang, H. Yuan, and H. Dong, “Introduction to reinforcement learning,” Deep reinforcement learning: fundamentals, research and applications, pp. 47–123, 2020, doi: 10.1007/978-981-15-4095-0.

S. P. Sethi and S. P. Sethi, “What is optimal control theory?,” Springer, 2019.

Y. Fang, Z. Huang, J. Pu, and J. Zhang, “Auv position tracking and trajectory control based on fast-deployed deep reinforcement learning method,” Ocean Engineering, vol. 245, p. 110452, 2022.

J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238–1274, 2013, doi: 10.1177/0278364913495721.

M. I. Hosseini, S. A. Khalilpour, and H. D. Taghirad, “Practical robust nonlinear pd controller for cable-driven parallel manipulators,” Nonlinear Dynamics, vol. 106, no. 1, pp. 405–424, 2021.

M. Tipaldi, R. Iervolino, and P. R. Massenio, “Reinforcement learning in spacecraft control applications: Advances, prospects, and challenges,” Annual Reviews in Control, 2022.

I. A. Zamfirache, R.-E. Precup, R.-C. Roman, and E. M. Petriu, “Reinforcement learning-based control using q-learning and gravitational search algorithm with experimental validation on a nonlinear servo system,” Information Sciences, vol. 583, pp. 99–120, 2022.

L. Ghorbani and V. E. Omurlu, “Neural networks based real time solution for forward kinematics of a 6× 6 upu flight simulator,” Intelligent Service Robotics, vol. 15, no. 5, pp. 611–626, 2022.

J. A. Houck, R. J. Telban, and F. M. Cardullo, “Motion cueing algorithm development: Human-centered linear and nonlinear approaches,” 2005.

H. Sadjadian, H. D. Taghirad, and A. Fatehi, “Neural networks approaches for computing the forward kinematics of a redundant parallel manipulator,” International Journal of Computer and Information Engineering, vol. 2, no. 1, pp. 1664–1671, 2008.

Z. E. Kuzeci, V. E. Omurlu, H. Alp, and I. Ozkol, “Workspace analysis of parallel mechanisms through neural networks and genetic algorithms,” in 2012 12th IEEE International Workshop on Advanced Motion Control (AMC), pp. 1–6, 2012, doi: 10.1109/AMC.2012.6197147.

H. Yadavari, V. Tavakol Aghaei, and S. ˙Ikizoglu, “Deep reinforcement ˘ learning-based control of stewart platform with parametric simulation in ros and gazebo,” Journal of Mechanisms and Robotics, vol. 15, no. 3, p. 035001, 2023, doi: 10.1115/1.4056971.

S. Pedrammehr, B. Danaei, H. Abdi, M. T. Masouleh, and S. Nahavandi, “Dynamic analysis of hexarot: axis-symmetric parallel manipulator,” Robotica, vol. 36, no. 2, pp. 225–240, 2018, doi: 10.1017/S0263574717000315.

S. Staicu, “Dynamics of Parallel Robots,” Springer, 2019.

A. Arian, B. Danaei, and M. Tale Masouleh, “Kinematic and dynamic analyses of tripteron, an over-constrained 3-dof translational parallel manipulator, through newton-euler approach,” AUT Journal of Modeling and Simulation, vol. 50, no. 1, pp. 61–70, 2018, doi: 10.22060/MISCJ.2018.13020.5055.

S. Pakzad, S. Akhbari, and M. Mahboubkhah, “Kinematic and dynamic analyses of a novel 4-dof parallel mechanism,” Journal of the Brazilian Society of Mechanical Sciences and Engineering, vol. 41, pp. 1–13, 2019, doi: 10.1007/s40430-019-2058-3.

R. F. Abo-Shanab, “Dynamic modeling of parallel manipulators based on lagrange–d’alembert formulation and jacobian/hessian matrices,” Multibody System Dynamics, vol. 48, no. 4, pp. 403–426, 2020, doi: 10.1007/s11044-019-09705-0C.

S. Chen, G. Cheng, and Y. Pang, “Dynamic analysis and trajectory tracking control for a parallel manipulator with joint friction,” Applied Sciences, vol. 12, no. 13, p. 6682, 2022, doi: 10.3390/app12136682.

L.-W. Tsai, “Solving the inverse dynamics of a stewart-gough manipulator by the principle of virtual work,” J. Mech. Des., vol. 122, no. 1, pp. 3–9, 2000, doi: 10.1115/1.533540.

A. Meurer, C. P. Smith, M. Paprocki, O. Cert ˇ ´ık, S. B. Kirpichev, M. Rocklin, A. Kumar, S. Ivanov, J. K. Moore, S. Singh, T. Rathnayake, S. Vig, B. E. Granger, R. P. Muller, F. Bonazzi, H. Gupta, S. Vats, F. Johansson, F. Pedregosa, M. J. Curry, A. R. Terrel, v. Roucka, ˇ A. Saboo, I. Fernando, S. Kulal, R. Cimrman, and A. Scopatz, “Sympy: symbolic computing in python,” PeerJ Computer Science, vol. 3, p. e103, 2017, doi: 10.7717/peerj-cs.103.

C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del R´ıo, M. Wiebe, P. Peterson, P. Gerard-Marchant, K. Sheppard, ´ T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, “Array programming with NumPy,” Nature, vol. 585, no. 7825, pp. 357–362, 2020, doi: 10.1038/s41586-020-2649-2.

I. Faktorovich, C. Bohn, and J. Vogelsang, “Reinforcement learning motivated feedforward control approach for disturbance rejection and tracking,” in 2021 European Control Conference (ECC), pp. 138–143, 2021, doi: 10.23919/ecc54610.2021.9655160.

J. Seo, Y.-S. Na, B. Kim, C. Lee, M. Park, S. Park, and Y. Lee, “Feedforward beta control in the kstar tokamak by deep reinforcement learning,” Nuclear Fusion, vol. 61, no. 10, p. 106010, 2021, doi: 10.1088/1741-4326/ac121b.

R. Liu, F. Nageotte, P. Zanne, M. de Mathelin, and B. Dresp-Langley, “Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review,” Robotics, vol. 10, no. 1, p. 22, 2021, doi: 10.3390/robotics10010022.

V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in International conference on machine learning, pp. 1928–1937, 2016.

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” CoRR, 2015.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.

M. Deisenroth and C. E. Rasmussen, “Pilco: A model-based and data-efficient approach to policy search,” in Proceedings of the 28th International Conference on machine learning (ICML-11), pp. 465–472, 2011.

M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,” Advances in neural information processing systems, vol. 32, 2019.

X. Li, L. Dong, L. Xue, and C. Sun, “Hybrid reinforcement learning for optimal control of non-linear switching system,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 11, pp. 9161–9170, 2023, doi: 10.1109/TNNLS.2022.3156287.

T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang, G. Zhang, P. Abbeel, and J. Ba, “Benchmarking model-based reinforcement learning,” Computer Science, 2019.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning, pp. 1861–1870, 2018.

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015, doi: 10.1038/nature14236.

F. Xue, Q. Hai, T. Dong, Z. Cui, and Y. Gong, “A deep reinforcement learning based hybrid algorithm for efficient resource scheduling in edge computing environment,” Information Sciences, vol. 608, pp. 362–374, 2022, doi: 10.1016/j.ins.2022.06.078.

S. Kizir and Z. Bingul, “Design and development of a stewart platform ¨ assisted and navigated transsphenoidal surgery,” Turkish Journal of Electrical Engineering and Computer Sciences, vol. 27, no. 2, pp. 961–972, 2019, doi: 10.3906/ELK-1608-145.

Z. Bingul and O. Karahan, “Dynamic modeling and simulation of stewart platform,” 2012, doi:10.5772/32470.

D. Rawat, M. K. Gupta, and A. Sharma, “Intelligent control of robotic manipulators: a comprehensive review,” Spatial Information Research, vol. 31, no. 3, pp. 345–357, 2023, doi: 10.1007/s41324-022-00500-2.

V. S. D. M. Sahu, P. Samal, and C. K. Panigrahi, “Modelling, and control techniques of robotic manipulators: A review,” Materials Today: Proceedings, vol. 56, pp. 2758–2766, 2022, doi: 10.1016/j.matpr.2021.10.009.

A. P. Singh, D. Deb, H. Agrawal, K. Bingi, and S. Ozana, “Modeling and control of robotic manipulators: A fractional calculus point of view,” Arabian Journal for Science and Engineering, vol. 46, no. 10, pp. 9541–9552, 2021, doi: 10.1007/s13369-020-05138-6.

Z. Liu, K. Peng, L. Han, and S. Guan, “Modeling and control of robotic manipulators based on artificial neural networks: a review,” Iranian Journal of Science and Technology, Transactions of Mechanical Engineering, pp. 1–41, 2023, doi: 10.1007/s40997-023-00596-3.

M. Engin, “Controller design for parallel mechanism solar tracker,” Machines, vol. 11, no. 3, p. 372, 2023, doi: 10.3390/machines11030372.

H. D. Taghirad, Parallel robots: mechanics and control. CRC press, 2013.

H. Yadavari, “library to calculate the inverse dynamic problem of the stewart platform.” Available: https://github.com/HadiYd/stewart inverse dynamic.

Gazebo Development Team, “Gazebo: An Open-Source Simulator for Robotics Research,” http://gazebosim.org, 2023.

Amazon Web Services. (2023) Amazon Elastic Compute Cloud (Amazon EC2). Amazon Web Services. [Online]. Available: https: //aws.amazon.com/ec2/.

T.-T. Do, V.-H. Vu, and Z. Liu, “Linearization of dynamic equations for vibration and modal analysis of flexible joint manipulators,” Mechanism and Machine Theory, vol. 167, p. 104516, 2022, doi: 10.1016/J.MECHMACHTHEORY.2021.104516.

A. A. Kumar, J.-F. Antoine, and G. Abba, “Input-output feedback linearization for the control of a 4 cable-driven parallel robot,” IFAC-PapersOnLine, vol. 52, no. 13, pp. 707–712, 2019, doi: 10.1016/j.ifacol.2019.11.154.

K. Senda, S. Mano, and S. Fujii, “A reinforcement learning accelerated by state space reduction,” in SICE 2003 Annual Conference (IEEE Cat. No. 03TH8734), vol. 2, pp. 1992–1997, 2003.

T. Sadamoto, A. Chakrabortty, and J.-i. Imura, “Fast online reinforcement learning control using state-space dimensionality reduction,” IEEE Transactions on Control of Network Systems, vol. 8, no. 1, pp. 342–353, 2020, doi: 10.1109/TCNS.2020.3027780.

V. T. Aghaei, A. Agababao ˘ glu, S. Yıldırım, and A. Onat, “A real-world ˘ application of markov chain monte carlo method for bayesian trajectory control of a robotic manipulator,” ISA transactions, vol. 125, pp. 580–590, 2022, doi: 10.1016/j.isatra.2021.06.010.

V. Konda and J. Tsitsiklis, “Actor-critic algorithms,” Advances in neural information processing systems, vol. 12, 1999.

N. D. Nguyen, T. T. Nguyen, P. Vamplew, R. Dazeley, and S. Nahavandi, “A prioritized objective actor-critic method for deep reinforcement learning,” Neural Computing and Applications, vol. 33, pp. 10335–10349, 2021, doi: 10.1007/s00521-021-05795-0.

Y. Yuan, K. Dehghanpour, Z. Wang, and F. Bu, “A joint distribution system state estimation framework via deep actor-critic learning method,” IEEE Transactions on Power Systems, vol. 38, no. 1, pp. 796–806, 2022, doi: 10.1109/TPWRS.2022.3155649.

D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in International conference on machine learning, pp. 387–395, 2014.

J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in International conference on machine learning, pp. 1889–1897, 2015. Hadi YADAVARI, Addressing Challenges in Dynamic Modeling of Stewart Platform using Reinforcement Learning-Based Control Approach Journal of Robotics and Control (JRC) ISSN: 2715-5072 130.

O. O. Abayomi-Alli, R. Damasevi ˇ cius, A. Qazi, M. Adedoyin-Olowe, ˇ and S. Misra, “Data augmentation and deep learning methods in sound classification: A systematic review,” Electronics, vol. 11, no. 22, p. 3795, 2022, doi: 10.3390/electronics11223795.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017, doi: 10.1145/3065386.

A. Mumuni and F. Mumuni, “Data augmentation: A comprehensive survey of modern approaches,” Array, p. 100258, 2022, doi: 10.1016/j.array.2022.100258.

R. S. Sutton, “Integrated architectures for learning, planning, and reacting based on approximating dynamic programming,” in Machine learning proceedings 1990, pp. 216–224, 1990, doi: 10.1016/b978-1-55860-141- 3.50030-4.

M. P. Deisenroth, D. Fox, and C. E. Rasmussen, “Gaussian processes for data-efficient learning in robotics and control,” IEEE transactions on pattern analysis and machine intelligence, vol. 37, no. 2, pp. 408–423, 2013, doi: 10.1109/TPAMI.2013.218.




DOI: https://doi.org/10.18196/jrc.v5i1.20582

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 Hadi Yadavari, Vahid Tavakol Aghaei, Serhat İkizoğlu

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

 

Journal of Robotics and Control (JRC)

P-ISSN: 2715-5056 || E-ISSN: 2715-5072
Organized by Peneliti Teknologi Teknik Indonesia
Published by Universitas Muhammadiyah Yogyakarta in collaboration with Peneliti Teknologi Teknik Indonesia, Indonesia and the Department of Electrical Engineering
Website: http://journal.umy.ac.id/index.php/jrc
Email: jrcofumy@gmail.com

Kuliah Teknik Elektro Terbaik