OPTIMAL TRACKING CONTROL FOR ROBOT MANIPULATORS WITH INPUT CONSTRAINT BASED REINFORCEMENT LEARNING
Author affiliations
DOI:
https://doi.org/10.15625/1813-9663/18099Keywords:
Reinforcement learning, Saturation torques, Saturated optimal tracking control, Robot.Abstract
This paper introduces an optimal tracking controller for robot manipulators with saturation torques. The robot model is presented as a strict-feedback nonlinear system. Firstly, the position tracking control problem is transformed into the optimal tracking control problem. Subsequently, the saturated optimal control law is designed. The optimal control law is determined through the solution of the Hamilton-Jacobi-Bellman (HJB) equation. We use a reinforcement learning algorithm with only one neural network (NN) to approximate the solution of the equation HJB. The technique of experience replay is used to relax a persistent citation condition. By Lyapunov analysis, the tracking and the approximation errors are uniformly ultimately bounded (UUB). Finally, the simulation on a robot manipulator with saturation torques is performed to verify the efficiency of the proposed controller.
Metrics
References
J. Baek, W. Kwon, and C. Kang, “A new widely and stably adaptive sliding-mode control with nonsingular terminal sliding variable for robot manipulators,” IEEE Access, vol. 8, pp. 43 443– 43 454, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2977434
D. Chwa and H. Kwon, “Nonlinear robust control of unknown robot manipulator systems with actuators and disturbances using system identification and integral sliding mode disturbance observer,” IEEE Access, vol. 10, pp. 35 410–35 421, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3163306
W. He, Y. Dong, and C. Sun, “Adaptive neural impedance control of a robotic manipulator with input saturation,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 46, no. 3, pp. 334–344, 2015. DOI: https://doi.org/10.1109/TSMC.2015.2429555
W. Jie, L. M. Cheol, K. Jaehyung, and K. H. Hee, “Fast fractional-order terminal sliding mode control with rbfnn based sliding perturbation observer for 7-dof robot manipulator,” IEEE Access, vol. 9, pp. 67 117–67 128, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3075697
W. Jie, Z. Yudong, B. Yulong, H. H. Kim, and M. C. Lee, “Trajectory tracking control using fractional-order terminal sliding mode control with sliding perturbation observer for a 7-dof robot manipulator,” IEEE/ASME Transactions on Mechatronics, vol. 25, no. 4, pp. 1886–1893, DOI: https://doi.org/10.1109/TMECH.2020.2992676
R. Kamalapurkar, H. Dinh, S. Bhasin, and W. E. Dixon, “Approximate optimal trajectory tracking for continuous-time nonlinear systems,” Automatica, vol. 51, pp. 40–48, 2015. DOI: https://doi.org/10.1016/j.automatica.2014.10.103
L. Kong, W. He, C. Yang, and C. Sun, “Robust neurooptimal control for a robot via adaptive dynamic programming,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 6, pp. 2584–2594, 2020. DOI: https://doi.org/10.1109/TNNLS.2020.3006850
F. L. Lewis, D. M. Dawson, and C. T. Abdallah, Robot manipulator control: theory and practice. CRC Press, 2003. DOI: https://doi.org/10.1201/9780203026953
F. L. Lewis, K. Liu, and A. Yesildirek, “Neural net robot controller with guaranteed tracking performance,” IEEE Transactions on Neural Networks, vol. 6, no. 3, pp. 703–715, 1995. DOI: https://doi.org/10.1109/72.377975
S. Ling, H. Wang, and P. X. Liu, “Adaptive fuzzy dynamic surface control of flexible-joint robot systems with input saturation,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 1, pp. 97–107, 2019.
N. T. Luy, “Reinforecement learning-based optimal tracking control for wheeled mobile robot,” in 2012 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER). IEEE, 2012, pp. 371–376.
L. F. Modares H and J. ZP, “h∞ tracking control of completely unknown continuous-time systems via off-policy reinforcement learning,” IEEE Trans Neural Netw Learn Syst, vol. 26, no. 10, pp. 2550–62, 2015.
V.-C. Nguyen, A.-T. Vo, and H.-J. Kang, “A non-singular fast terminal sliding mode control based on third-order sliding mode observer for a class of second-order uncertain nonlinear systems and its application to robot manipulators,” IEEE Access, vol. 8, pp. 78 109–78 120, 2020.
L. Nguyen Tan, “Distributed optimal control for nonholonomic systems with input constraints and uncertain interconnections,” Nonlinear Dynamics, vol. 93, no. 2, pp. 801–817, 2018.
L. N. Tan, “Distributed h∞ optimal tracking control for strict-feedback nonlinear large-scale systems with disturbances and saturating actuators,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 50, no. 11, pp. 4719–4731, 2018.
L. N. Tan and T. C. Pham, “Optimal tracking control for pmsm with partially unknown dynamics, saturation voltages, torque, and voltage disturbances,” IEEE Transactions on Industrial Electronics, vol. 69, no. 4, pp. 3481–3491, 2021.
T. N. Truong, A. T. Vo, and H.-J. Kang, “A backstepping global fast terminal sliding mode control for trajectory tracking control of industrial robotic manipulators,” IEEE Access, vol. 9, pp. 31 921–31 931, 2021.
K. G. Vamvoudakis and F. L. Lewis, “Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem,” Automatica, vol. 46, no. 5, pp. 878–888, 2010.
K. G. Vamvoudakis, M. F. Miranda, and J. P. Hespanha, “Asymptotically stable adaptive– optimal control algorithm with saturating actuators and relaxed persistence of excitation,” IEEE transactions on neural networks and learning systems, vol. 27, no. 11, pp. 2386–2398, 2015.
M. Van, M. Mavrovouniotis, and S. S. Ge, “An adaptive backstepping nonsingular fast terminal sliding mode control for robust fault tolerant control of robot manipulators,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 7, pp. 1448–1458, 2018.
C. Yang, D. Huang, W. He, and L. Cheng, “Neural control of robot manipulators with trajectory tracking constraints and input saturation,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 9, pp. 4231–4242, 2020.
H. Zargarzadeh, T. Dierks, and S. Jagannathan, “Adaptive neural network-based optimal control of nonlinear continuous-time systems in strict-feedback form,” International Journal of Adaptive Control and Signal Processing, vol. 28, no. 3-5, pp. 305–324, 2014.
S. Zeghloul, M. A. Laribi, and J.-P. Gazeau, “Robotics and mechatronics,” in Proceedings of the 4th IFToMM International Symposium on Robotics and Mechatronics. Springer, 2015.
L. Zhang, Y. Wang, Y. Hou, and H. Li, “Fixed-time sliding mode control for uncertain robot manipulators,” IEEE Access, vol. 7, pp. 149 750–149 763, 2019.
S. Ling, H. Wang, and P. X. Liu, “Adaptive fuzzy dynamic surface control of flexible-joint robot systems with input saturation,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 1, pp. 97–107, 2019. DOI: https://doi.org/10.1109/JAS.2019.1911330
N. T. Luy, “Reinforecement learning-based optimal tracking control for wheeled mobile robot,” in 2012 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER). IEEE, 2012, pp. 371–376. DOI: https://doi.org/10.1109/CYBER.2012.6392582
L. F. Modares H and J. ZP, “h∞ tracking control of completely unknown continuous-time systems via off-policy reinforcement learning,” IEEE Trans Neural Netw Learn Syst, vol. 26, no. 10, pp. 2550–62, 2015. DOI: https://doi.org/10.1109/TNNLS.2015.2441749
V.-C. Nguyen, A.-T. Vo, and H.-J. Kang, “A non-singular fast terminal sliding mode control based on third-order sliding mode observer for a class of second-order uncertain nonlinear systems and its application to robot manipulators,” IEEE Access, vol. 8, pp. 78 109–78 120, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2989613
L. Nguyen Tan, “Distributed optimal control for nonholonomic systems with input constraints and uncertain interconnections,” Nonlinear Dynamics, vol. 93, no. 2, pp. 801–817, 2018. DOI: https://doi.org/10.1007/s11071-018-4228-8
L. N. Tan, “Distributed h∞ optimal tracking control for strict-feedback nonlinear large-scale systems with disturbances and saturating actuators,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 50, no. 11, pp. 4719–4731, 2018. DOI: https://doi.org/10.1109/TSMC.2018.2861470
L. N. Tan and T. C. Pham, “Optimal tracking control for pmsm with partially unknown dynamics, saturation voltages, torque, and voltage disturbances,” IEEE Transactions on Industrial Electronics, vol. 69, no. 4, pp. 3481–3491, 2021. DOI: https://doi.org/10.1109/TIE.2021.3075892
T. N. Truong, A. T. Vo, and H.-J. Kang, “A backstepping global fast terminal sliding mode control for trajectory tracking control of industrial robotic manipulators,” IEEE Access, vol. 9, pp. 31 921–31 931, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3060115
K. G. Vamvoudakis and F. L. Lewis, “Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem,” Automatica, vol. 46, no. 5, pp. 878–888, 2010. DOI: https://doi.org/10.1016/j.automatica.2010.02.018
K. G. Vamvoudakis, M. F. Miranda, and J. P. Hespanha, “Asymptotically stable adaptive– optimal control algorithm with saturating actuators and relaxed persistence of excitation,” IEEE transactions on neural networks and learning systems, vol. 27, no. 11, pp. 2386–2398, 2015. DOI: https://doi.org/10.1109/TNNLS.2015.2487972
M. Van, M. Mavrovouniotis, and S. S. Ge, “An adaptive backstepping nonsingular fast terminal sliding mode control for robust fault tolerant control of robot manipulators,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 7, pp. 1448–1458, 2018. DOI: https://doi.org/10.1109/TSMC.2017.2782246
C. Yang, D. Huang, W. He, and L. Cheng, “Neural control of robot manipulators with trajectory tracking constraints and input saturation,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 9, pp. 4231–4242, 2020. DOI: https://doi.org/10.1109/TNNLS.2020.3017202
H. Zargarzadeh, T. Dierks, and S. Jagannathan, “Adaptive neural network-based optimal control of nonlinear continuous-time systems in strict-feedback form,” International Journal of Adaptive Control and Signal Processing, vol. 28, no. 3-5, pp. 305–324, 2014. DOI: https://doi.org/10.1002/acs.2432
S. Zeghloul, M. A. Laribi, and J.-P. Gazeau, “Robotics and mechatronics,” in Proceedings of the 4th IFToMM International Symposium on Robotics and Mechatronics. Springer, 2015. DOI: https://doi.org/10.1007/978-3-319-22368-1
L. Zhang, Y. Wang, Y. Hou, and H. Li, “Fixed-time sliding mode control for uncertain robot manipulators,” IEEE Access, vol. 7, pp. 149 750–149 763, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2946866
X. Zhao, B. Tao, L. Qian, and H. Ding, “Model-based actor-critic learning for optimal tracking control of robots with input saturation,” IEEE Transactions on Industrial Electronics, vol. 68, no. 6, pp. 5046–5056, 2020. DOI: https://doi.org/10.1109/TIE.2020.2992003
J. Zhou, E. Liu, X. Tian, and Z. Li, “Adaptive fuzzy backstepping control based on dynamic surface control for uncertain robotic manipulator,” IEEE Access, vol. 10, pp. 23 333–23 341, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3154779
Q. Zhou, S. Zhao, H. Li, R. Lu, and C. Wu, “Adaptive neural network tracking control for robotic manipulators with dead zone,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 12, pp. 3611–3620, 2018. DOI: https://doi.org/10.1109/TNNLS.2018.2869375
Downloads
Published
How to Cite
Issue
Section
License
1. We hereby assign copyright of our article (the Work) in all forms of media, whether now known or hereafter developed, to the Journal of Computer Science and Cybernetics. We understand that the Journal of Computer Science and Cybernetics will act on my/our behalf to publish, reproduce, distribute and transmit the Work.2. This assignment of copyright to the Journal of Computer Science and Cybernetics is done so on the understanding that permission from the Journal of Computer Science and Cybernetics is not required for me/us to reproduce, republish or distribute copies of the Work in whole or in part. We will ensure that all such copies carry a notice of copyright ownership and reference to the original journal publication.
3. We warrant that the Work is our results and has not been published before in its current or a substantially similar form and is not under consideration for another publication, does not contain any unlawful statements and does not infringe any existing copyright.
4. We also warrant that We have obtained the necessary permission from the copyright holder/s to reproduce in the article any materials including tables, diagrams or photographs not owned by me/us.
