OPTIMAL TRACKING CONTROL FOR ROBOT MANIPULATORS WITH ASYMMETRIC SATURATION TORQUES BASED ON REINFORCEMENT LEARNING
Author affiliations
DOI:
https://doi.org/10.15625/1813-9663/17641Keywords:
Robot manipulators, reinforcement learning, optimal control, competitive learning, asymmetry saturationAbstract
This paper introduces an optimal tracking controller for robot manipulators with asymmetrically saturated torques and partially - unknown dynamics based on a reinforcement learning method using a neural network. Firstly, the feedforward control inputs are designed based on the backstepping technique to convert the tracking control problem into the optimal tracking control problem. Secondly, a cost function of the system with asymmetrically saturated input is defined, and the constrained Hamilton-Jacobi-Bellman equation is built, which is solved by the online reinforcement learning algorithm using only a single neural network. Then, the asymmetric saturation optimal control rule is determined. Additionally, the concurrent learning technique is used to relax the demand for the persistence of excitation conditions. The built algorithm ensures that the closed-loop system is asymptotically stable, the approximation error is uniformly ultimately bounded (UUB), and the cost function converges to the near-optimal value. Finally, the effectiveness of the proposed algorithm is shown through comparative simulations.
Metrics
References
X. Bu, “An improvement of single-network adaptive critic design for nonlinear systems with
asymmetry constraints,” Journal of the Franklin Institute, vol. 356, no. 16, pp. 9646–9664, 2019. DOI: https://doi.org/10.1016/j.jfranklin.2019.09.021
J. J. Craig, Introduction to robotics: mechanics and control. Pearson Educacion, 2005.
L. Dai, Y. Yu, D.-H. Zhai, T. Huang, and Y. Xia, “Robust model predictive tracking control for
robot manipulators with disturbances,” IEEE Transactions on Industrial Electronics, vol. 68,
no. 5, pp. 4288–4297, 2021.
K. De Backer, T. DeStefano, C. Menon, and J. R. Suh, “Industrial robotics and the global
organisation of production,” OECD Science, Technology and Industry Working Papers.
W. He, Y. Dong, and C. Sun, “Adaptive neural impedance control of a robotic manipulator with
input saturation,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 46,
no. 3, pp. 334–344, 2015.
W. He, Z. Li, and C. P. Chen, “A survey of human-centered intelligent robots: issues and
challenges,” IEEE/CAA Journal of Automatica Sinica, vol. 4, no. 4, pp. 602–609, 2017. DOI: https://doi.org/10.1109/JAS.2017.7510604
P. Hippe, Windup in control: its effects and their prevention. Springer Science & Business
Media, 2006.
C.-L. Hwang and B.-S. Chen, “Adaptive finite-time saturated tracking control for a class of partially known robots,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, DOI: https://doi.org/10.1109/TSMC.2019.2957183
no. 9, pp. 5674–5685, 2021.
R. Kamalapurkar, H. Dinh, S. Bhasin, and W. E. Dixon, “Approximate optimal trajectory
tracking for continuous-time nonlinear systems,” Automatica, vol. 51, pp. 40–48, 2015. DOI: https://doi.org/10.1016/j.automatica.2014.10.103
W. Khalil and E. Dombre, Modeling identification and control of robots. CRC Press, 2002. DOI: https://doi.org/10.1016/B978-190399666-9/50014-2
L. Kong, W. He, Y. Dong, L. Cheng, C. Yang, and Z. Li, “Asymmetric bounded neural control
for an uncertain robot by state feedback and output feedback,” IEEE Transactions on Systems,
Man, and Cybernetics: Systems, vol. 51, no. 3, pp. 1735–1746, 2021. DOI: https://doi.org/10.1109/TSMC.2021.3110362
L. Kong, W. He, C. Yang, and C. Sun, “Robust neurooptimal control for a robot via adaptive
dynamic programming,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32,
no. 6, pp. 2584–2594, 2021.
F. L. Lewis, D. M. Dawson, and C. T. Abdallah, Robot manipulator control: theory and practice. CRC Press, 2003. DOI: https://doi.org/10.1201/9780203026953
S. Ling, H. Wang, and P. X. Liu, “Adaptive fuzzy dynamic surface control of flexible-joint robot
systems with input saturation,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 1, pp.
–107, 2019.
D. Liu, D. Wang, F.-Y. Wang, H. Li, and X. Yang, “Neural-network-based online hjb solution for
optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems,” IEEE
Transactions on Cybernetics, vol. 44, no. 12, pp. 2834–2847, 2014. DOI: https://doi.org/10.1109/TCYB.2014.2357896
D. Liu, S. Xue, B. Zhao, B. Luo, and Q. Wei, “Adaptive dynamic programming for control: A
survey and recent advances,” IEEE Transactions on Systems, Man, and Cybernetics: Systems,
vol. 51, no. 1, pp. 142–160, 2021. DOI: https://doi.org/10.1097/01.BMSAS.0000798444.40023.fb
J. Liu, Intelligent Control Design and MATLAB Simulation. Springer Singapore, 2018. DOI: https://doi.org/10.1007/978-981-10-5263-7
X. Long, Z. He, and Z. Wang, “Online optimal control of robotic systems with single critic
nn-based reinforcement learning,” Complexity, vol. 2021, 2021.
J. Ma, S. S. Ge, Z. Zheng, and D. Hu, “Adaptive nn control of a class of nonlinear systems
with asymmetric saturation actuators,” IEEE Transactions on Neural Networks and Learning
Systems, vol. 26, no. 7, pp. 1532–1538, 2015. DOI: https://doi.org/10.1109/TNNLS.2014.2344019
H. Modares, F. L. Lewis, and Z.-P. Jiang, “h∞ tracking control of completely unknown
continuous-time systems via off-policy reinforcement learning,” IEEE Transactions on Neural
Networks and Learning Systems, vol. 26, no. 10, pp. 2550–2562, 2015. DOI: https://doi.org/10.1109/TNNLS.2015.2441749
M. Nakamura, S. Goto, and N. Kyura, “Torque saturation of a mechatronic servo system,” in
Mechatronic Servo System Control. Springer, 2004, pp. 97–119.
L. Nguyen Tan, “Distributed optimal control for nonholonomic systems with input constraints
and uncertain interconnections,” Nonlinear Dynamics, vol. 93, no. 2, pp. 801–817, 2018. DOI: https://doi.org/10.1007/s11071-018-4228-8
K. Shojaei, A. Kazemy, and A. Chatraei, “An observer-based neural adaptive pid2 controller
for robot manipulators including motor dynamics with a prescribed performance,” IEEE/ASME
Transactions on Mechatronics, vol. 26, no. 3, pp. 1689–1699, 2021. DOI: https://doi.org/10.1109/TMECH.2020.3028968
L. N. Tan, “Distributed h∞ optimal tracking control for strict-feedback nonlinear large-scale
systems with disturbances and saturating actuators,” IEEE Transactions on Systems, Man, and
Cybernetics: Systems, vol. 50, no. 11, pp. 4719–4731, 2018.
L. N. Tan and T. C. Pham, “Optimal tracking control for pmsm with partially unknown dynamics, saturation voltages, torque, and voltage disturbances,” IEEE Transactions on Industrial
Electronics, vol. 69, no. 4, pp. 3481–3491, 2021. DOI: https://doi.org/10.1109/TIE.2021.3075892
K. G. Vamvoudakis and F. L. Lewis, “Online actor–critic algorithm to solve the continuous-time
infinite horizon optimal control problem,” Automatica, vol. 46, no. 5, pp. 878–888, 2010. DOI: https://doi.org/10.1016/j.automatica.2010.02.018
K. G. Vamvoudakis, M. F. Miranda, and J. P. Hespanha, “Asymptotically stable adaptive–
optimal control algorithm with saturating actuators and relaxed persistence of excitation,” IEEE
Transactions on Neural Networks and Learning Systems, vol. 27, no. 11, pp. 2386–2398, 2016. DOI: https://doi.org/10.1109/TNNLS.2015.2487972
M. Van and S. S. Ge, “Adaptive fuzzy integral sliding-mode control for robust fault-tolerant
control of robot manipulators with disturbance observer,” IEEE Transactions on Fuzzy Systems,
vol. 29, no. 5, pp. 1284–1296, 2020. DOI: https://doi.org/10.1109/TFUZZ.2020.2973955
R.-D. Xi, X. Xiao, T.-N. Ma, and Z.-X. Yang, “Adaptive sliding mode disturbance observer
based robust control for robot manipulators towards assembly assistance,” IEEE Robotics and
Automation Letters, vol. 7, no. 3, pp. 6139–6146, 2022. DOI: https://doi.org/10.1109/LRA.2022.3164448
L. Xia, Q. Li, R. Song, and H. Modares, “Optimal synchronization control of heterogeneous
asymmetric input-constrained unknown nonlinear mass via reinforcement learning,” IEEE/CAA
Journal of Automatica Sinica, vol. 9, no. 3, pp. 520–532, 2021. DOI: https://doi.org/10.1109/JAS.2021.1004359
S. Xue, B. Luo, D. Liu, and Y. Gao, “Event-triggered integral reinforcement learning for nonzerosum games with asymmetric input saturation,” Neural Networks, vol. 152, pp. 212–223, 2022. DOI: https://doi.org/10.1016/j.neunet.2022.04.013
C. Yang, D. Huang, W. He, and L. Cheng, “Neural control of robot manipulators with trajectory
tracking constraints and input saturation,” IEEE Transactions on Neural Networks and Learning
Systems, vol. 32, no. 9, pp. 4231–4242, 2021. DOI: https://doi.org/10.1109/TNNLS.2020.3017202
C. Yang, X. Wang, L. Cheng, and H. Ma, “Neural-learning-based telerobot control with guaranteed performance,” IEEE Transactions on Cybernetics, vol. 47, no. 10, pp. 3148–3159, 2017. DOI: https://doi.org/10.1109/TCYB.2016.2573837
T. Yang, N. Sun, Y. Fang, X. Xin, and H. Chen, “New adaptive control methods for n-link robot
manipulators with online gravity compensation: Design and experiments,” IEEE Transactions
on Industrial Electronics, vol. 69, no. 1, pp. 539–548, 2021. DOI: https://doi.org/10.1109/TIE.2021.3050371
B. M. Yilmaz, E. Tatlicioglu, A. Savran, and M. Alci, “Self-adjusting fuzzy logic based control of
robot manipulators in task space,” IEEE Transactions on Industrial Electronics, vol. 69, no. 2,
pp. 1620–1629, 2022.
H. Zargarzadeh, T. Dierks, and S. Jagannathan, “Adaptive neural network-based optimal control
of nonlinear continuous-time systems in strict-feedback form,” International Journal of Adaptive
Control and Signal Processing, vol. 28, no. 3-5, pp. 305–324, 2014. DOI: https://doi.org/10.1002/acs.2432
S. Zeghloul, M. A. Laribi, and J.-P. Gazeau, “Robotics and mechatronics,” in Procedings of the
th IFToMM International Symposium on Robotics and Mechatronics. Springer, 2015.
X. Zhao, B. Tao, L. Qian, and H. Ding, “Model-based actor-critic learning for optimal tracking
control of robots with input saturation,” IEEE Transactions on Industrial Electronics, vol. 68,
no. 6, pp. 5046–5056, 2021.
Z. Zhou, G. Tang, H. Huang, L. Han, and R. Xu, “Adaptive nonsingular fast terminal sliding mode control for underwater manipulator robotics with asymmetric saturation actuators,”
Control Theory and Technology, vol. 18, no. 1, pp. 81–91, 2020. DOI: https://doi.org/10.1007/s11768-020-9127-0
Downloads
Published
How to Cite
Issue
Section
License
1. We hereby assign copyright of our article (the Work) in all forms of media, whether now known or hereafter developed, to the Journal of Computer Science and Cybernetics. We understand that the Journal of Computer Science and Cybernetics will act on my/our behalf to publish, reproduce, distribute and transmit the Work.2. This assignment of copyright to the Journal of Computer Science and Cybernetics is done so on the understanding that permission from the Journal of Computer Science and Cybernetics is not required for me/us to reproduce, republish or distribute copies of the Work in whole or in part. We will ensure that all such copies carry a notice of copyright ownership and reference to the original journal publication.
3. We warrant that the Work is our results and has not been published before in its current or a substantially similar form and is not under consideration for another publication, does not contain any unlawful statements and does not infringe any existing copyright.
4. We also warrant that We have obtained the necessary permission from the copyright holder/s to reproduce in the article any materials including tables, diagrams or photographs not owned by me/us.
