OPTIMAL TRACKING CONTROL FOR ROBOT MANIPULATORS WITH ASYMMETRIC SATURATION TORQUES BASED ON REINFORCEMENT LEARNING

Nguyen Duc Dien, Nguyen Tan Luy, Lai Khac Lai
Author affiliations

Authors

  • Nguyen Duc Dien University of Economics – Technology for Industry, 456 Minh Khai, Hai Ba Trung, Ha Noi, Viet Nam
  • Nguyen Tan Luy Ho Chi Minh City University of Technology--VNU--HCM, 268 Ly Thuong Kiet street, District 10, Ho Chi Minh City, Viet Nam https://orcid.org/0000-0002-7732-261X
  • Lai Khac Lai Thai Nguyen University of Technology, 666, 3/2 street, Tich Luong ward, Thai Nguyen, Viet Nam

DOI:

https://doi.org/10.15625/1813-9663/17641

Keywords:

Robot manipulators, reinforcement learning, optimal control, competitive learning, asymmetry saturation

Abstract

This paper introduces an optimal tracking controller for robot manipulators with asymmetrically saturated torques and partially - unknown dynamics based on a reinforcement learning method using a neural network. Firstly, the feedforward control inputs are designed based on the backstepping technique to convert the tracking control problem into the optimal tracking control problem. Secondly, a cost function of the system with asymmetrically saturated input is defined, and the constrained Hamilton-Jacobi-Bellman equation is built, which is solved by the online reinforcement learning algorithm using only a single neural network. Then, the asymmetric saturation optimal control rule is determined. Additionally, the concurrent learning technique is used to relax the demand for the persistence of excitation conditions. The built algorithm ensures that the closed-loop system is asymptotically stable, the approximation error is uniformly ultimately bounded (UUB), and the cost function converges to the near-optimal value. Finally, the effectiveness of the proposed algorithm is shown through comparative simulations.

Metrics

Metrics Loading ...

References

X. Bu, “An improvement of single-network adaptive critic design for nonlinear systems with

asymmetry constraints,” Journal of the Franklin Institute, vol. 356, no. 16, pp. 9646–9664, 2019. DOI: https://doi.org/10.1016/j.jfranklin.2019.09.021

J. J. Craig, Introduction to robotics: mechanics and control. Pearson Educacion, 2005.

L. Dai, Y. Yu, D.-H. Zhai, T. Huang, and Y. Xia, “Robust model predictive tracking control for

robot manipulators with disturbances,” IEEE Transactions on Industrial Electronics, vol. 68,

no. 5, pp. 4288–4297, 2021.

K. De Backer, T. DeStefano, C. Menon, and J. R. Suh, “Industrial robotics and the global

organisation of production,” OECD Science, Technology and Industry Working Papers.

W. He, Y. Dong, and C. Sun, “Adaptive neural impedance control of a robotic manipulator with

input saturation,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 46,

no. 3, pp. 334–344, 2015.

W. He, Z. Li, and C. P. Chen, “A survey of human-centered intelligent robots: issues and

challenges,” IEEE/CAA Journal of Automatica Sinica, vol. 4, no. 4, pp. 602–609, 2017. DOI: https://doi.org/10.1109/JAS.2017.7510604

P. Hippe, Windup in control: its effects and their prevention. Springer Science & Business

Media, 2006.

C.-L. Hwang and B.-S. Chen, “Adaptive finite-time saturated tracking control for a class of partially known robots,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, DOI: https://doi.org/10.1109/TSMC.2019.2957183

no. 9, pp. 5674–5685, 2021.

R. Kamalapurkar, H. Dinh, S. Bhasin, and W. E. Dixon, “Approximate optimal trajectory

tracking for continuous-time nonlinear systems,” Automatica, vol. 51, pp. 40–48, 2015. DOI: https://doi.org/10.1016/j.automatica.2014.10.103

W. Khalil and E. Dombre, Modeling identification and control of robots. CRC Press, 2002. DOI: https://doi.org/10.1016/B978-190399666-9/50014-2

L. Kong, W. He, Y. Dong, L. Cheng, C. Yang, and Z. Li, “Asymmetric bounded neural control

for an uncertain robot by state feedback and output feedback,” IEEE Transactions on Systems,

Man, and Cybernetics: Systems, vol. 51, no. 3, pp. 1735–1746, 2021. DOI: https://doi.org/10.1109/TSMC.2021.3110362

L. Kong, W. He, C. Yang, and C. Sun, “Robust neurooptimal control for a robot via adaptive

dynamic programming,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32,

no. 6, pp. 2584–2594, 2021.

F. L. Lewis, D. M. Dawson, and C. T. Abdallah, Robot manipulator control: theory and practice. CRC Press, 2003. DOI: https://doi.org/10.1201/9780203026953

S. Ling, H. Wang, and P. X. Liu, “Adaptive fuzzy dynamic surface control of flexible-joint robot

systems with input saturation,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 1, pp.

–107, 2019.

D. Liu, D. Wang, F.-Y. Wang, H. Li, and X. Yang, “Neural-network-based online hjb solution for

optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems,” IEEE

Transactions on Cybernetics, vol. 44, no. 12, pp. 2834–2847, 2014. DOI: https://doi.org/10.1109/TCYB.2014.2357896

D. Liu, S. Xue, B. Zhao, B. Luo, and Q. Wei, “Adaptive dynamic programming for control: A

survey and recent advances,” IEEE Transactions on Systems, Man, and Cybernetics: Systems,

vol. 51, no. 1, pp. 142–160, 2021. DOI: https://doi.org/10.1097/01.BMSAS.0000798444.40023.fb

J. Liu, Intelligent Control Design and MATLAB Simulation. Springer Singapore, 2018. DOI: https://doi.org/10.1007/978-981-10-5263-7

X. Long, Z. He, and Z. Wang, “Online optimal control of robotic systems with single critic

nn-based reinforcement learning,” Complexity, vol. 2021, 2021.

J. Ma, S. S. Ge, Z. Zheng, and D. Hu, “Adaptive nn control of a class of nonlinear systems

with asymmetric saturation actuators,” IEEE Transactions on Neural Networks and Learning

Systems, vol. 26, no. 7, pp. 1532–1538, 2015. DOI: https://doi.org/10.1109/TNNLS.2014.2344019

H. Modares, F. L. Lewis, and Z.-P. Jiang, “h∞ tracking control of completely unknown

continuous-time systems via off-policy reinforcement learning,” IEEE Transactions on Neural

Networks and Learning Systems, vol. 26, no. 10, pp. 2550–2562, 2015. DOI: https://doi.org/10.1109/TNNLS.2015.2441749

M. Nakamura, S. Goto, and N. Kyura, “Torque saturation of a mechatronic servo system,” in

Mechatronic Servo System Control. Springer, 2004, pp. 97–119.

L. Nguyen Tan, “Distributed optimal control for nonholonomic systems with input constraints

and uncertain interconnections,” Nonlinear Dynamics, vol. 93, no. 2, pp. 801–817, 2018. DOI: https://doi.org/10.1007/s11071-018-4228-8

K. Shojaei, A. Kazemy, and A. Chatraei, “An observer-based neural adaptive pid2 controller

for robot manipulators including motor dynamics with a prescribed performance,” IEEE/ASME

Transactions on Mechatronics, vol. 26, no. 3, pp. 1689–1699, 2021. DOI: https://doi.org/10.1109/TMECH.2020.3028968

L. N. Tan, “Distributed h∞ optimal tracking control for strict-feedback nonlinear large-scale

systems with disturbances and saturating actuators,” IEEE Transactions on Systems, Man, and

Cybernetics: Systems, vol. 50, no. 11, pp. 4719–4731, 2018.

L. N. Tan and T. C. Pham, “Optimal tracking control for pmsm with partially unknown dynamics, saturation voltages, torque, and voltage disturbances,” IEEE Transactions on Industrial

Electronics, vol. 69, no. 4, pp. 3481–3491, 2021. DOI: https://doi.org/10.1109/TIE.2021.3075892

K. G. Vamvoudakis and F. L. Lewis, “Online actor–critic algorithm to solve the continuous-time

infinite horizon optimal control problem,” Automatica, vol. 46, no. 5, pp. 878–888, 2010. DOI: https://doi.org/10.1016/j.automatica.2010.02.018

K. G. Vamvoudakis, M. F. Miranda, and J. P. Hespanha, “Asymptotically stable adaptive–

optimal control algorithm with saturating actuators and relaxed persistence of excitation,” IEEE

Transactions on Neural Networks and Learning Systems, vol. 27, no. 11, pp. 2386–2398, 2016. DOI: https://doi.org/10.1109/TNNLS.2015.2487972

M. Van and S. S. Ge, “Adaptive fuzzy integral sliding-mode control for robust fault-tolerant

control of robot manipulators with disturbance observer,” IEEE Transactions on Fuzzy Systems,

vol. 29, no. 5, pp. 1284–1296, 2020. DOI: https://doi.org/10.1109/TFUZZ.2020.2973955

R.-D. Xi, X. Xiao, T.-N. Ma, and Z.-X. Yang, “Adaptive sliding mode disturbance observer

based robust control for robot manipulators towards assembly assistance,” IEEE Robotics and

Automation Letters, vol. 7, no. 3, pp. 6139–6146, 2022. DOI: https://doi.org/10.1109/LRA.2022.3164448

L. Xia, Q. Li, R. Song, and H. Modares, “Optimal synchronization control of heterogeneous

asymmetric input-constrained unknown nonlinear mass via reinforcement learning,” IEEE/CAA

Journal of Automatica Sinica, vol. 9, no. 3, pp. 520–532, 2021. DOI: https://doi.org/10.1109/JAS.2021.1004359

S. Xue, B. Luo, D. Liu, and Y. Gao, “Event-triggered integral reinforcement learning for nonzerosum games with asymmetric input saturation,” Neural Networks, vol. 152, pp. 212–223, 2022. DOI: https://doi.org/10.1016/j.neunet.2022.04.013

C. Yang, D. Huang, W. He, and L. Cheng, “Neural control of robot manipulators with trajectory

tracking constraints and input saturation,” IEEE Transactions on Neural Networks and Learning

Systems, vol. 32, no. 9, pp. 4231–4242, 2021. DOI: https://doi.org/10.1109/TNNLS.2020.3017202

C. Yang, X. Wang, L. Cheng, and H. Ma, “Neural-learning-based telerobot control with guaranteed performance,” IEEE Transactions on Cybernetics, vol. 47, no. 10, pp. 3148–3159, 2017. DOI: https://doi.org/10.1109/TCYB.2016.2573837

T. Yang, N. Sun, Y. Fang, X. Xin, and H. Chen, “New adaptive control methods for n-link robot

manipulators with online gravity compensation: Design and experiments,” IEEE Transactions

on Industrial Electronics, vol. 69, no. 1, pp. 539–548, 2021. DOI: https://doi.org/10.1109/TIE.2021.3050371

B. M. Yilmaz, E. Tatlicioglu, A. Savran, and M. Alci, “Self-adjusting fuzzy logic based control of

robot manipulators in task space,” IEEE Transactions on Industrial Electronics, vol. 69, no. 2,

pp. 1620–1629, 2022.

H. Zargarzadeh, T. Dierks, and S. Jagannathan, “Adaptive neural network-based optimal control

of nonlinear continuous-time systems in strict-feedback form,” International Journal of Adaptive

Control and Signal Processing, vol. 28, no. 3-5, pp. 305–324, 2014. DOI: https://doi.org/10.1002/acs.2432

S. Zeghloul, M. A. Laribi, and J.-P. Gazeau, “Robotics and mechatronics,” in Procedings of the

th IFToMM International Symposium on Robotics and Mechatronics. Springer, 2015.

X. Zhao, B. Tao, L. Qian, and H. Ding, “Model-based actor-critic learning for optimal tracking

control of robots with input saturation,” IEEE Transactions on Industrial Electronics, vol. 68,

no. 6, pp. 5046–5056, 2021.

Z. Zhou, G. Tang, H. Huang, L. Han, and R. Xu, “Adaptive nonsingular fast terminal sliding mode control for underwater manipulator robotics with asymmetric saturation actuators,”

Control Theory and Technology, vol. 18, no. 1, pp. 81–91, 2020. DOI: https://doi.org/10.1007/s11768-020-9127-0

Downloads

Published

29-03-2023

How to Cite

[1]
N. D. Dien, N. T. Luy, and L. K. Lai, “OPTIMAL TRACKING CONTROL FOR ROBOT MANIPULATORS WITH ASYMMETRIC SATURATION TORQUES BASED ON REINFORCEMENT LEARNING”, JCC, vol. 39, no. 1, p. 61–77, Mar. 2023.

Issue

Section

Articles