Model-free Optimal Control for Underactuated Quadrotor Aircraft via Reinforcement Learning

Abstract

The control of Unmanned Aerial Vehicles (UAVs), especially quadrotor aircraft, has many practical applications such as transporting, mapping, rescue, and agricultural applications. This paper investigates solving the optimal tracking control problem for a quadrotor system. First, an underactuated quadrotor system is considered a highly nonlinear system with six degrees of freedom and four inputs. Second, a hierarchical control structure consisting of position and attitude controller is adopted to address the underactuated problem, the position controller to achieve the desired tracking and generates the references for the attitude controller, and the attitude controller to achieve the reference attitude tracking. Third, to achieve optimal trajectory tracking, two Data- based Reinforcement Learning (RL) algorithms are applied to both position and attitude controllers to find the optimal control input by using the input- output quadrotor system data. Compared with the traditional optimal algorithms which require directly solving the Algebraic Ricatti Equation (ARE) or the Hamilton-Jacobi-Bellman (HJB) equation. It is impossible or difficult to implement due to the high nonlinear dynamic nature of the quadrotor system. By using RL in the proposed method, optimal policies can be learned without the knowledge of quadrotor dynamic information. Applying the learning control policies to the quadrotor system, the vehicle achieves optimal trajectory tracking. Finally, a simulation result is conducted to verify the optimal trajectory tracking for quadrotor with the proposed controller.