Real-time predictive artificial intelligence: deep reinforcement learning for closed-loop control systems and open-loop signal processing

Daryanavard, Sama (2024) Real-time predictive artificial intelligence: deep reinforcement learning for closed-loop control systems and open-loop signal processing. PhD thesis, University of Glasgow.

Abstract

Reactive mechanisms, such as reflexes, respond to disturbances only after they have occurred. In contrast, learning entities operate on principles of anticipation and prediction, enabling them to preemptively counteract potential disturbances. This research introduces a framework that integrates learning capabilities into traditional reflex systems, creating a comprehensive closed-loop platform tailored for reinforcement learning, particularly in robotics. Central to our approach is the use of backpropagation through deep neural networks. Although inherently an open-loop algorithm, we demonstrate through mathematical derivation that minimising the reflex error is equivalent to minimising the unknown open-loop error. We illustrate how the reflex error can be utilised to train the system.

Our innovative method involves applying backpropagation within closed-loop control systems, utilising z-transformation and intricate mathematical derivations. This approach offers significant advantages over existing algorithms. It functions as an online algorithm that learns in real-time, eliminating the need for pre-training or the use of physics engines. Additionally, it is particularly well-suited to continuous state-space applications, such as robotics, where defining all discrete states is non-trivial or impossible. One of the most striking advantages is the speed of convergence, which approximates one-shot learning due to the availability of the error signal from the reflex at every time-step for training the network, unlike scenarios where the reward or punishment signal is sparse. This makes the developed learning algorithm very fast compared to conventional Reinforcement Learning approaches.

This research culminates in the development of four distinct algorithms: CLDL, SAR, PAM, and Echo learning, each characterised by unique attributes and intricacies. These algorithms are inspired by biological processes and the functioning of the human brain. We rigorously tested these algorithms on a line-following robot, conducting experiments in both real-world scenarios and simulated environments to ensure reproducibility. The outcomes demonstrate successful path navigation by the robot without relying on its inherent reflex mechanism.

Furthermore, we present evidence that our platform can be effectively adapted for unsupervised open-loop systems with minimal adjustments. We show how EMG noise is removed from EEG signals without any training. This study not only proposes a novel approach to integrating learning into robotic reflex systems but also broadens the potential applications of such algorithms in various automated processes.

This work makes significant contributions to the field of reinforcement learning by providing an open-source C++ logic library containing the algorithms for all aforementioned paradigms. Additionally, it includes two deployment libraries that demonstrate how to implement these algorithms on both simulated and real-world robots, as well as another library for signal processing, such as noise cancellation.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Subjects: T Technology > TA Engineering (General). Civil engineering (General)
• Colleges/Schools: College of Science and Engineering > School of Engineering > Biomedical Engineering
• Supervisor's Name: Porr, Dr. Bernd and Neale, Professor Steven
• Date of Award: 2024
• Depositing User: Theses Team
• Unique ID: glathesis:2024-84692
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 14 Nov 2024 15:02
• Last Modified: 14 Nov 2024 15:03
• Thesis DOI: 10.5525/gla.thesis.84692
• URI: https://theses.gla.ac.uk/id/eprint/84692

Related URLs

• Article DOI
• Pre-print
• Pre-print
• Article DOI