Fuzzing software with deep learning

Sablotny, Martin (2023) Fuzzing software with deep learning. PhD thesis, University of Glasgow.

Abstract

Generation based fuzz testing can uncover various bug classes and security vulnerabilities. However, compared to mutation based fuzz testing it takes a great amount of time to develop a well balanced generator that generates good test cases and decides were to break the underlying structure to exercise new code paths.

This thesis provides an evaluation of generative deep learning algorithms to generate HTML test cases to fuzz test a browser’s HTML rendering engine. The experiments highlight that various deep learning algorithm are performing well in this setting. However, there are large differences in the stability of the training and code coverage performance. The best performing in terms of code coverage as well as training stability is a Temporal Convolutional Network (TCN).

The TCN model is then also used to learn from real world HTML data to generate novel test cases withouth the need of a generative fuzzer in the first place. The results show that the approach is able to discover new code areas that were neither discovered by the underlying fuzzer nor the prior models. Furthermore, this highlights how an existing fuzzer can be augmented with the help of a deep learning model and publicly available training data. Finally, reinforcement learning is used to further improve the existing fuzzer by utilizing the code coverage data from the browser under test. The designed DDQN agent is able to guide the test case creation of a TCN to even outperform the underlying baseline test case generator.

• Item Type: Thesis (PhD)
• Qualification Level: Doctoral
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science; Q Science > QA Mathematics > QA76 Computer software
• Colleges/Schools: College of Science and Engineering > School of Computing Science
• Supervisor's Name: Jensen, Dr. Bjørn and Singer, Dr. Jeremy
• Date of Award: 2023
• Depositing User: Theses Team
• Unique ID: glathesis:2023-83496
• Copyright: Copyright of this thesis is held by the author.
• Date Deposited: 22 Mar 2023 16:19
• Last Modified: 23 Mar 2023 09:55
• Thesis DOI: 10.5525/gla.thesis.83496
• URI: https://theses.gla.ac.uk/id/eprint/83496