Re: A new fuzz testing tool for PHP

On 15/11/2024 14:20, Yuancheng Jiang wrote: > Hi all, > > > I have been submitting hundreds of bugs > (see https://github.com/php/php-src/issues/created_by/YuanchengJiang > <)Æ!ÍÄy7Ñ > ôä”kÒ¦https://github.com/php/php-src/issues/created_by/YuanchengJiang>) during > the past months and I first thank all the developers who take time to fix these issues to make PHP > better. > > > I am thrilled to introduce one fully automated fuzz testing tool, FlowFusion, for discovering > various bugs of the PHP interpreter. > > > The core idea behind FlowFusion is to leverage dataflow as an effective representation of test > cases (.phpt files) maintained by PHP developers, merging two (or more) test cases to produce fused > test cases with more complex code semantics. We connect two (or more) test cases via interleaving > their dataflows, i.e., bringing the code context from one test case to another. This enables > interactions among existing test cases, which are mostly the unit tests verifying one single > functionality, making fused test cases interesting with merging code semantics. > > > FlowFusion additionally fuzzes all defined functions and class methods using the code contexts > of fused test cases. Available functions, classes, and methods are pre-collected and stored in > sqlite3 with necessary information like the number of parameters. FlowFusion will be automatically > upgrading if phpt files keep updating. Any new single test can bring thousands of new fused tests. > > > The search space of FlowFusion is huge, which means it can cover various corner cases. Reasons > for the huge search space are three-fold: (i) two random combinations of around 20,000 test cases > can generate 400,000,000 test cases, and we can combine even more; (ii) the interleaving has > randomness, given two test cases, there could be multiple ways to connect them; and (iii) FlowFusion > also mutates the test case, fuzzes the runtime environment/configuration like JIT. > > > *I can open-source the tool under my personal repository. I wonder by any chance if I can > contribute it as the official PHP tool under https://github.com/php > <https://github.com/php>, and I would be happy to maintain it > for a long time.* > > * > * > > Best, > > Yuancheng > Hi Yuancheng Thanks for all the reports you made, certainly an impressive feat! I don't know what other maintainers think, but FWIW I'd be in favor incorporating this into our toolchain. Kind regards Niels