Re: C Unit testing and mocking

On Dec 16, 2024, at 9:18 AM, Jakub Zelenka <bukka@php.net> wrote: > > Hi, > > I have been looking into how to test some cases where integration tests are very difficult or > even impossible to create for. Those are often found in networking related and system specific code > code (network.c, streams, FPM and more). I was recently fixing one such bug and decided to give a > try which resulted in this PR: ¤Ñ¦t¼ > iÌD¸'eÝÿ#https://github.com/php/php-src/pull/16987 . > > There was a suggestion of RFC but that might be a bit too much as it's just an internal > change / addition. But certainly some overview on internals should be done so writing this instead. > > I decided to use cmocka in that PR because I had some experience with that. It's quite > small and still very powerful and allow vast mocking options. It's a bit manual but it gives a > bigger control over the mock. It relies on --wrap linking option that replaces original functions > with wraps. This is however available only on Linux (or maybe some other Unix variants) but > doesn't work on MacOS or Windows. The developers that want to use it on those platforms would > need to use some Linux Virtualisation option (e.g. Docker). It also requires static library which is > supported by embed SAPI that can be compiled statically. That limits number of extensions to use but > the main use cases don't really have deps so it should be fine. > > I did also some research into the other mocking libraries in C. There is a Unity with CMock, > FFF and some C++ libs like GUnit, Criterion and Trompeloeil that I looked into. I quickly discarded > GUnit and Trompeloeil as they relay on C++ virtual methods and require wrapping C code to C++ which > is very inconvenient. FFF seems too simple and maybe quite inflexible for our needs as well. > Criterion also optionally uses wrap so I didn't see much advantages compare to cmocka. So it > left Unity with CMock that allows generating custom mocks using a Ruby script. That seemed initially > quite nice but after spending around two hours with trying to make it works for PHP codebase, I just > gave up. It gets quite messy for complex scenarios and I just didn't figure out how to nicely > mock libc functions without any modification to php-src. > > In terms of CI. It has got its own build which is very simple and it tests just specific parts > so we could just limit it to run only for changed files which might be quite convenient. > > So the proposed PR is probably the only reasonable unit testing that I can come up with. I > think it should be completely optional initially for people to use - more like an experiment. If it > becomes used, then good of course. And if it becomes pain, we can just get rid of it. Has anyone got > any objections to get this merged? If not I plan to merge it early in January. > > Cheers > > Jakub > I'm assuming that uses ELF symbol interposition or something like that, which is why it seems Linux/BSD specific. That seems fragile to me. I think currently for wanting to test C functions, we're adding custom functions into ext/zend_test and writing PHPT. Would this work? We already have that, after all. If not, it'd be helpful to list the challenges that approach faces.