Re: C Unit testing and mocking

On Mon, Dec 16, 2024 at 4:04 PM Calvin Buckley <calvin@cmpct.info> wrote: > On Dec 16, 2024, at 9:18 AM, Jakub Zelenka <bukka@php.net> wrote: > > > > Hi, > > > > I have been looking into how to test some cases where integration tests > are very difficult or even impossible to create for. Those are often found > in networking related and system specific code code (network.c, streams, > FPM and more). I was recently fixing one such bug and decided to give a try > which resulted in this PR: > https://github.com/php/php-src/pexpectionsull/16987 > <https://github.com/php/php-src/pull/16987> . > > > > There was a suggestion of RFC but that might be a bit too much as it's > just an internal change / addition. But certainly some overview on > internals should be done so writing this instead. > > > > I decided to use cmocka in that PR because I had some experience with > that. It's quite small and still very powerful and allow vast mocking > options. It's a bit manual but it gives a bigger control over the mock. It > relies on --wrap linking option that replaces original functions with > wraps. This is however available only on Linux (or maybe some other Unix > variants) but doesn't work on MacOS or Windows. The developers that want to > use it on those platforms would need to use some Linux Virtualisation > option (e.g. Docker). It also requires static library which is supported by > embed SAPI that can be compiled statically. That limits number of > extensions to use but the main use cases don't really have deps so it > should be fine. > > > > I did also some research into the other mocking libraries in C. There is > a Unity with CMock, FFF and some C++ libs like GUnit, Criterion and > Trompeloeil that I looked into. I quickly discarded GUnit and Trompeloeil > as they relay on C++ virtual methods and require wrapping C code to C++ > which is very inconvenient. FFF seems too simple and maybe quite inflexible > for our needs as well. Criterion also optionally uses wrap so I didn't see > much advantages compare to cmocka. So it left Unity with CMock that allows > generating custom mocks using a Ruby script. That seemed initially quite > nice but after spending around two hours with trying to make it works for > PHP codebase, I just gave up. It gets quite messy for complex scenarios and > I just didn't figure out how to nicely mock libc functions without any > modification to php-src. > > > > In terms of CI. It has got its own build which is very simple and it > tests just specific parts so we could just limit it to run only for changed > files which might be quite convenient. > > > > So the proposed PR is probably the only reasonable unit testing that I > can come up with. I think it should be completely optional initially for > people to use - more like an experiment. If it becomes used, then good of > course. And if it becomes pain, we can just get rid of it. Has anyone got > any objections to get this merged? If not I plan to merge it early in > January. > > > > Cheers > > > > Jakub > > > > I'm assuming that uses ELF symbol interposition or something like that, > which is why it seems Linux/BSD specific. That seems fragile to me. > > It is replaced when linking the program by ld (see https://man7.org/linux/man-pages/man1/ld.1.html - the --wrap option for more info). The flag should be stable on linux and it is not really fragile as few mocking libraries are built around it. > I think currently for wanting to test C functions, we're adding custom > functions into ext/zend_test and writing PHPT. Would this work? We > already have that, after all. If not, it'd be helpful to list the > challenges that approach faces. ext/zend_test is just for normal ext testing that does not require any mocking. What I need is to test certain flow where mocked function returns specific response / set output params in certain way. That requires mocking that function and setting exceptions. This is not possible in zend_test. The only exception there is https://github.com/php/php-src/blob/284c4e3e318a75b8133b52c153bc912b48d1bbab/ext/zend_test/test.c#L1499-L1514 which mocks copy_file_range but it uses dlsym with RTLD_NEXT to replace the function in runtime. I was actually checking if any mocking lib uses dlsym but it is not as it's probably a bit more fragile than wrap. And without the mocking lib, it's not really useful because we would have to re-implement what they do to support expectations from different tests (some global state manager) and other features. So I didn't really go there... Regards Jakub