Re: Identifying causes of "seems busy" entries

Thank you all for your incredible feedback. I'm going to study/review it over the coming days and follow up with questions :-) On Wed, Aug 6, 2025 at 4:46 AM AllenJB <php.lists@allenjb.me.uk> wrote: > On 06/08/2025 00:15, Alex wrote: > > Hi, > > I have a fedora40 system with apache-2.24.62 and php-8.3.20 (upgrading > > to fedora42 soon). I'm trying to identify how best to optimize php-fpm > > for problems involving "seems busy" errors. I'm sure this is a FAQ, > > but I'm really stuck. I've been using php-fpm and tweaking it for > > years, but our site is constantly growing and I don't know what I'm > > missing. Apache and php-fpm are the only processes on the system. The > > server is a Xeon Gold 6128 @ 3.4Ghz with 128GB of RAM. > > > > [05-Aug-2025 15:36:58] WARNING: [pool mysite] seems busy (you may need > > to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 > > children, there are 91 idle, and 137 total children > > Ref: > > > https://github.com/php/php-src/blob/1820dd9b61319b79d6a2f7570c24449c1a35250b/sapi/fpm/fpm/fpm_process_ctl.c#L417 > > This message is issued whenever PHP-FPM considers spawning 8 or more > children at the same time. It does not take account of the number of > idle children or the maximum number of children. (It seems to me the > conditions required to emit this message could be improved, but this is > how it's currently implemented) > > (I say "considers", because from the code below and a very brief search > of where this value is set, it looks like it will only spawn this many > if there aren't enough idle servers already - the number to consider is > based on doubling the number of processes spawned in the previous cycle, > so this message is in fact emitted when FPM spawns processes 4 cycles in > a row and hasn't reached pm.min_spare_servers, if I'm reading this > correctly) > > Based on my reading of the FPM process management code, and your > settings (from which I presume you run a relatively busy server), I > could see this message being emitted quite frequently, particularly > after FPM has just been restarted. For example: > > * FPM starts and starts 110 workers (start_servers) > * 10 workers are immediately busy (and the number of busy workers > remains at that level for the next 5 FPM process management cycles) > * cycle 1: FPM spawns 1 additional worker (wanted workers to reach > minimum is now 9) > * cycle 2: FPM spawns 2 additional workers (wanted workers to reach > minimum is now 7) > * cycle 3: FPM spawns 4 additional workers (wanted workers to reach > minimum is now 3) > * cycle 4: FPM emits the above message and considers spawning 8 > additional workers (but this gets floored to 3, the minimum required to > reach min_spare_servers) > > You'd see a similar situation in any case where the number of busy > workers increases by 8+ and stays at that level (or higher) for at least > 4 FPM process management cycles. > > (I could be wrong but I think a process management cycle occurs every 1 > second, so the above example scenario occurs over 5 seconds) > > The message is emitted each time the process management cycle runs, so > if the number of busy workers is 16+, the message will be emitted twice > in a row, 32+ will be 3 times in a row, and so on. > > The point I try to make here is that FPMs heuristic for "pool seems > busy" is fairly naive, and may trigger quite frequently on servers that > exceed a certain level of traffic (or at least changes in the amount of > traffic that is maintained for 5 seconds or more) > > > I note that you've implemented the FPM status page, but are you aware > that you can implement your own monitoring and alerting by utilizing it > (specifically the openmetrics format) with tools such as Grafana + > Prometheus? This would allow you to set up your own alerts based on your > setup, rather than relying on FPM's log warnings. > > For the available formats and how to use them see > > https://www.php.net/manual/en/fpm.status.php#fpm.status.parameters > >