Best way to monitor php-fpm container liveness on Kubernetes
| From: | Adam Hamsik | Date: | Sat, 22 Jan 2022 20:00:15 +0000 |
| Subject: | Best way to monitor php-fpm container liveness on Kubernetes | ||
| Groups: | php.internals | ||
| Request: | Send a blank email to internals+get-116913@lists.php.net to get a copy of this message | ||
Hello,
We are using PHP for our application backends, this works very well as we have developed s imple way
to clone them with minimal effort(they can be very similar). For our orchestration we are using
Kubernetes (>= 1.21). Our application pod generally contains NGINX + php-fpm and fluentbit for
log shipping. We generally want to have LivenessProbe(for an simple explanation this is a simple
check which is run against our pod to verify if it's alive, if it fails particular container
will be restarted).
This works very we(we are also using swoole which is roughly 80-70% better)l, but in certain
unstable situations when we see higher application latency (db problem or a bug in our application).
We often experience problems, because pods are falsely marked as dead (failed liveness probe and
restarted by kubelet). This happens all processes in our static pool are allocated to application
requests. For our livenessProbe we tried to use both fpm.ping and fpm.status endpoints but both of
them behave in a same way as they are managed with worker processes.
I had a look at pgp-src repo if e.g. we can use signals to verify if application server is running
as a way to go around our issue. When looking at this I saw fpm-systemd.c which is a SystemD
specific check. This check reports fpm status every couple seconds(configurable to systemd). Would
you be willing to integrate similar feature for kubernetes. This would be based on a pull model
probably with and REST interface.
My idea is following:
1) During startup if this is enabled php-fpm master will open a secondary port pm.health_port(9001)
and listen for a pm.health_path(/healtz)[2].
2) If we receive GET request fpm master process will respond with HTTP code 200 and string ok. If
anything is wrong (we can later add some checks/metrics to make sure fpm is in a good state). If
we do not respond or fpm is not ok our LivenessProbe will fail. based on configuration this will
trigger container restart.
Would you be interested to integrate feature like this ? or is there any other way how we can
achieve similar results ?
[1] https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#when-should-you-use-a-liveness-probe
[2] https://kubernetes.io/docs/reference/using-api/health-checks/
Best Regards,
Adam.
Adam Hamšík
Co-founder & CEO
Mobile: +421-904-937-495
www.lablabs.io