Bug #59909 [Opn->Asn]: Gearman Workers Run Away causing gearmand cpu load to 99%

Edit report at https://bugs.php.net/bug.php?id=59909&edit=1 ID: 59909 Updated by: felipe@php.net Reported by: mike at digitalstruct dot com Summary: Gearman Workers Run Away causing gearmand cpu load to 99% -Status: Open +Status: Assigned Type: Bug Package: gearman Operating System: RHEL 5.6 PHP Version: 5.3.5 Assigned To: hradtke Block user comment: N Private report: N Previous Comments: ------------------------------------------------------------------------ [2011-08-22 22:50:51] mike at digitalstruct dot com Unfortunately it is extremely hard to rule out either the extension or libgearman at this point. I've gone back and forth between both areas but I do not see any clear definition between the two in where it is breaking. Any ideas to help diagnose? ------------------------------------------------------------------------ [2011-08-22 20:07:36] hradtke@php.net Did we rule out libgearman? ------------------------------------------------------------------------ [2011-08-22 17:12:36] mike at digitalstruct dot com Description: ------------ See the following bug report: https://bugs.launchpad.net/gearmand/+bug/802850 It seems like it happens at random and only under heavier load. When the workers are killed the server returns to normal and after the restart of the worker it seems to function fine. What I've noticed is two main things between a working worker and a non-working worker (basically when things are fried). The working ones block and the non-working ones stop blocking but simply DDoS the gearmand process. Expected result: ---------------- I would think that the duration of time should not matter that the worker is running. The fact that it stops blocking should never happen while it is sitting in it's while loop. The fact that it DDoS the gearmand process means that it is likely losing some pointer to the connection and unable to resolve itself. It does not give any useful error message but continues in the way above until killed. Actual result: -------------- Working: poll([{fd=13, events=POLLIN}], 1, 1000) = 0 (Timeout) rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 sendto(13, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12 recvfrom(13, "\0RES\0\0\0\n\0\0\0\0", 8192, 0, NULL, NULL) = 12 sendto(13, "\0REQ\0\0\0\4\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12 poll([{fd=13, events=POLLIN}], 1, 1000 <unfinished ...> Non-Working: getsockopt(13, SOL_SOCKET, SO_ERROR, [284636038280773632], [4]) = 0 sendto(13, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12 sendto(13, "\0REQ\0\0\0\4\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0) = 12 poll([{fd=13, events=POLLIN}], 1, 1000) = 1 ([{fd=13, revents=POLLIN}]) getsockopt(13, SOL_SOCKET, SO_ERROR, [284636038280773632], [4]) = 0 sendto(13, "\0REQ\0\0\0'\0\0\0\0", 12, MSG_NOSIGNAL, NULL, 0 <unfinished ...> ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=59909&edit=1