Re: Net_Gearman Problems

On Jun 4, 2008, at 10:20 PM, Joshua Eichorn wrote:

Net_Gearman is using a non blocking socket, which means if you want to send a lot of data say 300k and your link isn't fast, say an ssh tunnel from a EC2 node then the socket_write will fail with EAGAIN. Now commenting out socket_set_nonblock on line 154 fixes my problem but i'm guessing thats not the correct solution since it should make everything slower. The manpage for send says When the message does not fit into the send buffer of the socket, send() normally blocks,

     unless the socket has been placed in non-blocking I/O mode.  In non-blocking mode it would
     return EAGAIN in this case.  The select(2) call may be used to determine when it is possi-
     ble to send more data.

So in theory you can use socket_select in this case and then break the data up into smaller chunks yourself Anyone who has worked on Net_Gearman or Sockets have any thoughts?

I've cc'd Chris Goffinet on this email as he's a.) used Net_Gearman with large job sizes and b.) rewrote the socket code recently to rectify some of those issues. Chris, do you have any input on this?

Make sure the docs for 0.1.2 let everyone know your json encoding the data now Maybe changing data encoding method should be configurable or at least have an example of changing it by extending whatever you need to extend

So the problem here is multiple and somewhat convoluted ... 1.) Net_Gearman was using JSON before I submitted the first one to PEAR. I had forgotten about it before submitting so I switched it back. 2.) There was a brief moment of time in SVN when json_decode() was called without the second "true" argument to force an array. That should be fixed. If you tried using it while json_decode() was returning objects instead of arrays this would have broken BC. That's fixed in trunk now. 3.) Chris uses Gearman in a multi-lingual environment so we think that JSON is the best solution and don't have plans to offer multiple types of encodings. I'll definitely be mentioning it in my release notes though. Sorry for any confusion this caused while using HEAD from trunk.

Have you thought of just passing an instance of the Job class to addAbility, the current setup is limiting and i don't really see the benefit

No. The addAbility() just broadcasts which functions a worker can handle to Gearman. I'm merely following the conventions and protocol here. Can you elaborate how you find this limiting or having no benefit (the main benefit is that, without doing so, Gearman thinks your workers can't do anything).

Are you going to add a method of doing easy sync calls, background ones are easy with the __call mapping but its 5 lines for a sync call.

You mean sending a single sync call? It makes little sense to send a *single* sync call since Net_Gearman isn't asynchronous. In other words, if you hand off a task that takes 50ms to Gearman via Net_Gearman your calling process will still block for 50ms. There's no point in using a synchronous library and Gearman for this type of task. The real benefit of Gearman is running *multiple* tasks in *parallel*. For instance, if you have 3 tasks that each take 50ms then you speed things up greatly using Gearman because they're all running in parallel. Despite the blocking nature of Net_Gearman you'll still see significant speedups (I've seen 2x - 3x speedups typically) running these things in parallel. So, that all being said, my answer currently is "no", but I'm happy to look at any patches you might have in mind. --Joe