Re: Re: Net_Gearman Problems

Joe Stump wrote:

Anyone who has worked on Net_Gearman or Sockets have any thoughts?

I've cc'd Chris Goffinet on this email as he's a.) used Net_Gearman with large job sizes and b.) rewrote the socket code recently to rectify some of those issues. Chris, do you have any input on this? Great that would be a big help

Make sure the docs for 0.1.2 let everyone know your json encoding the data now Maybe changing data encoding method should be configurable or at least have an example of changing it by extending whatever you need to extend

So the problem here is multiple and somewhat convoluted ... 1.) Net_Gearman was using JSON before I submitted the first one to PEAR. I had forgotten about it before submitting so I switched it back. 2.) There was a brief moment of time in SVN when json_decode() was called without the second "true" argument to force an array. That should be fixed. If you tried using it while json_decode() was returning objects instead of arrays this would have broken BC. That's fixed in trunk now. 3.) Chris uses Gearman in a multi-lingual environment so we think that JSON is the best solution and don't have plans to offer multiple types of encodings. I'll definitely be mentioning it in my release notes though. Sorry for any confusion this caused while using HEAD from trunk. I figured it out after reading the commit log, i just wanted to make sure no one else got bit by it.

Have you thought of just passing an instance of the Job class to addAbility, the current setup is limiting and i don't really see the benefit

No. The addAbility() just broadcasts which functions a worker can handle to Gearman. I'm merely following the conventions and protocol here. Can you elaborate how you find this limiting or having no benefit (the main benefit is that, without doing so, Gearman thinks your workers can't do anything). This isn't a big deal for me right now i have 3 jobs i'm using but its mainly a style thing. Normally a worker includes every class anyway so i'm not sure what the gain of having Net_Gearman require the code and create the instance is. If the user can do that I get a lot of flexibility plus i don't have to store all my jobs in the dir specified by the define.

Are you going to add a method of doing easy sync calls, background ones are easy with the __call mapping but its 5 lines for a sync call.

You mean sending a single sync call? It makes little sense to send a *single* sync call since Net_Gearman isn't asynchronous. In other words, if you hand off a task that takes 50ms to Gearman via Net_Gearman your calling process will still block for 50ms. There's no point in using a synchronous library and Gearman for this type of task. The real benefit of Gearman is running *multiple* tasks in *parallel*. For instance, if you have 3 tasks that each take 50ms then you speed things up greatly using Gearman because they're all running in parallel. Despite the blocking nature of Net_Gearman you'll still see significant speedups (I've seen 2x - 3x speedups typically) running these things in parallel. I'm Gearman less for straight up parallel performance and more for job distribution. I do mainly background calls but I have call that needs to be synchronous because it being completed is a dependency on the rest of the actions being taken.

I'll send a patch with what i have in mind. -josh