RE: [PHP] Building a scalable Control Panel In PHP Question . . .

Well its good that you are using a service. Basically think of the service as being a "call center", and the database being the "que", and the requests from the web script as the "people". Your employee fills out the web from in the browser. Data is submitted Data is stored in the database as a "que" The windows service runs automatically every 30 seconds unless it is currently processing from the last run. The windows service updates each request from the que and then after processing, removes the request from the que. THIS IS HOW I DO IT ON MY NETWORK If you are going to be executing some serious commands that require a lot of system overhead and time that might cause the que to become backed up for a particular machine, I would recommend thinking about distributed computing under the following senerio. I am going to assume the following just for example purposes; 1. You have 10 servers. 2. You are using a central database server that all these machines talk to 3. Each of them has the daemon installed. 3. In your form posts, you are also sending relevant machine information to be stored in the que so that way your daemons will know what machine the job is for. 4. Your employee fills out the web from in the browser. The request requires 3 simple processes that are expected to take a few seconds and 2 major processes that may take up to 10 minutes each to complete. The job is for machine "INTRAVAT2" 5. Request/Data is submitted by user 6. Data is stored in the database as a "que" but each part of the request is strored seperatley as unique requests. So in this example, you would have a total of 5 items (jobs) in the que for this particular request for machine "INTRAVAT2". 7. The windows service runs automatically on all machines every 30 seconds unless it is currently processing from the last run. The service/daemon checks the central database to see if it has any jobs scheduled for itself. 8. If the "INTAVAT2" daemon finds jobs for its machine, which in this example it does, it processes them one at a time in order of importance (pre-determined by the admin). 9. At this time, "INTRAVAT9" looks at the que and sees that no jobs are scheduled for itself but there are some outstanding jobs for "INTRAVAT2". 10."INTRAVAT9" checks out 1 of the big jobs for "INTRAVAT2" and starts processing it. When "INTRAVAT9" is done processing the job, it looks at the destination (INTRAVAT2) to see if it can finish the job and add it to that system. If another process is currently accessing the destination, then "INTRAVAT9" places the finished job into a temporary location and then schedules a new job to finalize the implimentation on "INTRAVAT2". This job is placed at the top of the que for "INTRAVAT2" to finish. Now in this senerio you have to make sure that you lock access to a System if another process is making changes to it. Also, you really are Only going to want to let machines process jobs from other machines If they meet certain criteria. You might have a ranking system like I do. Basically I rank requests on a scale from 1-10 in my daemon. My other Systems only process jobs from other machines if they are a 7 or higher. On my network, I am dealing with large volumes of data. Sometimes it takes 10-60 minutes to process a simple request when dealing with records more than 1 million. So instead of the que getting backed up, the other machines Take up the slack. FYI - I use this senerio for mass email processing. This system handles The mailing, sorting, importing and etc of all related email marketing. Some of the jobs take a few seconds and some of them can take days (when Sending out emails). On my system, if I send out a newsletter or something To 6 million email addresses, all my machines take control and the load is Distributed. When I implimented this system, the network performance was Awsome. The only thing I am always looking to get more of is Internet Bandwidth. I hope this lengthy email helps you out in trying to figure out a strategy. I probably left out a few things here and there... But I am writing this On a whim to help you out. -----Original Message----- From: Jonathan Chum [mailto:jchum@jps.net] Sent: Saturday, September 14, 2002 9:30 PM To: php-general@lists.php.net Subject: Re: [PHP] Building a scalable Control Panel In PHP Question . . . Well, The plan is that the daemon or service will sit alone which will scan the database for changes. The config file can only be processed one at a time. If there are 10 users online at that very moment, one person wants to suspend an account, another wants to create a new account, another wants to delete an account, the list goes on. However, the daemon or service scans the database, making the changes one at a time to insure that the file does not become corrupted. So there will be one instance of the interface for the control panel sitting on any dedicated box to handle user intervent Anytime a new server needs to be added, another daemon/service is installed and run which listens and scans for changes when they occur for that particular server it's scanning changes for. What I'm worried about is that the script in the future may need to execute some commands that might take a while to execute. Therefore it cannot wait for it to complete. However, it should also not start another child process if the previous one is not done to prevent corruption of data. So it should wait till the first child process complete before starting another. There's a lot of ways, but all of them have limitations. Not sure of the best possible way to handle it. . . "Jeremy Wilson" <jwilson@intravative.com> wrote in message news:003401c25c49$269c2310$4ac23342@intravative.com... > Make sure that you implement file locking into your application if you > work directly with the files so that way your config file will not > become corrupt when multiple clients are accessing it. > > I don't know your skill level, but I would recommend writing a Windows > 2000 service in C++ or Visual Basic .NET (you need something > multi-threaded) that will take care of the processing of your database > updates and converting them into your config files. Use PHP to process > the web based "requests" for config file updates, and then use the > Windows 2000 service to process all the requests. The problem with > using php in windows 2000 cron situations at the operating system > level is that it is pretty undependable. > > I would not recommend installing the script in several locations on > all the different boxes. Do you have any NAS (network attached > storage) in your infrastructure? You can setup just one script and > then virtually map the path to that script from apache. That way if > you ever need to upgrade your script or fix a bug, you only have to do > it in one location and not in several. It will make the code, version > management and file administration much easier in the future. > > You can also do the same thing with the windows 2000 service. Set it > up as a network installation and then just use your service control > panel in windows 2000 to map it to the NAS location. Also, with a > windows 2000 service, you can set actions if your service stops > responding or if anything weird happens. > > I have several e-books from Microsoft Press that might help you in > creating your Windows 2000 service. Email me directly if you would > like to get a copy of them and I can email them to you. > > -----Original Message----- > From: Jonathan Chum [mailto:jchum@jps.net] > Sent: Saturday, September 14, 2002 3:14 PM > To: php-general@lists.php.net > Subject: [PHP] Building a scalable Control Panel In PHP Question . . . > > > I've just started work on building a control panel in PHP to manage > web hosting services (Apache, Sendmail, Bind, etc.). > > Considering the security risks involved and possible data corruption > if there are several users online attempting to write to a conf file, > I'm taking the approach of seperating the user interface from the > script that writes to the system. That is, the user only interfaces to > the database. > > Now, I'm trying to figure out how should I take the data from the > database, possibly by scanning the database for changes then write it > to a file. I'm also taking into consideration of how to scale to > Windows 2K. > > I have a few ideas that sounds good, but could be a problem. Let me > explain. . . I was going to code a seperate script that sits on each > server at URL under a SSL link which will have to authenticate to get > through. This script will capture POST data and returns data > accordingly. This allows me to scale to another machine easily by just > installing a script. > > The issue is that if Apache dies, then it needs to send data to a > machine, it will cause some issues. Also, if a POST was made to that > URL, if it takes a while to return data back, Apache could just time > out on the request. > > So that's probably not a good idea. > > Another application I reviewed coded up a daemon using PCNTL to > control child processes or run interactively. This would be feasible > on *NIX machines, but not sure how well it scales to a Windows > machine. Maybe recoding the daemon as a service. > > Is there any other ideas out there would be even better? > > > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php