Re: big data base

just a few thoughts myself bastien

From: "Kevin" <aliaghan@hotmail.com> To: php-db@lists.php.net Subject: Re: [PHP-DB] big data base Date: Wed, 16 Mar 2005 04:32:49 +0100 "Martin Norland" <martin.norland@stjude.org> wrote in message news:422F6517.1010506@stjude.org... luis medrano wrote:

Hi All, I need you help. I need to design a very big data base, around 900GB or more of information but I never had design or manage this magnitude of information before. I really appreciate if you can point me to documentation to how can do this properly or you can help me with some experiance you have doing this.

I wasn't going to chime in, but after reading some replies I feel I have to. Bottom line - you need to give more details. The design of the database will not depend greatly on the amount of data inside it, unless there are operations/calculations to be done on/with it. Most people talk number of rows / tables / read|write operations / joins / etc. when talking database design, instead of size - as with proper indexes, the size can affect things very little. Ok... I don't completely agree with the "Design of the database will not depend greatly on the amount of data", unless of course I am misinterpreting. If the size of the data and the usage rather simple one mind choose to ignore some of the rules when developing databases. I have seen databases pass my desk where parts of it aren't even in 3NF (3rd degree Normal Form). Then again, if the database is complex one might need to get into 4NF or BCNF. Also the design when it comes down to how to organise the database, storage wise, one might choose to seperate the database into several pieces - store one piece here and another there. It would suggest that the design is rather important if you do not wish to see the performance drop like a brick in water.

One thing to consider here is that it IS acceptable to not have fully normalized tables in a web based environment. Under high load situations, avoiding joins can mean significant performance gains. For example I split up a four table join query into parts and moved some of the logic into the ASP application from the db. With this I was able to take a query that ran in 4-5minutes for 100 records into a query that can produce 1000+ (depending on system configuration) records in about 10 seconds. I agree that design is important, but Martin's point is that the amount of data is less relevant than the design. Whether you have 10,000 rows ro 10,000,000 rows won't affect the db much if its well designed and that includes indexing.

Also if the data in the database is small you can easily get away with design mistakes, like bad index and the sort. While when the data is extensive one needs fast ways to get through the data. A table of several tens of thousands of records or more can take a long time to process when not having the right indexes and design schema's. I am wondernig though that if the size of the db is anyway related to the complexity and size of the project if PHP is the right choice of language. I would most likely move towards something like Java. Well just a thought...

Nope, language is dictated by the environment and what the goal of the project is and how comfortable you are with that environment. Java and other complied environments may give you some speed increases, it is likely a pig to maintain (if there are frequent changes to be made or new features to be added). For us, a .com, basically, involved in retail security, our systems (ASP and PHP based) run 33 queries a second and that number is only going to grow as our customer base increases. Our biggest issue is not the language or the db, but the crappy windows servers that are fraught with issues.

For example - if you're "designing" a database to store 20GB raw movie files - then you might want to rethink storing the gigs of data inside the database, then suddenly you aren't working with a "large" database at all. So, if you can't give more details - you're going to have to search on your own. Otherwise, you might want to speak up with some more info so you can get better recommendations. My only suggestion with the information you've given - is basically what Bastien said - multiple big iron and big fast hard disks / SAN. Other than that, I can only suggest that you design it well P.S. - I wanted to toss in a troll reply about trying SQLLite, since it "comes with PHP!" - but that's mean P.P.S. - I was also tempted to link to a "great tutorial on this" - an arbitrary resume writing tutorial, as a mean spiteful joke about being in over your head - but that's just too mean. It all depends on the system requirements, your budget, and your time. Just because someone may be in over his head one shouldn't try? Didn't we all have our first big project at some point? cheers, -- - Martin Norland, Sys Admin / Database / Web Developer, International Outreach x3257 The opinion(s) contained within this email do not necessarily represent those of St. Jude Children's Research Hospital. -- PHP Database Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php