Re: FolderDB - advice wanted

Off topic: If objectionable, sorry; Larry Garfield, didn't you author a text on php? It has been a while and I don't have the hard copy text anymore, but didn't you author a Quick Start Guide about php. It was long enough ago to be on php v4x. > On Jul 23, 2019, at 3:23 PM, Larry Garfield <larry@garfieldtech.com> wrote: > > On Sun, Jul 21, 2019, at 10:54 AM, Rene Veerman wrote: >> Hi. >> >> I like JSON a lot better than the SQL data model, >> but now that i've been using it for a few years, >> i find myself combining file system storage of lots of JSON files in >> many sub-folders into a web app, >> like news gathering with commandline PHP per N minutes from 280+ RSS >> online sources, >> into a PHP web-app that loads the latest news, via a web-powered PHP >> filesystem search of the latest new-items JSON files, stored on a >> filesystem thats got folders like >> ./YYYY/MM/DD/hh/mm/ss/CATEGORY_SUBMENU_PATH leading to JSON files. >> >> so now i want something more flexible. something that can give me a >> web-based treeview of data spanning many JSON-like files (in more >> complicated ways than my news app) and even into other file formats >> stored somewhere on the filesystem of some server. >> >> i plan to license this right from the start under a LGPL typed license, >> because i believe input from other developers is key to making a DB >> system that is not only easy to use, but also very efficient to run. >> >> i'd like the code to stay pure PHP, not even using the object oriented >> functionality of PHP, just plain properly named functions and their >> return values. >> >> i'll name this DB architecture FolderDB, and to prevent web-apps from >> forcing the server to reload (and decode!) many .jsone files upon a >> simple request to change something in some referenced .jsone file, i >> plan to run a single PHP server daemon that will respond to a yet to be >> written ASCII/UNICODE protocol. >> >> the file format itself, has to differ from JSON somewhat i think, and i >> also believe we can lose some of the overhead. >> i post this all here to hear your opinion, especially when it points to >> insurmountable problems related to my designs. >> >> i'm also interested to learn about how to add cloud redundancy to this >> DB architecture, preferably with tiered backup servers, or at least be >> prepared for such expansions of the DB architecture later on. >> >> here's the string based format that i came up with so far : >> >> JSON-Extended file format (ASCII / UNICODE) >> >> ------------------ >> { >> key1 : s|string value, >> key2 : 2, >> key3 : 3.4, >> key4a : t, >> key4b : f, >> key5 : [ >> numbered array >> ], >> key6 : { >> named array >> }, >> key7 : url|/siteData/bla/bla/file.jsone, >> key8 : key|key7/someKeyInsideFileBehindURLofKey7_key1/key1a, >> key9 : >> url|/siteMedia/backgrounds/someFile.png||thumb|/siteMedia/backgrounds/thumbs/someFile.png, >> key10 : >> >> url|/siteData/bla2/bla2a/file.odt||thumb|/siteData/bla2/bla2a/file.odt.thumb.png||thumbGenerator|URL >> } >> ------------------ >> >> key4a = true >> key4b = false >> >> key1 would need it's value escaped, using \| and \, > > > 1) "i'd like the code to stay pure PHP, not even using the object oriented > functionality of PHP, just plain properly named functions and their return values." > > OK, right there you're already lost. There is zero reason to avoid OOP in modern PHP. > Objects are, in modern PHP versions, *faster* and *vastly more memory efficient* than array > structures. (cf > https://steemit.com/php/@crell/php-use-associative-arrays-basically-never) > It's also the API style most current developers will be looking for. So you're already > setting yourself up for failure with that sentence. > > 2) "the file format itself, has to differ from JSON somewhat i think" > > So you're not even using real-JSON, which means you can't use the C-based JSON parser > and need to make your own in user-space. That's going to be dog slow compared to even just > using pure JSON, and as Ashley noted modern DBs are blazing fast at searching/sorting/filtering > behavior. Doing that in PHP will never be able to compete. > > 3) " i'm also interested to learn about how to add cloud redundancy to this DB > architecture, preferably with tiered backup servers, or at least be prepared for such expansions of > the DB architecture later on." > > If you're using a file system as your primary storage, Git is the obvious solution for > data replication. I've toyed with that approach before and while it wasn't a completed > project I think it's promising. > > But seriously, this is not a good approach at all. If you don't like SQL, there are other > options that may fit your data model better. MongoDB is the obvious one (JSON native), but MySQL, > MariaDB, PostgreSQL, and SQLite *all* have (slightly incompatible of course) JSON data types and > query-ability. You can store and query JSON in an SQL database just fine, and it's going to be > waaaaaay faster and waaaaaay more flexible than just using the file system, especially when you > start mixing and matching JSON and relational concepts to leverage both in the places where they > make the most sense. > > --Larry Garfield