advise needed: storing huge structure persistent or alternative approach

busnet

Hi.
I've got the following problem: I have a huge xml-file that represents
a street map of a city. Each street is stored as a couple of
coordinates (start, end and all the turns).
When I set a marker in a google map, I want to find all streets that
are near this marker position.
I now parse the whole file as a simpleXML object, and calculate the
distances to each street segment (using linear algebra algorithms,
that is, constructing the vector from two points and computing the
distance from the given point to this vector).
Up to now it works like this: I set a marker and the gmaps method
"gdownload()" calls a server side php script, which
1. reads the xml file into an simplexml object
2. for every two coordinates that belong to the same street, it
calculates the vector and computes its distance to the point
3. returns the nearest x streets (as polylines) back to the client

It repeats this, when I set the next marker.

Problems/need for ideas:
1) It always reads the XML file anew, when I set another marker. Is
there some way to cache this or a more intelligent approach? I tried
saving the whole Simplexml-object in a session variable, but it was
too huge. I guess I could create some tables in mysql, but I'm not
sure thats the best solution yet.
2) Not php-specific: Do you have any ideas how to speed up the process
of finding the nearest lane(s)? I tried to consider only those streets
whose endpoints are within a certain range from the given marker-
point, but it was not considerably faster and if there was a street
thats very long and straight the endpoints may be far away, but the
street itself is quite near.

Sebastiaan 'CrashandDie' Lauwers

busnet wrote:
> Hi.

Hi,

> Problems/need for ideas:
> 1) It always reads the XML file anew, when I set another marker. Is
> there some way to cache this or a more intelligent approach? I tried
> saving the whole Simplexml-object in a session variable, but it was
> too huge. I guess I could create some tables in mysql, but I'm not
> sure thats the best solution yet.

Serialize () the simpleXML-object, and save that in a text file ?

On load, $xmlSimpleObject = unserialize (file_get_contents ()) ?

Dunno if it's faster, but it just might be.

> 2) Not php-specific: Do you have any ideas how to speed up the process
> of finding the nearest lane(s)? I tried to consider only those streets
> whose endpoints are within a certain range from the given marker-
> point, but it was not considerably faster and if there was a street
> thats very long and straight the endpoints may be far away, but the
> street itself is quite near.

Can't help you with that, sorry.

HTH,

S/

busnet

On 12 Jul., 14:19, Sebastiaan 'CrashandDie' Lauwers <crashanddie
+> wrote:
> busnet wrote:
> > 1) It always reads the XML file anew, when I set another marker. Is
> > there some way to cache this or a more intelligent approach? I tried
> > saving the whole Simplexml-object in a session variable, but it was
> > too huge. I guess I could create some tables in mysql, but I'm not
> > sure thats the best solution yet.
>
> Serialize () the simpleXML-object, and save that in a text file ?
>
> On load, $xmlSimpleObject = unserialize (file_get_contents ()) ?
>
> Dunno if it's faster, but it just might be.

I also thought about that, but it wont be that much faster, I'm
afraid. A file still has to be read every time.

> > 2) Not php-specific: Do you have any ideas how to speed up the process
> Can't help you with that, sorry.

Thanks anyway.

Sebastiaan 'CrashandDie' Lauwers

busnet wrote:

> I also thought about that, but it wont be that much faster, I'm
> afraid. A file still has to be read every time.

Not exactly.

It is possible to store a file in the RAM (copy would have to be saved
regularly on the disk so to not lose the contents if there was a
failure, but still).

What I mean is, that a good operating system, is going to cache the
files that are under heavy access. GNU/Linux can do this, if the kernel
is of a decent flavour. I recall the Debian one doing it without any
problems.

Also, hacking the operating system quite severely, I'm pretty sure it
should be experimentally possible to map the RAM, and allocate some
place you could mount, as such, you would have a very sharp and fast
memory, that would keep any kind of files stored for at least some time,
and reload it, if it hasn't been accessed in a given amount of time.

A last solution, I guess, would be to fork () the application, and keep
one part of the application running in the background, keeping all the
variables and objects in RAM, and trying to get those values back, one
way or another.

Doing this like this, you might in fact be bothered, as PHP has a max
execution time of 45 seconds (default value), and that isn't
specifically *long* ! As such, it might just be easier to run a daemon
in whatever language, (be it PHP, C, C++), and that would act as a
server, you could query easily. The daemon could be started up by the
PHP script, if it is run, and there is no answer, a small exec () and
there you go...

These are all pretty random ideas, I can try to elaborate one of these
if you are interested.

> Thanks anyway.

HTH,

S.