maps-dockercompose with full planet

[ghost]

Some weeks ago I did my first tries with maps-docker.compose (with nominatim). I used Germany only and it worked fine.
The only issue I identified with this setup was that OSRM does not use my own nominatim-instance but still uses http://nominatim.openstreetmap.org/
This is caues by this.options.geocoder = new L.Control.Geocoder.Nominatim();

The next try was the whole planet. After adjusting the postgres setup I startet the environment on a 8core/64GB cloud-server. As expected the initial load took some time (rendered-initdb 4days). After this 1st step finished rendered-updatedb and nominatim-initdb startet. This

The file planet.osm.pdb is approx 50GB and the load took at least 1,5h. Sometime after midnight the file planet.osm.pdf and it's md5-file are updated on the source. When the download of the pdb-file was active at that time the md5 will not match, (it now matches to the new pdb-file).
Its better to first load the md5 (takes 1 second or so) and then the pdb-file, this could save some hours.

Because rendered-initdb took some days the container renderd-updatedb has to update sseveral days. A day for the whole planet took in my environment up to 14 hours. The last step for each day in rendered-updateddb is the merge the changes into planet.osm.pdb. This also cause an issue with the md5 of the pdb-file when nominatim-initdb is loadloading the file at the same time (nominatim-initdb will start the download again). Becuase the merge of the changes took >8hours/day I commented it out, so the procerssing of the

Infos on planet:
Storage 2.2TB (grow some 100MB/day).
Runtimes:
rendered-initdb 4days
nominatim-initdb 9days

1. run rendered-updatedb 12hours
2. run nominatim-updatedb 2 days
   next runs rendered-updatedb 3h/day
   next runs nominatim-updatedb 3h/day without osmosis --apply-changes, otherwise 12h/day

I need 2 turns of nominatim-initdb because I started with a 2TB volume only, so it terminates during the indexing process. In this case it would be great when the processes are not only restartable - as now - but start at the step where the error occured, this could save a lot of time (in my case over one week).

Currently renderd and nominatim are working fine. osrm returns memory exceptions after some runtime (I have to investigae).

[ghost]

One smal issue.
Due to the long processingtime of the changes of one day the loay is moving the day, because the sleep at the end of a loop is fixed (defaule 86400 seconds). To fix the processing of the changes to a time during the night (for example 03:00) the sleep time should be reduced by the processing time.

[chesty]

Hi Simon, thanks for the report.

This is what I used to use before my job changed 7 months ago and I shared it in case others found it useful. I no longer use it myself, no one besides me used it as far as I know and it worked OK for me in my environment with a small subset of the planet. I haven't touched it in 6 months, all the images are out of date.

I don't have the interest at this time to continue working on it. Feel free to fork it and fix all the problems.

I hope you understand.

[tooneamelter]

@uwesimon
Question for you. I know this is an old thread but I am now using the planet osm.pbf and it is importing just fine. However, I am unsure of how to utilize @chesty osm.env to provide an update url. How did you accomplish this?

[ghost]

I used a German mirror site but I think it should look everywhere the same, except the base URL.
OSM_PBF_URL=https://ftp.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-latest.osm.pbf
OSM_PBF_UPDATE_URL=https://ftp.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day/