Crash when exporting world-wide dataset

[georgbachmann]

I did import a planet.osm file using osmosis, but only streets. I then did some modification to the data and now I'd like to export that data again for further processing. I did so before for a smaller dataset and it worked fine, but now that I am using it on the whole world, it always fails about an hour or so in.

To export the data I use the following command:

osmosis --read-pgsql host="localhost:5432" database="my-db" user="my-user" password="my-password" outPipe.0=pg --dd inPipe.0=pg outPipe.0=dd --write-pbf inPipe.0=dd file=output.osm.pbf

Then as mentioned it works fine for about an hour and then throws some exceptions and fails.
Did anybody export a whole-planet dataset again using osmosis? My system should be more than capable. It hast about 300GB of RAM, 50 Core-CPU and plenty of fast storage.

Any tipps on making it go through would be greatly appreciated :)

Mar 18, 2023 10:23:05 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
Mar 18, 2023 10:23:05 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Mar 18, 2023 10:23:05 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Mar 18, 2023 10:23:05 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Mar 18, 2023 10:23:06 AM org.openstreetmap.osmosis.core.database.DatabaseLocker <init>
WARNING: Locking functions do not exist in database. Disabling locking.
Mar 18, 2023 11:17:03 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-pgsql failed
java.lang.RuntimeException: java.lang.NullPointerException
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDatasetReader.run(PostgreSqlDatasetReader.java:59)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException
        at java.base/java.io.DataOutputStream.writeUTF(DataOutputStream.java:347)
        at java.base/java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
        at org.openstreetmap.osmosis.core.store.DataOutputStoreWriter.writeString(DataOutputStoreWriter.java:108)
        at org.openstreetmap.osmosis.core.domain.v0_6.Tag.store(Tag.java:60)
        at org.openstreetmap.osmosis.core.domain.v0_6.TagCollectionImpl.store(TagCollectionImpl.java:70)
        at org.openstreetmap.osmosis.core.domain.v0_6.CommonEntityData.store(CommonEntityData.java:230)
        at org.openstreetmap.osmosis.core.domain.v0_6.Entity.store(Entity.java:154)
        at org.openstreetmap.osmosis.core.domain.v0_6.Node.store(Node.java:201)
        at org.openstreetmap.osmosis.core.store.BaseObjectWriter.writeObject(BaseObjectWriter.java:63)
        at org.openstreetmap.osmosis.core.store.SegmentedObjectStore.add(SegmentedObjectStore.java:130)
        at org.openstreetmap.osmosis.core.store.ChunkedObjectStore.add(ChunkedObjectStore.java:78)
        at org.openstreetmap.osmosis.core.sort.common.FileBasedSort.flushAddBuffer(FileBasedSort.java:93)
        at org.openstreetmap.osmosis.core.sort.common.FileBasedSort.add(FileBasedSort.java:118)
        at org.openstreetmap.osmosis.core.database.SortingStoreRowMapperListener.process(SortingStoreRowMapperListener.java:38)
        at org.openstreetmap.osmosis.core.database.SortingStoreRowMapperListener.process(SortingStoreRowMapperListener.java:17)
        at org.openstreetmap.osmosis.pgsnapshot.common.RowMapperRowCallbackListener.processRow(RowMapperRowCallbackListener.java:46)
        at org.springframework.jdbc.core.JdbcTemplate$RowCallbackHandlerResultSetExtractor.extractData(JdbcTemplate.java:1575)
        at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:465)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:477)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:482)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.getFeaturelessEntity(EntityDao.java:208)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.iterate(EntityDao.java:253)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.EntityDao.iterate(EntityDao.java:279)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.impl.PostgreSqlDatasetContext.iterate(PostgreSqlDatasetContext.java:214)
        at org.openstreetmap.osmosis.dataset.v0_6.DumpDataset.process(DumpDataset.java:46)
        at org.openstreetmap.osmosis.pgsnapshot.v0_6.PostgreSqlDatasetReader.run(PostgreSqlDatasetReader.java:57)
        ... 1 more

Mar 18, 2023 11:17:03 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:321)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:234)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

I am also running on the latest version of osmosis: INFO: Osmosis Version 0.48.3