Unable to run private Docker instance

[clarknova]

Hello, i am trying to run a private Docker instance of the openrouteservice.
I can run a federal state like North-Rhein Westfalia with no problem using -Xms4g -Xmx4g.
Trying to use the Germany extract from Geofabrik crashes after about 2 hours of processing with -Xms8g -Xmx8g. Docker stats shows that the container never uses more than about 4.2 gb of ram before crashing.
I am using a fresh 16 gb ram digital ocean droplet with nothing but docker installed.

at java.lang.Thread.run(Thread.java:748)
ors-app    | Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
ors-app    | 13 Nov 19:06:33 ERROR [routing.RoutingProfileManager] - java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space - problem when allocating new memory. Old capacity: 1048576, new bytes:1453111424, segmentSizeIntsPower:20, new segments:1386, existing:1
ors-app    | java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space - problem when allocating new memory. Old capacity: 1048576, new bytes:1453111424, segmentSizeIntsPower:20, new segments:1386, existing:1
ors-app    | 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
ors-app    | 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
ors-app    | 	at heigit.ors.routing.RoutingProfileManager.initialize(RoutingProfileManager.java:218)
ors-app    | 	at heigit.ors.routing.RoutingProfileManager.getInstance(RoutingProfileManager.java:66)
ors-app    | 	at heigit.ors.servlet.listeners.ORSInitContextListener.lambda$contextInitialized$0(ORSInitContextListener.java:43)
ors-app    | 	at java.lang.Thread.run(Thread.java:748)
ors-app    | Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space - problem when allocating new memory. Old capacity: 1048576, new bytes:1453111424, segmentSizeIntsPower:20, new segments:1386, existing:1
ors-app    | 	at com.graphhopper.routing.lm.LMAlgoFactoryDecorator.loadOrDoWork(LMAlgoFactoryDecorator.java:304)
ors-app    | 	at com.graphhopper.GraphHopper.loadOrPrepareLM(GraphHopper.java:1329)
ors-app    | 	at com.graphhopper.GraphHopper.postProcessing(GraphHopper.java:942)
ors-app    | 	at heigit.ors.routing.graphhopper.extensions.ORSGraphHopper.postProcessing(ORSGraphHopper.java:497)
ors-app    | 	at com.graphhopper.GraphHopper.process(GraphHopper.java:719)
ors-app    | 	at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:688)
ors-app    | 	at heigit.ors.routing.graphhopper.extensions.ORSGraphHopper.importOrLoad(ORSGraphHopper.java:158)
ors-app    | 	at heigit.ors.routing.RoutingProfile.initGraphHopper(RoutingProfile.java:185)
ors-app    | 	at heigit.ors.routing.RoutingProfile.<init>(RoutingProfile.java:130)
ors-app    | 	at heigit.ors.routing.RoutingProfileLoader.call(RoutingProfileLoader.java:37)
ors-app    | 	at heigit.ors.routing.RoutingProfileLoader.call(RoutingProfileLoader.java:21)
ors-app    | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
ors-app    | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
ors-app    | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
ors-app    | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
ors-app    | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
ors-app    | 	... 1 more
ors-app    | Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space - problem when allocating new memory. Old capacity: 1048576, new bytes:1453111424, segmentSizeIntsPower:20, new segments:1386, existing:1
ors-app    | 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
ors-app    | 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
ors-app    | 	at com.graphhopper.routing.lm.LMAlgoFactoryDecorator.loadOrDoWork(LMAlgoFactoryDecorator.java:300)
ors-app    | 	... 16 more
ors-app    | Caused by: java.lang.OutOfMemoryError: Java heap space - problem when allocating new memory. Old capacity: 1048576, new bytes:1453111424, segmentSizeIntsPower:20, new segments:1386, existing:1
ors-app    | 	at com.graphhopper.storage.RAMDataAccess.ensureCapacity(RAMDataAccess.java:108)
ors-app    | 	at com.graphhopper.routing.lm.LandmarkStorage.createLandmarks(LandmarkStorage.java:215)
ors-app    | 	at com.graphhopper.routing.lm.PrepareLandmarks.doWork(PrepareLandmarks.java:135)
ors-app    | 	at com.graphhopper.routing.lm.LMAlgoFactoryDecorator$1.run(LMAlgoFactoryDecorator.java:290)
ors-app    | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
ors-app    | 	... 6 more

[adam]

Hi @clarknova ,
have a look at your app.config that you use and see if you are telling it to use two threads for landmarks during the build process. If so, I would recommend to reduce it to one as that can sometimes cause too much memory usage.

"methods": {
   "ch": {
       "enabled": true,
       "threads": 2,
       "weightings": "fastest"
  },
  "lm": {
        "enabled": true,
        "threads": 2,
        "weightings": "fastest,shortest",
        "landmarks": 24
   }
 }

Also, when building larger areas you should limit it to one profile being built per run.

Docker in general can be a bit troublesome to get settings right for though, so it may just be a case of needing to tweak around settings

[clarknova]

Hello, its actually a bug in regard how openrouteservice is setup.
The environment variables in the docker-compose.yml are never used by tomcat because this part in the docker file sets them to empty string.

RUN echo "CATALINA_OPTS=\"$CATALINA_OPTS\"" >> /usr/local/tomcat/bin/setenv.sh
RUN echo "JAVA_OPTS=\"$JAVA_OPTS\"" >> /usr/local/tomcat/bin/setenv.sh

Its actually quite hard to debug because the content of setenv seems to be specific to tomcat only. So when you look inside the container the global environment variables seem fine, but are overridden by the content of setenv.

On top of that. Starting with empty java_opts makes the process just take 1/4 of the available ram.
So this poor soul is using a machine with over 120 gb of ram just to run North America because eventually 1/4 of ram was enough:

Source point(s) [0, 1, 2, 3, 4, ..., n] out of bounds

Hello. I’m fairly new on this subject. I’m having an issue with the /ors/matrix API endpoint of a deployed instance of openrouteservice (an AWS server deployed using docker). I’ve downloaded the data I needed (USA data) from here. I made the corresponding changes to the docker-compose.yml (OSM_FILE: ./docker/data/north-america-latest.osm.pbf). I removed the graphs folder as suggested here, but I’m getting the following error: API error response: b’{“error”:{“code”:6010,“message”:“Source poin…

So my quick fix was just commenting those two lines out. This obviously breaks building the image directly. I can do a pull-request with an more elegant fix next week, but should be easy enough to fix regardless.

[adam]

OK thanks, there is actually a PR which will get merged soon which is to do with docker and the environment variables which could be the same thing, so that will go into the next release.

My experience of Docker is pretty limited and I normally build things using tomcat rather than Docker

[clarknova]

Yes its the same Problem. But the fix is terrible. Every time you want to change the environment variable you have to rebuild the image from scratch.