Route calculation taking forever, using only a fraction of one CPU core

Hi,

I managed to host my own ORS server with Docker and a global map on a large server. However, a standard request like http://localhost:8080/ors/routes?profile=foot-walking&coordinates=8.676581,49.418204|8.692803,49.409465 as mentioned in the Github instructions is taking over 3 minutes to complete … During this time, I do not see any CPU spike, the java process is using 20 % of one of many cores, so physical resources should not be the problem.
Peculiars about my configuration (app.config and docker-compose.yml provided below): 1) I use very large limits for all parameters in app.config, because I need to calculate thousands, partly quite obscure and long routes for a scientific purpose and do no want some of these calculations to fail because of server limits.
2) In the source code (in openrouteservice/src/main/java/org/heigit/ors/routing/RoutingProfile.java), I set “index.high_resolution” to 250000 (as recommended here https://ask.openrouteservice.org/t/can-not-find-point-within-radius/1543/7) because I need to get results even on points that are far off the next reachable point.
As I understood my modifications, they are just raising the limits in case a “difficult” route has to be calculated, but maybe one of them is nevertheless causing the problem? No logs (neither docker/logs/ors nor docker/logs/tomcat) come up during the execution of route calculations.
Thank you very much for your help in advance!

app.config:

{
   "ors":{
     "info": {
       "base_url": "https://openrouteservice.org/",
       "support_mail": "support@openrouteservice.org",
       "author_tag": "openrouteservice",
       "content_licence": "LGPL 3.0"
      },
      "services":{
         "matrix":{
            "enabled":true,
            "maximum_locations":400,
            "maximum_visited_nodes":200000,
            "allow_resolve_locations":true,
            "attribution":"openrouteservice.org, OpenStreetMap contributors"
         },
         "isochrones":{
            "enabled":true,
            "maximum_range_distance":[
               {
                  "profiles":"any",
                  "value":50000
               },
               {
                  "profiles":"driving-hgv",
                  "value":10000000
               }
            ],
            "maximum_range_time":[
               {
                  "profiles":"any",
                  "value":18000
               },
               {
                  "profiles":"driving-hgv",
                  "value":360000
               }
            ],
            "maximum_intervals":100,
            "maximum_locations":50,
            "allow_compute_area":true,
            "attribution":"openrouteservice.org, OpenStreetMap contributors"
         },
         "routing":{
            "enabled":true,
            "mode":"normal",
            "sources":[
               "data/osm_file.pbf"
            ],
            "init_threads":2,
            "attribution":"openrouteservice.org, OpenStreetMap contributors",
            "distance_approximation":true,
            "profiles":{
               "active":[
                   "vehicles-hgv"
               ],
               "default_params": {
                   "encoder_flags_size": 8,
                   "graphs_root_path": "data/graphs",
                   "elevation_provider": "multi",
                       "elevation_cache_path": "data/elevation_cache",
                   "elevation_clear_cache": true,
                   "instructions": true,
                   "maximum_distance": 10000000,
                   "maximum_segment_distance_with_dynamic_weights": 10000000,
                   "maximum_distance_round_trip_routes": 10000000,
                   "maximum_waypoints": 1000
                   "maximum_avoid_polygon_area": 200000000,
                   "maximum_avoid_polygon_extent": 20000,
               },
               "profile-vehicles-hgv":{
                  "profiles":"driving-hgv",
                  "parameters":{
                     "encoder_options":"consider_elevation=true|turn_costs=true|block_fords=false|use_acceleration=true",
                     "elevation":true,
                     "preparation":{
                        "min_network_size":200,
                        "min_one_way_network_size":200,
                        "methods":{
                           "ch":{
                              "enabled":true,
                              "threads":8,
                              "weightings":"fastest"
                           },
                           "lm":{
                              "enabled":true,
                              "threads":8,
                              "weightings":"fastest,shortest",
                              "landmarks":24
                           }
                        }
                     },
                     "execution":{
                        "methods":{
                           "astar":{
                              "approximation":"BeelineSimplification",
                              "epsilon":1
                           },
                           "ch":{
                              "disabling_allowed":true
                           },
                           "lm":{
                              "disabling_allowed":true,
                              "active_landmarks":6
                           }
                        }
                     },
                     "ext_storages":{
                        "WayCategory":{

                        },
                        "HeavyVehicle":{
                           "restrictions":true
                        },
                        "WaySurfaceType":{

                        },
                        "Tollways":{

                        }
                     }
                  }
               }
            }
         }
      },
      "logging": {
          "enabled": true,
          "level_file": "PRODUCTION_LOGGING.json",
          "location": "/var/log/ors",
          "stdout": true
      },
      system_message: [
          {
              active: false,
              text: "This message would be sent with every routing bike fastest request",
              condition: {
                  "request_service": "routing",
                  "request_profile": "cycling-regular,cycling-mountain,cycling-road,cycling-electric",
                  "request_preference": "fastest"
              }
          },
          {
              active: false,
              text: "This message would be sent with every request for geojson response",
              condition: {
                  "api_format": "geojson"
              }
          },
          {
              active: false,
              text: "This message would be sent with every request on API v1 from January 2020 until June 2050",
              condition: {
                  "api_version": 1,
                  "time_after": "2020-01-01T00:00:00Z",
                  "time_before": "2050-06-01T00:00:00Z"
              }
          },
          {
              active: false,
              text: "This message would be sent with every request"
          }
      ]
   }
}

docker-compose.yml:

version: '3'
services:
  ors-app:
    container_name: ors-app
    ports:
    - 8080:8080
    build:
      context: ../
      args:
        APP_CONFIG: ./docker/conf/app.config
        OSM_FILE: ./docker/data/planet-latest.osm.pbf
    volumes:
      - ./graphs:/ors-core/data/graphs
      - ./elevation_cache:/ors-core/data/elevation_cache
      - ./logs/ors/:/var/log/ors/
      - ./logs/tomcat/:/usr/local/tomcat/logs
      - ./conf/app.config.sample:/ors-core/openrouteservice/target/classes/app.config
      - ./data/planet-latest.osm.pbf:/ors-core/data/osm_file.pbf
    environment:
      - BUILD_GRAPHS=False  # Forces the container to rebuild the graphs, e.g. when PBF is changed in app.config
      - JAVA_OPTS="-Djava.awt.headless=true -server -XX:TargetSurvivorRatio=75 -XX:SurvivorRatio=64 -XX:MaxTenuringThreshold=3 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4 -Xms64g -Xmx140g -XX:MaxMetaspaceSize=250m"
      - CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9001 -Dcom.sun.management.jmxremote.rmi.port=9001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost"

You might wanna try a new image from this branch:

In the old (actually current) image, changes in the app.config weren’t properly reflected in the container, so you’d always run off the one that’s provided in the image. Anyways, that’s probably not your problem here, just be aware of it.

And eventually the response you get after 3 mins is the correct one? Tbh, not sure how much we can help here, as this is sorta uncharted territory, basically on own risk. Still curious behaviour…

Thank you!
I am pretty sure that the correct app.config is used because I can now calculate routes with which I was running into duration/km limits before changing app.config. But I’ll try this version if it makes any difference.
The response after 3 minutes is correct, yes – it’s a 3 km route, but 1000 km routes do not take much longer. What is puzzling me is that I do not know what ORS is actually doing in these 3 minutes, as it’s nothing CPU intensive (what I would have guessed might happen when some parameter I modified is not a limit but something that affects every calculation) …

Thanks for reaching out!

I might have not been clear enough on this. Even though there are indeed 2 parameters which control the location index in GH: index.high_resolution and index.max_region_search, I don’t recall advising on adjusting the former one. You should rather increase the latter parameter to get a broader search coverage. The value index.high_resolution=250000 is way to high, no surprise you are experiencing such issues. In fact, your observation that the delay does not depend on the route length indicates a problem with the initial step of snapping the coordinates to the road network.

You could use the feature branch from PR #713 and try setting location_index_search_iterations=8 or even higher while keeping the default location_index_resolution. The resolution controls the size of the cells into which the graph is partitioned, while the search iterations sets the limit on the expansion of the search into neighboring cells. The expansion happens until a road segment is found, so increasing the limit will address cases when the queried location is in a remote area. Rather than getting an error message “Could not find point” the search will carry on.

Cheers,
Andrzej

Thank you very much for clarification! I misunderstood your advice regarding index.high_resolution, but with the most recent changes from PR #713 I set the parameters correctly (in addition to a very high maximum_snapping_radius) and am now able to calculate most of the routes I need with reasonable speed of calculation! Thank you very much for your help!