I am using a very large PBF extract, and starting openrouteservice. It starts and goes into a NOT READY state as it builds the graphs… Which is fine, but I see absolutely no status on where it is in that process.
It creates directories and puts gh.lock files in them (car and hgv)… but I never see any output being create there… so how do I get any sort of status on the state of the graph builds?
Is this normal?
Hi @stuffinpgh
there is currently no way of knowing the stage that the build process is in as the core library we use for that does not provide that information. For larger pbf files though, it can take quite some time, especially if you are using elevation and this is the first time you have run it.
The best way to see if there is a problem is to look at the logs that are produced. Also, depending on how you are deploying it you can attach a java monitor (e.g. VisualVM) and then using that you can see what is currently happening.
As a guide though, on our services when building a planet wide graph, we don’t see anything in the graphs folder for at least 24 hours, longer if we are redownloading elevation data.
Well currently it just runs for a long time, nothing is written to graphs and then eventually
/docker-entrypoint.sh: line 29: 7 Killed /usr/local/tomcat/bin/catalina.sh run
I am running in a 5gig Image with 5 gig heap.
I believe I have elevation turned off.
You need to offer more information. Large PBF, what’s that? docker-compose.yml, app.config and docker logs ors-app. If asking for (especially free of charge) support, just imagine it’d be you trying to help and ask yourself what information would be useful in order to help that person.
I am attempting to ingest the north-america pbf. It is obviously quite large. I really just need the hgv routing and matrtix for distance and route plotting, so there is a lot of extraneous stuff in this profile I am sure. I don’t care about elevation.
And not at this time interested in waypoints or landmarks… just point to point… I see that the java process is running, I could I suppose try to connect to the JMX and see more details, but I find it really odd nothing is getting written to disk, given how large this will be when done I would expect some output files to appear at some point during the processing.
the config file is as follows:
"ors": {
"info": {
"base_url": "https://openrouteservice.org/",
"support_mail": "support@openrouteservice.org",
"author_tag": "openrouteservice",
"content_licence": "LGPL 3.0"
},
"services": {
"matrix": {
"enabled": true,
"maximum_routes": 2500,
"maximum_routes_flexible": 25,
"maximum_search_radius": 5000,
"maximum_visited_nodes": 100000,
"allow_resolve_locations": true,
"attribution": "openrouteservice.org, OpenStreetMap contributors"
},
"isochrones": {
"enabled": true,
"maximum_range_distance": [
{ "profiles": "any", "value": 50000 },
{ "profiles": "driving-hgv", "value": 100000 }
],
"maximum_range_time": [
{ "profiles": "any", "value": 18000 },
{ "profiles": "driving-hgv", "value": 3600 }
],
"maximum_intervals": 10,
"maximum_locations": 2,
"allow_compute_area": true
},
"routing": {
"enabled": true,
"mode": "normal",
"routing_description": "This is a routing file from openrouteservice",
"routing_name": "openrouteservice routing",
"sources": ["data/osm_file.pbf"],
"init_threads": 2,
"attribution": "openrouteservice.org, OpenStreetMap contributors",
"elevation_preprocessed": false,
"profiles": {
"active": [
"hgv"
],
"default_params": {
"encoder_flags_size": 8,
"graphs_root_path": "data/graphs",
"elevation_provider": "multi",
"elevation_cache_path": "data/elevation_cache",
"elevation_cache_clear": false,
"instructions": true,
"maximum_distance": 100000,
"maximum_distance_dynamic_weights": 100000,
"maximum_distance_avoid_areas": 100000,
"maximum_waypoints": 50,
"maximum_snapping_radius": 400,
"maximum_avoid_polygon_area": 200000000,
"maximum_avoid_polygon_extent": 20000,
"maximum_distance_alternative_routes": 100000,
"maximum_alternative_routes": 3,
"maximum_distance_round_trip_routes": 100000,
"preparation": {
"min_network_size": 200,
"min_one_way_network_size": 200,
"methods": {
"lm": {
"enabled": true,
"threads": 1,
"weightings": "fastest,shortest",
"landmarks": 16
}
}
},
"execution": {
"methods": {
"lm": {
"disabling_allowed": true,
"active_landmarks": 8
}
}
}
},
"profile-car": {
"profiles": "driving-car",
"parameters": {
"encoder_flags_size": 8,
"encoder_options": "turn_costs=true|block_fords=false|use_acceleration=false",
"maximum_distance": 100000,
"elevation": false,
"maximum_snapping_radius": 350,
"preparation": {
"min_network_size": 200,
"min_one_way_network_size": 200,
"methods": {
"ch": {
"enabled": true,
"threads": 1,
"weightings": "fastest"
},
"lm": {
"enabled": false,
"threads": 1,
"weightings": "fastest,shortest",
"landmarks": 16
},
"core": {
"enabled": true,
"threads": 1,
"weightings": "fastest,shortest",
"landmarks": 64,
"lmsets": "highways;allow_all"
}
}
},
"execution": {
"methods": {
"ch": {
"disabling_allowed": true
},
"lm": {
"disabling_allowed": true,
"active_landmarks": 6
},
"core": {
"disabling_allowed": true,
"active_landmarks": 6
}
}
},
"ext_storages": {
"WayCategory": {},
"HeavyVehicle": {},
"WaySurfaceType": {},
"RoadAccessRestrictions": {
"use_for_warnings": true
}
}
}
},
"profile-hgv": {
"profiles": "driving-hgv",
"parameters": {
"encoder_flags_size": 8,
"encoder_options": "turn_costs=true|block_fords=false|use_acceleration=false",
"maximum_distance": 100000,
"elevation": false,
"preparation": {
"min_network_size": 200,
"min_one_way_network_size": 200,
"methods": {
"ch": {
"enabled": true,
"threads": 1,
"weightings": "fastest"
},
"lm": {
"enabled": true,
"threads": 1,
"weightings": "fastest,shortest",
"landmarks": 16
},
"core": {
"enabled": true,
"threads": 1,
"weightings": "fastest,shortest",
"landmarks": 64,
"lmsets": "highways;allow_all"
}
}
},
"execution": {
"methods": {
"ch": {
"disabling_allowed": true
},
"lm": {
"disabling_allowed": true,
"active_landmarks": 6
},
"core": {
"disabling_allowed": true,
"active_landmarks": 6
}
}.
},
"ext_storages": {
"WayCategory": {},
"HeavyVehicle": {
"restrictions": true
},
"WaySurfaceType": {}
}
}
},
"profile-bike-regular": {
"profiles": "cycling-regular",
"parameters": {
"encoder_options": "consider_elevation=true|turn_costs=true|block_fords=false",
"elevation": false,
"ext_storages": {
"WayCategory": {},
"WaySurfaceType": {},
"HillIndex": {},
"TrailDifficulty": {}
}
}
},
"profile-bike-mountain": {
"profiles": "cycling-mountain",
"parameters": {
"encoder_options": "consider_elevation=true|turn_costs=true|block_fords=false",
"elevation": false,
"ext_storages": {
"WayCategory": {},
"WaySurfaceType": {},
"HillIndex": {},
"TrailDifficulty": {}
}
}
},
"profile-bike-road": {
"profiles": "cycling-road",
"parameters": {
"encoder_options": "consider_elevation=true|turn_costs=true|block_fords=false",
"elevation": false,
"ext_storages": {
"WayCategory": {},
"WaySurfaceType": {},
"HillIndex": {},
"TrailDifficulty": {}
}
}
},
"profile-bike-electric": {
"profiles": "cycling-electric",
"parameters": {
"encoder_options": "consider_elevation=true|turn_costs=true|block_fords=false",
"elevation": false,
"ext_storages": {
"WayCategory": {},
"WaySurfaceType": {},
"HillIndex": {},
"TrailDifficulty": {}
}
}
},
"profile-walking": {
"profiles": "foot-walking",
"parameters": {
"encoder_options": "block_fords=false",
"elevation": false,
"ext_storages": {
"WayCategory": {},
"WaySurfaceType": {},
"HillIndex": {},
"TrailDifficulty": {}
}
}
},
"profile-hiking": {
"profiles": "foot-hiking",
"parameters": {
"encoder_options": "block_fords=false",
"elevation": false,
"ext_storages": {
"WayCategory": {},
"WaySurfaceType": {},
"HillIndex": {},
"TrailDifficulty": {}
}
}
},
"profile-wheelchair": {
"profiles": "wheelchair",
"parameters": {
"encoder_options": "block_fords=true",
"elevation": false,
"maximum_snapping_radius": 50,
"ext_storages": {
"WayCategory": {},
"WaySurfaceType": {},
"Wheelchair": {
"KerbsOnCrossings": "true"
},
"OsmId": {}
}
}
}
}
}
},
"logging": {
"enabled": true,
"level_file": "DEBUG_LOGGING.json",
"location": "/var/log/ors",
"stdout": true
},
"system_message": [
{
"active": false,
"text": "This message would be sent with every routing bike fastest request",
"condition": {
"request_service": "routing",
"request_profile": "cycling-regular,cycling-mountain,cycling-road,cycling-electric",
"request_preference": "fastest"
}
},
{
"active": false,
"text": "This message would be sent with every request for geojson response",
"condition": {
"api_format": "geojson"
}
},
{
"active": false,
"text": "This message would be sent with every request on API v1 from January 2020 until June 2050",
"condition": {
"api_version": 1,
"time_after": "2020-01-01T00:00:00Z",
"time_before": "2050-06-01T00:00:00Z"
}
},
{
"active": false,
"text": "This message would be sent with every request"
}
]
}
}
OK, 5g heap is no where near enough for North America - for that you would need at least around 20g (The pbf file alone is near 10g, and our guidelines are that the heap is twices the size of the pbf file).
Adam, Thank You, I did bump it up and did get output, location_index, landmarks_fastest, Landmars_shortest, subnetwork_landmarks fastest and shortests files, but is still in a “NOT READY” state, and hasn’t written anything out now for quite some time 12+ hours.
It looks like you are on the “core-alt” stage now then, so keep an eye out for any errors in the logs, but it should hopefully work. That stage can take a bit of time, but if it doesnt work you can try turning off lm or core (or even both if you are not going to be using restrictions) and then the build process will be faster
well 5 days later and still just sitting in NOT-READY… no errors… do see network traffic with DOCKER STATS… but nothing else… guess I will tryin go turn off LM and CORE and see if I get anywhere.
Hmm, yeh it shouldn’t be that long. Do you have the output of docker logs -n 200 ors-app?