JSONDecodeError when self-hosting ORS and calling from QGIS


#1

Hi there,

I’m exploring using the ORS plugin in QGIS. I’ve replaced the sample dataset with a dataset for India, and adapted the docker-compose.yml accordingly, and managed to host the service on my Google Cloud. The service is up when I check the status at /ors/health.

The dataset is from geofabrik.de
The file format is .osm.pbf

In QGIS, I’ve also added a new localhost in the ORS Tool->Provider setting, using url https://8080-dot-10511203-dot-devshell.appspot.com/ors

then I was trying to generate an ishochrone layer using the localhost. However, the run failed and the error log shows:

Since it is a JSONDecodeError, I’m wondering whether the root cause is the dataset? Or I’m somehow hosting wrongly.

Thank you!


#2

The ORS Tool panel shows
image

The python error shown in the panel is
2020-01-16T10:59:59 WARNING Traceback (most recent call last):
File “C:/Users/O48260/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\ORStools\common\networkaccessmanager.py”, line 359, in replyFinished
self.reply.url().toString()))
AttributeError: ‘NoneType’ object has no attribute ‘url’


#3

Looks like the API is still using the sample dataset (heidelberg.osm.gz) instead of the new dataset (india-latest.osm.pbf). Testing calling route using coordinate 8.676581,49.418204|8.692803,49.409465 (in heidelberg) is good.

But using coordinate 76.3495,10.1099|76.3496,10.1098 (in India) is not working:
{“error”:{“code”:2010,“message”:“Could not find point 0: 76.3495000 10.1099000 within a radius of 350.0 meters.”},“info”:{“engine”:{“version”:“6.0.0”,“build_date”:“2020-01-16T05:39:40Z”},“timestamp”:1579163562788}}

This is strange as I’ve change the param value in the yml file:
image

Not sure why when building the app, the india dataset is not used


#4

Hi,
yes, that’s what I would’ve thought as well. However, it shouldn’t throw a JSONDecode error. Weird…

You need to delete the graphs folder if you ran the Docker container before with the sample dataset. Everytime you change the PBF you need to delete it in fact.

I also didn’t like our Docker setup too much, so I changed it:

However, it’s not in master yet. You can just copy the whole Docker folder from development to master, should work. The app.config.sample is not up-to-date AT ALL in the Docker folder, sorry for that. Doesn’t hurt you too much, I think, but you might consider replacing it with openrouteservice/src/main/files/app.config.sample. I’ll raise an issue on Github.


#5

Hi Nils,

Thanks for the quick reply and help. Ya now I can see there are some significant differences in app.config.sample and docker-compose.yml.

I think in the instruction you missed one more point, namely one more adaption to the new dataset in the app.config.sample file:


where we need to repalce the heidelberg.osm.gz with the new dataset.

I’m testing it out today and will update the result in this post. Thanks for the help again!


#6

Oh yeah, sorry, I should’ve elaborated a little more. That’s right, you’ll have to adapt that, and maybe a few other things. Best is take both app.config side-by-side and watch out for any paths, they should follow the pattern of the old Docker app.config.sample.

Let me know how it went.


#7

I take another look at the app.config.sample file, and now a little confused now.

it looks like the openrouteservice/src/main/resources/app.config.sample file is using source data in directory openrouteservice/src/main/files/

And so far I’m depositing the new dataset in docker/data. The old app.config.sample in the Docker folder somehow seems to be working fine, as the source related code from line 47 to 49 is
“sources”:[
“data/osm_file.pbf”
],

So do I need to replace the old app.config.sample file?

Right now the changes I did is:

  1. Place the new dataset in docker/data
  2. used the yml file in the dev branch, and made two adaptions related to the new dataset, one for OSM_FILE under build and another for /ors-core/data/osm_file.pbf under volumes. Namely:

build:
context: …/
args:
APP_CONFIG: ./docker/conf/app.config.sample
OSM_FILE: ./docker/data/india-latest.osm.pbf
volumes:
- ./graphs:/ors-core/data/graphs
- ./elevation_cache:/ors-core/data/elevation_cache
- ./logs/ors/:/var/log/ors/
- ./logs/tomcat/:/usr/local/tomcat/logs
- ./conf/app.config.sample:/ors-core/openrouteservice/target/classes/app.config
- ./data/india-latest.osm.pbf:/ors-core/data/osm_file.pbf

I have not changed the old app.config.sample file yet.

I’m deploying the app now, and so far it is showing not ready for 30 mins. Could I ask how long usually it would take for the app to come alive, for a 640MB dataset, based on your experience?


#8

So I did some more debugging today. There is an additional adaption I did in Dockerfile in the home directory.
ARG OSM_FILE=docker/data/india-latest.osm.pbf
so that when setting up the ors-app, docker will be able to copy the correct dataset to
COPY $OSM_FILE /ors-core/data/osm_file.pbf

However, the ors/health still keep showing “not ready”. After checking the Tomcat deployment log, I realized the deployment hanging is due to
Caused by: java.lang.IllegalArgumentException: Input file is not of valid type data/osm_file.pbf

So I think something is wrong when ors-app is setting up and trying to parse the india-latest.osm.pbf file, which is downloaded from geofabrik.de.

May I ask whether anyone encountered similar exceptions?


#9

So far my changes are:

  1. Two dataset name changes in docker-compose.yml
  2. One dataset name change in Dockerfile in root directory
  3. Replace the app.config.sample in docker folder with

and

image

Testing results:

  1. Perform step 3 only. Using the default sample heidelberg.osm.gz file. Result is good. The health page shows that the API service is up
  2. Perform all the 3 steps and replace the dataset in docker/data
    I’ve tried several different pbf files, while the deployments for any of them is encountering the same error msg: Input file is not of valid type data/osm_file.pbf
    The error always occurs at the same step:
    INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/examples

Then I did a throughout search of the keyword heidelberg.osm.gz. There are still 3 places:

So I’m wondering, do I need to change the dataset name in openrouteservice\src\main\resources\app.config.sample as well? What is this config file’s role during the deployment, as I thought only the app.config.sample file will be used?

Thank you and hope someone can share some ideas.


#10

I adapted this app.config.sample, and put the new dataset Tokyo.osm.gz (70+MB) both in
openrouteservice/src/main/files/
and
openrouteservice/docker/data

image
so far the deployment has been going on for one hour, and has not encountered any java exception yet.

*Edited.
OutofMemoryError hit after 60 mins:
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space - problem when allocating new memory. Old capacity: 7340032, new bytes:8, segmentSizeIntsPower:18, new segments:1, existing:7
Now I’m going to other posts for this issue…:worried:

**Edited.
By selecting “walking” as the only active profile, I finally manage to run the API using Tokyo.osm.gz.
(Google Cloud free trial version seems to provide very limited memory, only 409 MB in my case)
Requesting to Route is good:
coordinates=139.736993,35.590514|139.737032,35.590534 (within Tokyo)

Current Issue I’m running into the same old “raise JSONDecodeError(“Expecting value”, s, err.value) from None” error again when building Isochrones layer in QGIS…:persevere:

man this is the recursion of errors…

The funny thing is that I tried using Tokyo.osm.pbf (30MB) first, and it hit the

again, despite the fact that both Tokyo.osm.pbf and Tokyo.osm.gz were downloaded from https://download.bbbike.org/osm/bbbike/
Given the previous experiences of hitting the ‘valid type’ exception when using the india-latest.osm.pbf, I’m not sure what’s wrong with the pbf format file


#11

Right now I’m tracing the python error in python/plugins\ORStools\common\networkaccessmanager.py


image
So basically the JSON error is due to nothing is returned from the API

I’m thinking of two possible reasons

  1. APP is still not running completely correctly. Maybe I need to change this section in the app.config.sample file
    image
  2. Some set up related to url is incorrect in QGIS? :persevere:
    image

Going to update here if any finding.
@nils May I ask whether you or your colleagues have ever encountered similar error? Thank you


#12

Hey, sorry for the late reply…

It’s a lot of info here, hard for me to know the full current state and give recommendations. Generally, I didn’t have problems so far with localhost and QGIS plugins. However, I didn’t try it with the latest ORS version yet (6.0.x). Can’t really imagine any regressions though.

You’re sure you have SSL activated on that domain?

Can you paste (in properly Markdown format):

  • full current app.config
  • the output of https://.../ors/v2/status
  • tree -L 2 ./graphs

Still didn’t have any time for this, I’ll try the whole workflow before the weekend though and let you know.


#13

Hi Nils,

Sorry for the very late reply… Just got back from China and shifted my focus back to the geo project.

Your point on the SSL inspired me, as I realized the previous error should be due to the local QGIS not able to calling the ORSTools app hosted on Google Cloud.
So I installed docker on my personal Mac, deployed the customized ORS app successfully, and starting to create Isochrone layer in QGIS. Initially it goes well, as I could see the points being processed. But at certain point, an exception occurred:

Have your team encountered similar RuntimeError before? Usually what would be the cause?

Thank you!