Elevation API not returning same number of points as sent to it

abhiank · March 21, 2019, 8:51am

Hi folks at openRouteService,

I’m using the elevation api to query elevation along an entire path. The path which I’m querying has more than 2000 points so I’m breaking the main polyline into polylines of 2000 points each and hitting the API with those sub-polylines. However, the response encoded polyline does not contain 2000 points.

Consider this request (which has a encoded polyline containing 2000 points) :
→ POST https://api.openrouteservice.org/elevation/line
BODY -
https://textuploader.com/152rq

The Response encoded polyline however contains only 1980 points:
https://textuploader.com/152rm

Is this a bug or does SRTM not returning data for those 20 points?

Thanks.

Best,
Abhimanyu

nils · March 21, 2019, 9:07am

Hey,

good question, never tried the full limit… Do you have a plot of the LineString with background map?

To be easier for me to debug: can you pls just post those 20 points? Format doesn’t matter, though normal polyline would be best.

Thanks
Nils

abhiank · March 21, 2019, 9:10am

Hi @nils,

Do you just want the 20 points for which I didn’t receive data? It’ll take a little time to find the missing points. I’ll post it in a bit.

nils · March 21, 2019, 9:21am

Yes, please. That saves me the work, I’d appreciate it.

Thanks!

abhiank · March 21, 2019, 1:53pm

Hey @nils

After some thorough and dead-end hitting experiments I’m unable to figure this out. So, my first thought was to filter out received points (1980) from the original point list (2000), which should have given me 20 points. But I was getting some 336 points. Here’s where I found another issue. Many of the returned lat/long values had been rounded off and thats why latLng1 != latLng2. For eg. (latitude=12.9334, longitude=77.59906) was returned as (latitude=12.9334, longitude=77.59905). The last digit in longitude has changed and even though its not a major change, latLng1 == latLng2 logic was breaking. So thats another issue.

Coming back to the main issue, I hit the api with a polyline containing 50 points and received 48 points. Here are the requested and received polylines.

Requested Polyline (without elevation data)
ya}mAsyrxM?Cd@}ECMGCe@K??N}A`@qCV}B??kDa@aAK_AKs@CSAoBMq@KA?i@GaCYy@Ow@OgHaAOCK?KAa@A}@Gy@GOAu@IoAKKAa@EaAIe@Gs@G}@KQAa@Ek@KMCUEa@GYESEc@EI?e@C

Received Polyline (containing elevation data)
ya}mAsyrxMgwoD?C?d@}EvQCK?GE?e@K?N}AgE`@qCwQV}BfEkDa@wQaAKgE_AK?s@CfESA?oBM?q@KoKA??i@G?aCYgEy@OwQw@O?gHaAnd@OA?K??KC?a@AvQ}@G?y@G?OA?u@If^oAK?K??a@G?aAIvQe@G?s@G?}@InKQC?a@E?k@K?MC?UEgEa@G?YE?SE?c@E?I??e@CgE

And after looking closely, these two points were missing from the original polyline -
Point(latitude=12.93363, longitude=77.59914)
Point(latitude=12.93326, longitude=77.60097)

When I hit the elevation/point API, its working for these points, but somehow not being returned in the polyline.

Thats all I could figure out. I hope this info helps.

Thanks.

nils · March 21, 2019, 2:26pm

This is great info! Thanks so much for investigating closely! With that I should be able to find the bug fairly quickly. No promises when I get to it, but I’ll keep you posted.

nils · March 21, 2019, 7:26pm

FYI:

You can track progress there.

Can you tell me which library you used to decode the polyline? Encoding shouldn’t have any rounding issues as it’s basically just doing an integer out of your float coordinates by multiplying it with 10**5. Decoding however can have rounding problems, esp in Python, where rounding behaviour changed from v2.x to v3.x. So more likely it’s an issue with your decoder.

abhiank · March 22, 2019, 12:22am

Hey @nils,

Thank you. I’ll be following the issue.

I’m using the code I found on Graphhopper’s repo to decode the polyline with the elevation. I code with Kotlin/Java.

github.com

graphhopper/graphhopper/blob/469cf3c9634440e52821832c52280ca7e28322f4/web/src/main/java/com/graphhopper/http/WebHelper.java#L47


*/
public class WebHelper {
   public static String encodeURL(String str) {
       try {
           return URLEncoder.encode(str, "UTF-8");
       } catch (Exception _ignore) {
           return str;
       }
   }


   public static PointList decodePolyline(String encoded, int initCap, boolean is3D) {
       PointList poly = new PointList(initCap, is3D);
       int index = 0;
       int len = encoded.length();
       int lat = 0, lng = 0, ele = 0;
       while (index < len) {
           // latitude
           int b, shift = 0, result = 0;
           do {
               b = encoded.charAt(index++) - 63;
               result |= (b & 0x1f) << shift;

Its from an old commit though. Lemme try and use the code from their latest commit.

EDIT -
I tried the code from the latest commit. Its the same. Hasn’t changed.
Also, I tried setting the format_out for elevation/line to ‘geojson’ and it too is retuning only 1980 points in a list.

nils · March 22, 2019, 6:10am

Ah yes, that’s where I ported my implementation from too;) FYI, my question was more targeting the rounding issue.

I’ll look at this today actually, found some free time:)

abhiank · March 22, 2019, 6:14am

Actually I don’t understand how the encoding and decoding is working, so I didn’t really understand what you mentioned about the rounding. Is that something I should look into?

Oh thats awesome!

nils · March 22, 2019, 6:34am

In a nutshell:

https://developers.google.com/maps/documentation/utilities/polylinealgorithm

It could make a rounding error in encoding too though (thought it wouldn’t…), if you specify floats with a precision of > 5. So likely that’s where that error comes from. openelevationservice runs in Py3, which did some changes in rounding behaviour compared to Py2.

I think this is more on my side. I recently implemented a slightly different decoding lib into another project, which also allows you to encode precision > 5. Useful for openelevationservice too, so I might just add this today.

abhiank · March 22, 2019, 10:54am

This encoding algorithm is really genius stuff. I didn’t understand anything after the first step. I wonder how they came up with it.

Ah I see. So points sent with precision>5 are probably causing the issue.

Thanks for taking this up so soon and working on it.

nils · March 22, 2019, 11:04am

Yeah it is! Someday I might try to understand the bit shifting stuff as well;) I got the rounding issue sorted (cross-checked with other decoding libs):

Now, the precision shouldn’t matter anymore. I’ll also implement polyline6, so you can get precision=6 encoded. A fairly easy implementation of the decoding is here:

github.com

frederickjansen/polyline/blob/0764302d7b622a6bd10f3f05b8bc3c12815fb131/polyline/codec.py#L39


      
          
              while byte is None or byte >= 0x20:
                  byte = ord(value[index]) - 63
                  index += 1
                  result |= (byte & 0x1f) << shift
                  shift += 5
                  comp = result & 1
          
              return ~(result >> 1) if comp else (result >> 1), index
          
          def decode(self, expression, precision=5):
              coordinates, index, lat, lng, length, factor = [], 0, 0, 0, len(expression), float(10 ** precision)
          
              while index < length:
                  lat_change, index = self._trans(expression, index)
                  lng_change, index = self._trans(expression, index)
                  lat += lat_change
                  lng += lng_change
                  coordinates.append((lat / factor, lng / factor))
          
              return coordinates

Just got to your real issue:)

I didn’t touch this service in ages, so it’s good practice really! Should be all live on our servers tonight, the lastest early next week.

abhiank · March 22, 2019, 11:09am

Awesome!!

Does this also fix the missing points?

nils · March 22, 2019, 11:11am

Just got to it. But yes, that’ll be fixed for sure! Hope until tonight.

nils · March 22, 2019, 2:39pm

abhiank · March 22, 2019, 11:49pm

Two questions here -

In an ideal scenario, this wouldn’t really happen right? As in, a polyline returned by the direction api containing multiple consecutive points which are the same.
This only applies when location[n] == location[n-1] and not location[n] == location[x], right?

nils · March 23, 2019, 12:11pm

Oh, I see. Didn’t think you’d have done that to an ORS route. In fact, you can use elevation=true to get the elevation directly in the route response.

You’re right, it shouldn’t IMO. I first thought it might have smth to do with via points, i.e. more than 2 waypoints. I tried around a little, didn’t seem to be the case normally. However, I found a case where I observed the behavior loc[n] == loc[n-1]. Didn’t save the URL though (immediately tried another example…) and can’t reproduce anymore… What I saw was: when there’s via points, there can be the case of a middle leg that only routes from one point to the same point, i.e. duration & distance are zero for that leg. In that case it doubled the coordinate in the coordinate LineString. Hmpf, just can’t reproduce it…

Long story, short: @abhiank can you please share a (preferably small) example where you observed that behaviour? I.e. the query body/URL? Maybe the one with the 50 waypoints would be ideal.

Correct. Think I might just issue a warning in that case, instead of supporting things like that. After all, the endpoint is called line, where consecutive waypoints shouldn’t be the same inherently (I believe). Need the routing response to be fixed though…

abhiank · March 23, 2019, 1:13pm

Nope, I’m not. I’m indeed using the elevation = true flag. Its great to have the elevation data along with the points itself encoded in the same string.

The elevation requests I’m using to convert old routes with polylines received from Google’s API to be saved in the same format as the responses received from ORS with elevation data.
The polyline with 50 points which I’d shared in a previous reply and my original 2000 points are all polylines received from Google which seem to have consecutive points with the same value.

So, I’m not sure if that same behaviour/error exists in ORS directions api.

A question here - is it a loss or incorrect to simply send back the same points as received in the polyline, regardless of whether they have same consecutive points or not, in the elevation api?

Also, didnt understand -

doubled the coordinate in the coordinate LineString

A polyline could be encoded with random points too, right? (I’m not aware if that technically possible). In case it is, it wouldn’t be a line and would be just a collection of points needing elevation data…

nils · March 23, 2019, 3:07pm

Ok, got ya.

Yes, it did, but like I said, didn’t document the exact query and was unable to reproduce. Not a biggie though generally. Seems to be very rare.
Right. The encoding is not spatially aware, you could throw any integers at it.

See my latest commit, I fixed the behavior returning less coordinates. Now, I wanna include polyline6 as well. And currently struggling with travis, which can take an indeterminable amount of time;) Should be online early next week still I hope.