POI import error message

openpoiservice
1

Hello there,

I’m trying to set up an own instance of OpenPOIService as a docker container following the instructions given in the manual. The only thing I change is to write “concurrent_workers: 5” at the beginning of the ops_settings.yml file.

I am running everything on a Linux PhotonOS virtual machine and using the “kartoza/postgis:latest” image for the database.

The process fails after executing the following command as root:

docker exec -it poiservice /ops_venv/bin/python manage.py import-data

getting this message:

Found directory: /deploy/app/osm
INFO:main:Starting to import OSM data…[’/deploy/app/osm/bremen-tests.osm.pbf’]
INFO:openpoiservice.server.db_import.parser:Starting to read /deploy/app/osm/bremen-tests.osm.pbf
INFO:openpoiservice.server.db_import.parser:Parsing and importing nodes…
INFO:openpoiservice.server.db_import.parser:Parsing relations…
INFO:openpoiservice.server.db_import.parser:Found 1 ways in relations
INFO:openpoiservice.server.db_import.parser:Parsing ways…
INFO:openpoiservice.server.db_import.parser:Found 11 ways
INFO:openpoiservice.server.db_import.parser:Importing ways… (note this wont work concurrently)
INFO:openpoiservice.server.db_import.parser:Storing remaining pois
Traceback (most recent call last):
File “manage.py”, line 60, in
cli()
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/flask/cli.py”, line 380, in main
return AppGroup.main(self, *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/flask/cli.py”, line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “manage.py”, line 56, in import_data
parser.run_import(osm_files)
File “/deploy/app/openpoiservice/server/utils/decorators.py”, line 105, in timed
result = method(*args, **kw)
File “/deploy/app/openpoiservice/server/db_import/parser.py”, line 76, in run_import
parse_file(osm_file)
File “/deploy/app/openpoiservice/server/utils/decorators.py”, line 51, in wrapper
raise ex_type(message)
TypeError: function takes exactly 5 arguments (1 given)

Any idea what I could do?

Thanks in advance,
Jakob

2

Greetings @jakob thank you for your message - are the tables created in the postgis db?

3

Hi timothy,

yes they are created.

4

Does this mean it works with 4? Or is the same error raised?

5

I get the same error message. I also tryed: 0, 1, 2, 3, 4, 5, 10 with no different outcome.

6

It seems that it can’t write to the database, are you able to psql into the database from the poiservice?

7

psql was not installed on the poiservice container so I installed it with:

apt-get install postgresql-client

inside the container.

After doing that I was able to connect to the Database.
I also again tried to import data, but it raised the same error.

8

Weird, I can’t figure out much more from the error messages. Can you please comment line 66 (@processify) in parser.py and try again?

9

now I get the following error:

Found directory: /deploy/app/osm
INFO:main:Starting to import OSM data…[’/deploy/app/osm/bremen-tests.osm.pbf’]
INFO:openpoiservice.server.db_import.parser:Starting to read /deploy/app/osm/bremen-tests.osm.pbf
INFO:openpoiservice.server.db_import.parser:Parsing and importing nodes…
INFO:openpoiservice.server.db_import.parser:Parsing relations…
INFO:openpoiservice.server.db_import.parser:Found 1 ways in relations
INFO:openpoiservice.server.db_import.parser:Parsing ways…
INFO:openpoiservice.server.db_import.parser:Found 11 ways
INFO:openpoiservice.server.db_import.parser:Importing ways… (note this wont work concurrently)
INFO:openpoiservice.server.db_import.parser:Storing remaining pois
Traceback (most recent call last):
File “manage.py”, line 60, in
cli()
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/flask/cli.py”, line 380, in main
return AppGroup.main(self, *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/flask/cli.py”, line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “manage.py”, line 56, in import_data
parser.run_import(osm_files)
File “/deploy/app/openpoiservice/server/utils/decorators.py”, line 105, in timed
result = method(*args, **kw)
File “/deploy/app/openpoiservice/server/db_import/parser.py”, line 76, in run_import
parse_file(osm_file)
File “/deploy/app/openpoiservice/server/db_import/parser.py”, line 68, in parse_file
parse_import(osm_file)
File “/deploy/app/openpoiservice/server/db_import/parser.py”, line 54, in parse_import
osm_importer.save_remainder()
File “/deploy/app/openpoiservice/server/db_import/parse_osm.py”, line 481, in save_remainder
db.session.bulk_save_objects(self.tags_objects)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/scoping.py”, line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2692, in bulk_save_objects
False,
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2880, in _bulk_save_mappings
transaction.rollback(_capture_exception=True)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py”, line 68, in exit
compat.reraise(exc_type, exc_value, exc_tb)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 129, in reraise
raise value
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2874, in _bulk_save_mappings
render_nulls,
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py”, line 102, in _bulk_insert
bookkeeping=return_defaults,
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py”, line 1066, in _emit_insert_statements
c = cached_connections[connection].execute(statement, multiparams)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 988, in execute
return meth(self, multiparams, params)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py”, line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1107, in _execute_clauseelement
distilled_params,
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1248, in _execute_context
e, statement, parameters, cursor, context
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1468, in _handle_dbapi_exception
util.reraise(*exc_info)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 129, in reraise
raise value
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1224, in _execute_context
cursor, statement, parameters, context
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py”, line 752, in do_executemany
cursor.executemany(statement, parameters)
UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xfc’ in position 10: ordinal not in range(128)

10

Can you please show me the output of \d within psql ?

11

poi_db-# \d
List of relations
Schema | Name | Type | Owner
----------±----------------------------±---------±---------
public | geography_columns | view | postgres
public | geometry_columns | view | postgres
public | pois | table | docker
public | pois_categories | table | docker
public | pois_categories_id_seq | sequence | docker
public | pois_tags | table | docker
public | pois_tags_id_seq | sequence | docker
public | pois_test | table | docker
public | pois_test_categories | table | docker
public | pois_test_categories_id_seq | sequence | docker
public | pois_test_tags | table | docker
public | pois_test_tags_id_seq | sequence | docker
public | raster_columns | view | postgres
public | raster_overviews | view | postgres
public | spatial_ref_sys | table | postgres
topology | layer | table | postgres
topology | topology | table | postgres
topology | topology_id_seq | sequence | postgres
(18 rows)

poi_db-#

ps: the username is “docker”

12

I have the same error when I try to import data. I used default configurations, without changes.
I´m following the instructions given in the manual, with docker containers, all-in-one image.
I can connect to DB inside docker… But i can´t import osm files… always get exactly the same error, like Jakob.
I’m running everything in Ubuntu 18.04 server… I’m trying the same installation in a physical server and in Hauwei Cloud… The same result in both scenarios.

DB connection:
gis=# \d
List of relations
Schema | Name | Type | Owner
----------±------------------±---------±---------
public | geography_columns | view | postgres
public | geometry_columns | view | postgres
public | raster_columns | view | postgres
public | raster_overviews | view | postgres
public | spatial_ref_sys | table | postgres
topology | layer | table | postgres
topology | topology | table | postgres
topology | topology_id_seq | sequence | postgres
(8 rows)

gis=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------±----------±---------±--------±--------±----------------------
gis | gis_admin | UTF8 | C.UTF-8 | C.UTF-8 |
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)

gis=# \dt
List of relations
Schema | Name | Type | Owner
----------±----------------±------±---------
public | spatial_ref_sys | table | postgres
topology | layer | table | postgres
topology | topology | table | postgres
(3 rows)

Thanks in advance,

Luis.-

13

I found the problem…
The database port in ops_settings.yml was 5433… I changed that port with 5432 and the creating DB process works fine and then the import process its running now…

Luis.-

14

Awesome, happy to hear that Luis! And thanks for providing the solution!