POI import error message

Hello there,

I’m trying to set up an own instance of OpenPOIService as a docker container following the instructions given in the manual. The only thing I change is to write “concurrent_workers: 5” at the beginning of the ops_settings.yml file.

I am running everything on a Linux PhotonOS virtual machine and using the “kartoza/postgis:latest” image for the database.

The process fails after executing the following command as root:

docker exec -it poiservice /ops_venv/bin/python manage.py import-data

getting this message:


Found directory: /deploy/app/osm
INFO:main:Starting to import OSM data…[’/deploy/app/osm/bremen-tests.osm.pbf’]
INFO:openpoiservice.server.db_import.parser:Starting to read /deploy/app/osm/bremen-tests.osm.pbf
INFO:openpoiservice.server.db_import.parser:Parsing and importing nodes…
INFO:openpoiservice.server.db_import.parser:Parsing relations…
INFO:openpoiservice.server.db_import.parser:Found 1 ways in relations
INFO:openpoiservice.server.db_import.parser:Parsing ways…
INFO:openpoiservice.server.db_import.parser:Found 11 ways
INFO:openpoiservice.server.db_import.parser:Importing ways… (note this wont work concurrently)
INFO:openpoiservice.server.db_import.parser:Storing remaining pois
Traceback (most recent call last):
File “manage.py”, line 60, in
cli()
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/flask/cli.py”, line 380, in main
return AppGroup.main(self, *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/flask/cli.py”, line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “manage.py”, line 56, in import_data
parser.run_import(osm_files)
File “/deploy/app/openpoiservice/server/utils/decorators.py”, line 105, in timed
result = method(*args, **kw)
File “/deploy/app/openpoiservice/server/db_import/parser.py”, line 76, in run_import
parse_file(osm_file)
File “/deploy/app/openpoiservice/server/utils/decorators.py”, line 51, in wrapper
raise ex_type(message)
TypeError: function takes exactly 5 arguments (1 given)


Any idea what I could do?

Thanks in advance,
Jakob

Greetings @jakob thank you for your message - are the tables created in the postgis db?

Hi timothy,

yes they are created.

Does this mean it works with 4? Or is the same error raised?

I get the same error message. I also tryed: 0, 1, 2, 3, 4, 5, 10 with no different outcome.

It seems that it can’t write to the database, are you able to psql into the database from the poiservice?

psql was not installed on the poiservice container so I installed it with:

apt-get install postgresql-client

inside the container.

After doing that I was able to connect to the Database.
I also again tried to import data, but it raised the same error.

Weird, I can’t figure out much more from the error messages. Can you please comment line 66 (@processify) in parser.py and try again?

now I get the following error:


Found directory: /deploy/app/osm
INFO:main:Starting to import OSM data…[’/deploy/app/osm/bremen-tests.osm.pbf’]
INFO:openpoiservice.server.db_import.parser:Starting to read /deploy/app/osm/bremen-tests.osm.pbf
INFO:openpoiservice.server.db_import.parser:Parsing and importing nodes…
INFO:openpoiservice.server.db_import.parser:Parsing relations…
INFO:openpoiservice.server.db_import.parser:Found 1 ways in relations
INFO:openpoiservice.server.db_import.parser:Parsing ways…
INFO:openpoiservice.server.db_import.parser:Found 11 ways
INFO:openpoiservice.server.db_import.parser:Importing ways… (note this wont work concurrently)
INFO:openpoiservice.server.db_import.parser:Storing remaining pois
Traceback (most recent call last):
File “manage.py”, line 60, in
cli()
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/flask/cli.py”, line 380, in main
return AppGroup.main(self, *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/flask/cli.py”, line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/click/core.py”, line 555, in invoke
return callback(*args, **kwargs)
File “manage.py”, line 56, in import_data
parser.run_import(osm_files)
File “/deploy/app/openpoiservice/server/utils/decorators.py”, line 105, in timed
result = method(*args, **kw)
File “/deploy/app/openpoiservice/server/db_import/parser.py”, line 76, in run_import
parse_file(osm_file)
File “/deploy/app/openpoiservice/server/db_import/parser.py”, line 68, in parse_file
parse_import(osm_file)
File “/deploy/app/openpoiservice/server/db_import/parser.py”, line 54, in parse_import
osm_importer.save_remainder()
File “/deploy/app/openpoiservice/server/db_import/parse_osm.py”, line 481, in save_remainder
db.session.bulk_save_objects(self.tags_objects)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/scoping.py”, line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2692, in bulk_save_objects
False,
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2880, in _bulk_save_mappings
transaction.rollback(_capture_exception=True)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py”, line 68, in exit
compat.reraise(exc_type, exc_value, exc_tb)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 129, in reraise
raise value
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 2874, in _bulk_save_mappings
render_nulls,
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py”, line 102, in _bulk_insert
bookkeeping=return_defaults,
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py”, line 1066, in _emit_insert_statements
c = cached_connections[connection].execute(statement, multiparams)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 988, in execute
return meth(self, multiparams, params)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py”, line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1107, in _execute_clauseelement
distilled_params,
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1248, in _execute_context
e, statement, parameters, cursor, context
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1468, in _handle_dbapi_exception
util.reraise(*exc_info)
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 129, in reraise
raise value
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1224, in _execute_context
cursor, statement, parameters, context
File “/ops_venv/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py”, line 752, in do_executemany
cursor.executemany(statement, parameters)
UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xfc’ in position 10: ordinal not in range(128)


Can you please show me the output of \d within psql ?


poi_db-# \d
List of relations
Schema | Name | Type | Owner
----------±----------------------------±---------±---------
public | geography_columns | view | postgres
public | geometry_columns | view | postgres
public | pois | table | docker
public | pois_categories | table | docker
public | pois_categories_id_seq | sequence | docker
public | pois_tags | table | docker
public | pois_tags_id_seq | sequence | docker
public | pois_test | table | docker
public | pois_test_categories | table | docker
public | pois_test_categories_id_seq | sequence | docker
public | pois_test_tags | table | docker
public | pois_test_tags_id_seq | sequence | docker
public | raster_columns | view | postgres
public | raster_overviews | view | postgres
public | spatial_ref_sys | table | postgres
topology | layer | table | postgres
topology | topology | table | postgres
topology | topology_id_seq | sequence | postgres
(18 rows)

poi_db-#


ps: the username is “docker”

I have the same error when I try to import data. I used default configurations, without changes.
I´m following the instructions given in the manual, with docker containers, all-in-one image.
I can connect to DB inside docker… But i can´t import osm files… always get exactly the same error, like Jakob.
I’m running everything in Ubuntu 18.04 server… I’m trying the same installation in a physical server and in Hauwei Cloud… The same result in both scenarios.

DB connection:
gis=# \d
List of relations
Schema | Name | Type | Owner
----------±------------------±---------±---------
public | geography_columns | view | postgres
public | geometry_columns | view | postgres
public | raster_columns | view | postgres
public | raster_overviews | view | postgres
public | spatial_ref_sys | table | postgres
topology | layer | table | postgres
topology | topology | table | postgres
topology | topology_id_seq | sequence | postgres
(8 rows)

gis=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------±----------±---------±--------±--------±----------------------
gis | gis_admin | UTF8 | C.UTF-8 | C.UTF-8 |
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)

gis=# \dt
List of relations
Schema | Name | Type | Owner
----------±----------------±------±---------
public | spatial_ref_sys | table | postgres
topology | layer | table | postgres
topology | topology | table | postgres
(3 rows)

Thanks in advance,

Luis.-

I found the problem…
The database port in ops_settings.yml was 5433… I changed that port with 5432 and the creating DB process works fine and then the import process its running now…

Luis.-

Awesome, happy to hear that Luis! And thanks for providing the solution!