Unable to create DB or import data

kshnkvn · 2020-02-08 00:54:45 UTC

kshnkvn@kshnkvn-vb:~/openpoiservice$ docker exec -it 9fa3b8a85748 /ops_venv/bin/python manage.py create-db
Traceback (most recent call last):
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect
    return fn()
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
    return _ConnectionFairy._checkout(self)
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
    connection = pool._invoke_creator(self)
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/ops_venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 489, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/ops_venv/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Cannot assign requested address
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?

docker logs for db instance:

Postgres initialisation process completed .... restarting in foreground
2020-02-08 00:45:45.369 UTC [184] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2020-02-08 00:45:45.370 UTC [184] LOG:  listening on IPv6 address "::", port 5432
2020-02-08 00:45:45.411 UTC [184] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-02-08 00:45:45.510 UTC [186] LOG:  database system was shut down at 2020-02-08 00:45:44 UTC
2020-02-08 00:45:45.526 UTC [184] LOG:  database system is ready to accept connections

I also tried to import data, but also unsuccessfully:

kshnkvn@kshnkvn-vb:~/openpoiservice$ docker exec -it 9fa3b8a85748 /ops_venv/bin/python manage.py import-data
Found directory: /deploy/app/osm
INFO:__main__:Starting to import OSM data...['/deploy/app/osm/bremen-latest.osm.pbf', '/deploy/app/osm/bremen-tests.osm.pbf']
INFO:openpoiservice.server.db_import.parser:Starting to read /deploy/app/osm/bremen-latest.osm.pbf
INFO:openpoiservice.server.db_import.parser:Parsing and importing nodes...
INFO:openpoiservice.server.db_import.parse_osm:Pois: 1000, tags: 1378, categories: 1023
Traceback (most recent call last):
  File "manage.py", line 60, in <module>
    cli()
  File "/ops_venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/ops_venv/lib/python3.6/site-packages/flask/cli.py", line 380, in main
    return AppGroup.main(self, *args, **kwargs)
  File "/ops_venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/ops_venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/ops_venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/ops_venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/ops_venv/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/ops_venv/lib/python3.6/site-packages/flask/cli.py", line 257, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/ops_venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "manage.py", line 56, in import_data
    parser.run_import(osm_files)
  File "/deploy/app/openpoiservice/server/utils/decorators.py", line 105, in timed
    result = method(*args, **kw)
  File "/deploy/app/openpoiservice/server/db_import/parser.py", line 76, in run_import
    parse_file(osm_file)
  File "/deploy/app/openpoiservice/server/utils/decorators.py", line 51, in wrapper
    raise ex_type(message)
TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig'

Any thoughts what I could try to do with this?

nils · 2020-02-09 03:34:39 UTC

There is a very similar issue:

Can you post there pls? And include

docker-compose.yml with the image for the DB
ops_settings_docker.yml
indicate whether you ping the DB container from the flask container successfully
logs for both containers (flask & DB)

In case you used the default docker-compose.yml from the Github repo, did you respect this?

 # Don't forget to change the host name inside ops_settings_docker.yml by the one given to docker container.
 # Also port should be set to 5432 (default value) inside the same file since they are on the same network

In case you’re working with a DB installed on the host machine, you’ll need to uncomment the #network_mode: "host" line.

kshnkvn · 2020-02-11 13:53:32 UTC

Sorry, I started the service, the error was in the configuration file, I did not correctly indicate the ports.
But it seems that the import of files does not occur, it is stuck in one place:

ivan@geo-api:~/openpoiservice$ docker exec -it d60b62812d7a /ops_venv/bin/python manage.py import-data
Found directory: /deploy/app/osm
Found directory: /deploy/app/osm/north-america-latest.osm-gh
INFO:__main__:Starting to import OSM data...['/deploy/app/osm/north-america-latest.osm.pbf']
INFO:openpoiservice.server.db_import.parser:Starting to read /deploy/app/osm/north-america-latest.osm.pbf
INFO:openpoiservice.server.db_import.parser:Parsing and importing nodes...
INFO:openpoiservice.server.db_import.parse_osm:Pois: 1000, tags: 963, categories: 1074

I tried different files, even very small ones, but nothing happens.

nils · 2020-02-12 01:40:16 UTC

Ok, good to hear you got a little further.

I’ll try it myself one of these days. Didn’t do that in a looong time and the creator of the project left the group a while ago. Could take a bit though, currently on “vacation” for another week.

nils · 2020-02-12 06:59:11 UTC

So I was interested while lying on the beach and tried it out again: it works fine for me with Geofabrik’s Andorra PBF and the current master, with Kartoza’s PostGIS Docker container running independently on port 5432 and the following docker-compose.yml:

version: '2.2'
services:
 gunicorn_flask:
  network_mode: "host"
  build: .
  volumes:
  - ./osm:/deploy/app/osm
  - ./ops_settings_docker.yml:/deploy/app/openpoiservice/server/ops_settings.yml
  - ./categories_docker.yml:/deploy/app/openpoiservice/server/categories/categories.yml
  ports:
   - "5000:5000"
  mem_limit: 2g

openpoiservice showed the follwoing logs:

docker exec -it openpoiservice_gunicorn_flask_1 /ops_venv/bin/python manage.py import-data
Found directory: /deploy/app/osm
INFO:__main__:Starting to import OSM data...['/deploy/app/osm/andorra-latest.osm.pbf']
INFO:openpoiservice.server.db_import.parser:Starting to read /deploy/app/osm/andorra-latest.osm.pbf
INFO:openpoiservice.server.db_import.parser:Parsing and importing nodes...
INFO:openpoiservice.server.db_import.parse_osm:Pois: 1000, tags: 884, categories: 1006
INFO:openpoiservice.server.db_import.parser:Parsing relations...
INFO:openpoiservice.server.db_import.parser:Found 11 ways in relations
INFO:openpoiservice.server.db_import.parser:Parsing ways...
INFO:openpoiservice.server.db_import.parser:Found 784 ways
INFO:openpoiservice.server.db_import.parser:Importing ways... (note this wont work concurrently)
INFO:openpoiservice.server.db_import.parse_osm:Pois: 2000, tags: 1635, categories: 2010
INFO:openpoiservice.server.db_import.parser:Storing remaining pois
INFO:openpoiservice.server.db_import.parser:Found 2385 pois
INFO:openpoiservice.server.db_import.parser:Finished import of /deploy/app/osm/andorra-latest.osm.pbf
INFO:openpoiservice.server.utils.decorators:'run_import'  3795.29 ms