Docker error on creating graphs of big pdb

rki · December 13, 2019, 3:08pm

Hello,
trying to create docker as described failed.

It is a Server with 8 kernel and 300 GB ram. pdb is europe with 22 Gig. After three hours of calculation the graph-files an error accours.

Here is the error log

Here our docker-compose file:
version: ‘3’
services:
ors-app:
container_name: ors-app
ports:
- 8080:8080
build:
context: …/
args:
APP_CONFIG: ./docker/conf/app.config.sample
OSM_FILE: ./docker/data/europe-latest.osm.pbf
JAVA_OPTS: -Djava.awt.headless=true -server -XX:TargetSurvivorRatio=75 -XX:SurvivorRatio=64 -XX:MaxTenuringThreshold=3 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4 -Xms220g -Xmx220g
CATALINA_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9001 -Dcom.sun.management.jmxremote.rmi.port=9001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost
volumes:
- ./graphs:/ors-core/data/graphs
- ./elevation_cache:/ors-core/data/elevation_cache
- ./logs/ors/:/var/log/ors/
- ./logs/tomcat/:/usr/local/tomcat/logs

Can anyone help?

nils · December 13, 2019, 3:27pm

jeez 300 GB RAM?!

Can you pls paste the full stack trace in text and format it properly like this

container_name: ors-app
ports:
- 8080:8080
build:
context: …/

It’s just markdown here.

rki · December 16, 2019, 1:03pm

Her are the logs:

ors-log

2019-12-16 12:52:19,976 ERROR [routing.RoutingProfileManager] - java.util.concurrent.ExecutionException: 
java.lang.RuntimeException: Couldn't process file data/osm_file.pbf, error: 
java.net.SocketTimeoutException: Read timed out

tomcat-log

16-Dec-2019 09:46:51.887 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log 2 
Spring WebApplicationInitializers detected on classpath
16-Dec-2019 09:47:03.508 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log 
Initializing Spring embedded WebApplicationContext
16-Dec-2019 09:47:09.418 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log 
ContextListener: contextInitialized()
16-Dec-2019 09:47:09.418 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log 
SessionListener: contextInitialized()

shell-output

Attaching to ors-app
ors-app    | 16-Dec-2019 09:46:42.947 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.0.32
ors-app    | 16-Dec-2019 09:46:42.949 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Feb 2 2016 19:34:53 UTC
ors-app    | 16-Dec-2019 09:46:42.949 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:         8.0.32.0
ors-app    | 16-Dec-2019 09:46:42.949 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
ors-app    | 16-Dec-2019 09:46:42.950 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.19.0-6-amd64
ors-app    | 16-Dec-2019 09:46:42.950 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
ors-app    | 16-Dec-2019 09:46:42.950 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/local/openjdk-8/jre
ors-app    | 16-Dec-2019 09:46:42.950 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_232-b09
ors-app    | 16-Dec-2019 09:46:42.951 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
ors-app    | 16-Dec-2019 09:46:42.951 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /usr/local/tomcat
ors-app    | 16-Dec-2019 09:46:42.951 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /usr/local/tomcat
ors-app    | 16-Dec-2019 09:46:42.952 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties
ors-app    | 16-Dec-2019 09:46:42.952 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
ors-app    | 16-Dec-2019 09:46:42.952 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
ors-app    | 16-Dec-2019 09:46:42.953 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:TargetSurvivorRatio=75
ors-app    | 16-Dec-2019 09:46:42.953 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:SurvivorRatio=64
ors-app    | 16-Dec-2019 09:46:42.953 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxTenuringThreshold=3
ors-app    | 16-Dec-2019 09:46:42.954 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseConcMarkSweepGC
ors-app    | 16-Dec-2019 09:46:42.954 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseParNewGC
ors-app    | 16-Dec-2019 09:46:42.954 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:ParallelGCThreads=4
ors-app    | 16-Dec-2019 09:46:42.954 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms220g
ors-app    | 16-Dec-2019 09:46:42.955 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx220g
ors-app    | 16-Dec-2019 09:46:42.955 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote
ors-app    | 16-Dec-2019 09:46:42.955 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.port=9001
ors-app    | 16-Dec-2019 09:46:42.955 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.rmi.port=9001
ors-app    | 16-Dec-2019 09:46:42.956 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
ors-app    | 16-Dec-2019 09:46:42.956 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.ssl=false
ors-app    | 16-Dec-2019 09:46:42.956 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.rmi.server.hostname=localhost
ors-app    | 16-Dec-2019 09:46:42.956 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.endorsed.dirs=/usr/local/tomcat/endorsed
ors-app    | 16-Dec-2019 09:46:42.957 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat
ors-app    | 16-Dec-2019 09:46:42.957 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
ors-app    | 16-Dec-2019 09:46:42.957 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
ors-app    | 16-Dec-2019 09:46:42.957 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
ors-app    | 16-Dec-2019 09:46:43.059 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
ors-app    | 16-Dec-2019 09:46:43.072 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
ors-app    | 16-Dec-2019 09:46:43.075 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-8009"]
ors-app    | 16-Dec-2019 09:46:43.077 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
ors-app    | 16-Dec-2019 09:46:43.077 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 453 ms
ors-app    | 16-Dec-2019 09:46:43.102 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
ors-app    | 16-Dec-2019 09:46:43.103 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.32
ors-app    | 16-Dec-2019 09:46:43.129 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/local/tomcat/webapps/ors.war
ors-app    | 16-Dec-2019 09:46:51.870 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
ors-app    |
ors-app    |   .   ____          _            __ _ _
ors-app    |  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
ors-app    | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
ors-app    |  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
ors-app    |   '  |____| .__|_| |_|_| |_\__, | / / / /
ors-app    |  =========|_|==============|___/=/_/_/_/
ors-app    |  :: Spring Boot ::        (v2.0.4.RELEASE)
ors-app    |
ors-app    | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
ors-app    | SLF4J: Defaulting to no-operation (NOP) logger implementation
ors-app    | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ors-app    | <?xml version="1.0" ?>
ors-app    | <Configuration>
ors-app    |   <Properties>
ors-app    |     <Property name="filename">/var/log/ors/ors-logs.log</Property>
ors-app    |   </Properties>
ors-app    | </Configuration>
ors-app    | 16-Dec-2019 09:47:07.698 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/local/tomcat/webapps/ors.war has finished in 24,567 ms
ors-app    | 16-Dec-2019 09:47:07.699 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/docs
ors-app    | 16-Dec-2019 09:47:07.715 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/docs has finished in 16 ms
ors-app    | 16-Dec-2019 09:47:07.716 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/manager
ors-app    | 16-Dec-2019 09:47:07.742 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/manager has finished in 26 ms
ors-app    | 16-Dec-2019 09:47:07.742 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/ROOT
ors-app    | 16-Dec-2019 09:47:07.757 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/ROOT has finished in 15 ms
ors-app    | 16-Dec-2019 09:47:07.757 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/examples
ors-app    | 16-Dec-2019 09:47:09.423 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/examples has finished in 1,666 ms
ors-app    | 16-Dec-2019 09:47:09.424 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/host-manager
ors-app    | 16-Dec-2019 09:47:09.444 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/host-manager has finished in 20 ms
ors-app    | 16-Dec-2019 09:47:09.459 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
ors-app    | 16-Dec-2019 09:47:09.477 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
ors-app    | 16-Dec-2019 09:47:09.479 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 26401 ms
ors-app    | Exception in thread "PBF Reader" java.lang.RuntimeException: Unable to read PBF file.
ors-app    |    at com.graphhopper.reader.osm.pbf.PbfReader.run(PbfReader.java:50)
ors-app    |    at java.lang.Thread.run(Thread.java:748)
ors-app    | Caused by: java.lang.RuntimeException: java.lang.InterruptedException
ors-app    |    at com.graphhopper.reader.osm.OSMInputFile.process(OSMInputFile.java:259)
ors-app    |    at com.graphhopper.reader.osm.pbf.PbfDecoder.sendResultsToSink(PbfDecoder.java:96)
ors-app    |    at com.graphhopper.reader.osm.pbf.PbfDecoder.processBlobs(PbfDecoder.java:151)
ors-app    |    at com.graphhopper.reader.osm.pbf.PbfDecoder.run(PbfDecoder.java:162)
ors-app    |    at com.graphhopper.reader.osm.pbf.PbfReader.run(PbfReader.java:47)
ors-app    |    ... 1 more
ors-app    | Caused by: java.lang.InterruptedException
ors-app    |    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
ors-app    |    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
ors-app    |    at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
ors-app    |    at com.graphhopper.reader.osm.OSMInputFile.process(OSMInputFile.java:257)
ors-app    |    ... 5 more
ors-app    | 16 Dec 12:52:19 ERROR [routing.RoutingProfileManager] - java.util.concurrent.ExecutionException: java.lang.RuntimeException: Couldn't process file data/osm_file.pbf, error: java.net.SocketTimeoutException: Read timed out
ors-app    | java.util.concurrent.ExecutionException: java.lang.RuntimeException: Couldn't process file data/osm_file.pbf, error: java.net.SocketTimeoutException: Read timed out
ors-app    |    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
ors-app    |    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
ors-app    |    at heigit.ors.routing.RoutingProfileManager.initialize(RoutingProfileManager.java:218)
ors-app    |    at heigit.ors.routing.RoutingProfileManager.getInstance(RoutingProfileManager.java:66)
ors-app    |    at heigit.ors.servlet.listeners.ORSInitContextListener.lambda$contextInitialized$0(ORSInitContextListener.java:43)
ors-app    |    at java.lang.Thread.run(Thread.java:748)
ors-app    | Caused by: java.lang.RuntimeException: Couldn't process file data/osm_file.pbf, error: java.net.SocketTimeoutException: Read timed out
ors-app    |    at com.graphhopper.reader.osm.OSMReader.writeOsm2Graph(OSMReader.java:406)
ors-app    |    at com.graphhopper.reader.osm.OSMReader.readGraph(OSMReader.java:256)
ors-app    |    at com.graphhopper.GraphHopper.importData(GraphHopper.java:741)
ors-app    |    at com.graphhopper.GraphHopper.process(GraphHopper.java:710)
ors-app    |    at com.graphhopper.GraphHopper.importOrLoad(GraphHopper.java:688)
ors-app    |    at heigit.ors.routing.graphhopper.extensions.ORSGraphHopper.importOrLoad(ORSGraphHopper.java:158)
ors-app    |    at heigit.ors.routing.RoutingProfile.initGraphHopper(RoutingProfile.java:185)
ors-app    |    at heigit.ors.routing.RoutingProfile.<init>(RoutingProfile.java:130)
ors-app    |    at heigit.ors.routing.RoutingProfileLoader.call(RoutingProfileLoader.java:37)
ors-app    |    at heigit.ors.routing.RoutingProfileLoader.call(RoutingProfileLoader.java:21)
ors-app    |    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
ors-app    |    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
ors-app    |    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
ors-app    |    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
ors-app    |    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
ors-app    |    ... 1 more
ors-app    | Caused by: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
ors-app    |    at com.graphhopper.reader.dem.AbstractTiffElevationProvider.downloadFile(AbstractTiffElevationProvider.java:159)
ors-app    |    at com.graphhopper.reader.dem.AbstractTiffElevationProvider.getEle(AbstractTiffElevationProvider.java:121)
ors-app    |    at com.graphhopper.reader.dem.MultiSourceElevationProvider.getEle(MultiSourceElevationProvider.java:52)
ors-app    |    at com.graphhopper.reader.osm.OSMReader.getElevation(OSMReader.java:782)
ors-app    |    at com.graphhopper.reader.osm.OSMReader.addNode(OSMReader.java:755)
ors-app    |    at com.graphhopper.reader.osm.OSMReader.processNode(OSMReader.java:723)
ors-app    |    at com.graphhopper.reader.osm.OSMReader.writeOsm2Graph(OSMReader.java:373)
ors-app    |    ... 15 more
ors-app    | Caused by: java.net.SocketTimeoutException: Read timed out
ors-app    |    at java.net.SocketInputStream.socketRead0(Native Method)
ors-app    |    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
ors-app    |    at java.net.SocketInputStream.read(SocketInputStream.java:171)
ors-app    |    at java.net.SocketInputStream.read(SocketInputStream.java:141)
ors-app    |    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
ors-app    |    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
ors-app    |    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
ors-app    |    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
ors-app    |    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
ors-app    |    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
ors-app    |    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
ors-app    |    at com.graphhopper.util.Downloader.fetch(Downloader.java:74)
ors-app    |    at com.graphhopper.util.Downloader.downloadFile(Downloader.java:114)
ors-app    |    at com.graphhopper.reader.dem.AbstractTiffElevationProvider.downloadFile(AbstractTiffElevationProvider.java:155)
ors-app    |    ... 21 more

space 72G free

nils · December 16, 2019, 1:28pm

It’s a little hidden, but what happens is that it can’t download the SRTM tiles it needs to set the elevation correctly, e.g. ors-app | Caused by: java.lang.RuntimeException: Couldn't process file data/osm_file.pbf, error: java.net.SocketTimeoutException: Read timed out

Are you behind a proxy or smth?

rki · December 16, 2019, 1:32pm

No, there is no proxy. The server has a direct connection to the internet.
Can you told me the address. So I can test it via curl or wget.

In the data folder is a folder elevtion_cache. There is a srtm_38_03.zip.

nils · December 16, 2019, 1:54pm

OK, then it might the SRTM data mirror who’s having problems.

@adam didn’t you have problems here recently too?

rki · December 17, 2019, 7:55am

I did another try after deleting the folder eveloation_cache. But the same errors accours.
Need help!

rki · December 17, 2019, 9:52am

While building the dockercontainers this error is shown:

Step 19/29 : RUN mvn -q -f ./openrouteservice/pom.xml package -DskipTests
 ---> Running in 4ac310228aed
Dec 17, 2019 9:46:08 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec 
execute
INFO: I/O exception (org.apache.maven.wagon.providers.http.httpclient.NoHttpResponseException) caught 
when processing request to {}->http://repo.heigit.org:80: The target server failed to respond
Dec 17, 2019 9:46:08 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec 
execute
INFO: Retrying request to {}->http://repo.heigit.org:80

adam · December 17, 2019, 2:39pm

The error relating to the “retrying request” from docker might be due to something happening with the server infrastructure here earlier today, so hopefully that should work again now.

@nils the problems I was having was to do with the gmted elevation, and most likely because of the server dropping the connection mid download - but that isn’t an issue to do with the ors code and more to do with the server network itself. Normally when there is a problem with the elevation, normally completely wiping the elevation data folder that it is downloaded into and then restarting fixes it. I just tested downloading from the srtm website for that tile and it downloads ok, so it could either be a temporary thing on their end, or something relating to the netwok

rki · December 18, 2019, 1:56pm

Hi,

the error with the repo.heigit.org is still there.

Please help!

nils · December 19, 2019, 12:43pm

The Docker container has access to the internet? Try to wget or curl something from inside the container.

rki · December 19, 2019, 2:11pm

Yes the containers have access to the internet. curl, and wget ok. ping to google ok too.

amandus · December 19, 2019, 2:56pm

Could it make a difference if we use https://repo.heigit.org instead?
Just a thought

nils · December 20, 2019, 7:21am

I really have no clue why this is happening… The heigit repository is online. Don’t think it’s a SSL issue;) Can’t hurt of course, but the pom is like that since ages.

Must be some configuration on your server that needs changing. The image builds fine on our servers.