CGNAT Service crashed and appears down

Overview

The customer reports on the CGNAT Server one or more of the services are not running. It appears running the cli.sh command for the service is throwing an error:

bash-4.2$ sh cli.sh 8944
[ 22-May-2023 05:48:23 ] [ TRACE ] [ JMXINTERFACE****************** ] : java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at com.elitecore.core.util.cli.JMXInterface.init(JMXInterface.java:74)
at com.elitecore.core.util.cli.cmd.Commands.<init>(Commands.java:40)
at com.elitecore.core.util.cli.BaseConsoleManager.init(BaseConsoleManager.java:101)
at com.elitecore.core.util.cli.BaseConsoleManager.<init>(BaseConsoleManager.java:56)
at com.elitecore.mediation.ConsoleManager.<init>(ConsoleManager.java:35)
at com.elitecore.mediation.ConsoleManager.main(ConsoleManager.java:95)

Solution

The solution will be start or restart the services. 

• First step is to run jps -v and confirm if the Service related process is running. In this port is 8944 and jps shows process is not running
  • bash-4.2$ jps -v
    69 ServerMgmt-7.4.0.jar -Dlog4j.configuration=serverMgmtLogger.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8901 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=8901 -Dcom.sun.management.jmxremote.local.only=false
    32405 Main -Xms8192m -Xmx18432m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8946 -Djava.rmi.server.hostname=192.168.9.18 -Dcom.sun.management.jmxremote.rmi.port=8946 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false
    22633 Jps -Dapplication.home=/opt/crestelsetup/jdk1.8.0_65 -Xms8m
    32491 Main -Xms3072m -Xmx8192m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8947 -Djava.rmi.server.hostname=192.168.9.18 -Dcom.sun.management.jmxremote.rmi.port=8947 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false
    29484 ConsoleManager
    32350 Main -Xms10240m -Xmx25600m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8945 -Djava.rmi.server.hostname=192.168.9.18 -Dcom.sun.management.jmxremote.rmi.port=8945 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false
• If the process is running kill the process using kill <pid>
• If the process is not running then simply start the service from the  crestelpengine/modules/mediation/bin/ path and running - sh startServer_<service port>.sh
  • The list of scripts to start services can be found under the crestelpengine/modules/mediation/bin/ directory
  • 

Verification

After starting the server, then execute sh cli.sh and then check services and COUNTERSTATUS commands