MDA ElasticSearch propagation fails: NoNodeAvailableException

Problem

MDA load fails with the following (or similar) exception:

19.07.2018 07:17:29 [FATAL]   [MdaBackendComponent][MdaBackendService] Elasticsearch bulk request failed
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{<IP_ADDRESS>}{<HOSTNAME>/<IP_ADDRESS>:9300}]]
    at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
    at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
    at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
......

Solution

This error generally indicates either a network or hardware issue:

• In order to eliminate the resources aspect (more unlikely), check if your ElasticSearch process has been killed before/around the time the process failed.

On most Linux distributions automatic process terminations are done by the OutOfMemory (OOM) killer and its activity can be checked as below:

[root@host ~]# dmesg | egrep -i 'killed process'
Killed process 30622, UID 494, (java) total-vm:5753452kB, anon-rss:4526652kB, file-rss:114600kB

If such entries are present, confirm that your hardware conforms to ElasticSearch's minimum requirements and is adequate for the project at hand. The most important rule is that a node needs 8Gb+ of RAM, half of which are dedicated for the ES heap (no more than half!) and the rest - for Lucene index cache.

• The more plausible troubleshooting track would be to look for a network issue. Especially if the MDC server and the ES master/data nodes are located on different networks with a firewall(s)/switche(s) between them.

Replace the domain names in any configuration files with IP addresses. This is intended to circumvent any DNS server troubles. Inquire client about any VPN or proxy specifics.

Read through this article  - you have to enable OS' keepalive mechanisms on the ES connections and lower the interval between the keepalive probes. Also, if using an older version of Linux kernel, this post explains a bug and how its behavior can be circumvented.

Finally, switch to a setup, where the Elastic Search master node is on the machine running MDC itself and the rest of the machines are data nodes only. Such configuration can be achieved by switching to a Node Clent, as explained in MDA ElasticSearch Connectivity and adding the following to the elasticsearch.yml file of the other nodes:

node.master: true
node.data: false

The Node Client uses a proprietary protocol which internal testing has proven to be more robust than the Transport Client.