RIAK & Node is not reachable!

RIAK node is not reachable! 

This has happened a few times when setting up RIAK clusters and by doing a little googling it looks like somewhat common problem around RIAK users, especially new ones. So I decided to do a short blog post hopefully providing a little more understanding on why this might have happened to you :)

This is the scenario:

You've setup a new RIAK cluster on your preferred method and notice that the nodes are happily up but have no idea on each other.

root@riak-test-cluster-10-8-2-5:~# riak-admin ring-status
Attempting to restart script through sudo -H -u riak
================================== Claimant ===================================
Claimant:  'riak@10.8.2.5'
Status:     up
Ring Ready: true

============================== Ownership Handoff ==============================
No pending changes.

============================== Unreachable Nodes ==============================
All nodes are up and reachable

Sounds good when all nodes are up and reachable, but looks bad when there is just one of'em :) Your first option is to go and do the ping, if for no other reason, just to see the pong reply and have a smile:

root@riak-test-cluster-10-8-2-5:~# riak ping
Attempting to restart script through sudo -H -u riak
pong

Voilá! RIAK seems happy in this sense. Let's try to join another member into the cluster:

root@riak-test-cluster-10-8-2-5:~:~# riak-admin cluster join riak@10.8.2.6
Attempting to restart script through sudo -H -u riak
Node riak@10.8.2.6 is not reachable!

Oh noes :( So this is why you're here, let's get busy solving the problem! First of all: check your firewall. The checklist of doing that can be found here in the RIAK documentation:

The documentation states that following ports are needed to be opened:

Riak nodes in a cluster need to be able to communicate freely with one another on the following ports:

• epmd listener: TCP:4369
• handoff_port listener: TCP:8099
• range of ports specified in app.config

Riak clients must be able to contact at least one machine in a Riak cluster on the following ports:

• web_port: TCP:8098
• pb_port: TCP:8087

So make sure this is indeed what you have in place. If you're using iptables, start from there. If you're more open about yourself and rely on, say, Amazon security groups, that's your next target. What ever you do, check that the ports are both open, being listened and reachable between nodes by using your favorite approach (and yes, use of telnet is fine, just delete it if it wasn't part of your install base ;)).

Depending on if any changes were made, give it another go.

Still no luck? Let's do the next step and go dumpster diving. Enter the world of tcpdump. Your usage may vary, but since this node has pretty much no other traffic than our connection to the ssh server in port 22, this does the trick (note the 'not' before port number or you'll immediately regret this decision):

root@riak-test-cluster-10-8-2-5:~# tcpdump -ni eth0 port not 22 &
[1] 31478

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

12:31:38.221485 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG kernel.info, length: 140

root@riak-test-cluster-10-8-2-5:~# riak-admin cluster join riak@10.8.2.612:31:43.225372 ARP, Request who-has 10.8.2.18 tell 10.8.2.5, length 28

12:31:43.225500 ARP, Reply 10.8.2.18 is-at 0a:c0:08:c4:92:6d, length 28

-admin cluster join riak@10.8.2.6

Attempting to restart script through sudo -H -u riak

12:31:46.138724 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG authpriv.notice, length: 170

12:31:46.139716 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG authpriv.info, length: 122

12:31:47.337759 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [S], seq 903580437, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:31:47.338163 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [S.], seq 3918144107, ack 903580438, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:31:47.338191 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [.], ack 1, win 115, length 0

12:31:47.338231 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [P.], seq 1:8, ack 1, win 115, length 7

12:31:47.338461 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [.], ack 8, win 115, length 0

12:31:47.338549 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [P.], seq 1:19, ack 8, win 115, length 18

12:31:47.338566 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [.], ack 19, win 115, length 0

12:31:47.338587 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [F.], seq 19, ack 8, win 115, length 0

12:31:47.338618 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [F.], seq 8, ack 20, win 115, length 0

12:31:47.338710 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:31:47.338805 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [.], ack 9, win 115, length 0

12:31:48.337371 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:31:50.341369 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

Whoa! What's this. First it looks like everything is there's traffic to port 4369 which goes fine, but what is this traffic back to 57459 that fails? This isn't listed in the port range! Halp! So we end up with the inevitable:

Node riak@10.8.2.6 is not reachable!

Now if you're like me, you'll go back to the page where the ports were listened and consider writing an angry email to Basho stating they've ruined your afternoon by not documenting their stuff properly, which is when you'll probably notice the talk about port mapping and app config before the ports were being listed.

Oh yes, enter the world of port mapping. The way RIAK behaves is much like any other program that requires portmapper. If no limits are set for the port range to use, it simply sends a port number 0 to the operating system, which in general returns the next available port and uses that. This is not a problem if your nodes have no trouble communicating through basically ports between 0-65536, but usually the security people start twitching if you even suggest this being the case. That's where the basho's app.config file steps in:

{ kernel, [
            {inet_dist_listen_min, 6000},
            {inet_dist_listen_max, 7999}
          ]},

There we go. You can restrict the port range to something reasonable yet big enough range for RIAK to do its job. Do this, set your firewall accordingly, restart your node and I think things turn better in no time :)

They didn't? Perhaps you'll need to write that email before, but just before you do and if this is an option for you: allow the whole port range (TCP) and try again. If this works out for you, you'll probably have a rule missing or not applied somewhere. And if you don't, it's time to, well bash the basho...

I'll get my coat.