RIAK & Node is not reachable!

RIAK node is not reachable! 

This has happened a few times when setting up RIAK clusters and by doing a little googling it looks like somewhat common problem around RIAK users, especially new ones. So I decided to do a short blog post hopefully providing a little more understanding on why this might have happened to you :)

This is the scenario:

You've setup a new RIAK cluster on your preferred method and notice that the nodes are happily up but have no idea on each other.

root@riak-test-cluster-10-8-2-5:~# riak-admin ring-status
Attempting to restart script through sudo -H -u riak
================================== Claimant ===================================
Claimant:  'riak@10.8.2.5'
Status:     up
Ring Ready: true

============================== Ownership Handoff ==============================
No pending changes.

============================== Unreachable Nodes ==============================
All nodes are up and reachable

Sounds good when all nodes are up and reachable, but looks bad when there is just one of'em :) Your first option is to go and do the ping, if for no other reason, just to see the pong reply and have a smile:

root@riak-test-cluster-10-8-2-5:~# riak ping
Attempting to restart script through sudo -H -u riak
pong

Voilá! RIAK seems happy in this sense. Let's try to join another member into the cluster:

root@riak-test-cluster-10-8-2-5:~:~# riak-admin cluster join riak@10.8.2.6
Attempting to restart script through sudo -H -u riak
Node riak@10.8.2.6 is not reachable!

Oh noes :( So this is why you're here, let's get busy solving the problem! First of all: check your firewall. The checklist of doing that can be found here in the RIAK documentation:

The documentation states that following ports are needed to be opened:

Riak nodes in a cluster need to be able to communicate freely with one another on the following ports:

• epmd listener: TCP:4369
• handoff_port listener: TCP:8099
• range of ports specified in app.config

Riak clients must be able to contact at least one machine in a Riak cluster on the following ports:

• web_port: TCP:8098
• pb_port: TCP:8087

So make sure this is indeed what you have in place. If you're using iptables, start from there. If you're more open about yourself and rely on, say, Amazon security groups, that's your next target. What ever you do, check that the ports are both open, being listened and reachable between nodes by using your favorite approach (and yes, use of telnet is fine, just delete it if it wasn't part of your install base ;)).

Depending on if any changes were made, give it another go.

Still no luck? Let's do the next step and go dumpster diving. Enter the world of tcpdump. Your usage may vary, but since this node has pretty much no other traffic than our connection to the ssh server in port 22, this does the trick (note the 'not' before port number or you'll immediately regret this decision):

root@riak-test-cluster-10-8-2-5:~# tcpdump -ni eth0 port not 22 &
[1] 31478

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

12:31:38.221485 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG kernel.info, length: 140

root@riak-test-cluster-10-8-2-5:~# riak-admin cluster join riak@10.8.2.612:31:43.225372 ARP, Request who-has 10.8.2.18 tell 10.8.2.5, length 28

12:31:43.225500 ARP, Reply 10.8.2.18 is-at 0a:c0:08:c4:92:6d, length 28

-admin cluster join riak@10.8.2.6

Attempting to restart script through sudo -H -u riak

12:31:46.138724 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG authpriv.notice, length: 170

12:31:46.139716 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG authpriv.info, length: 122

12:31:47.337759 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [S], seq 903580437, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:31:47.338163 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [S.], seq 3918144107, ack 903580438, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:31:47.338191 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [.], ack 1, win 115, length 0

12:31:47.338231 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [P.], seq 1:8, ack 1, win 115, length 7

12:31:47.338461 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [.], ack 8, win 115, length 0

12:31:47.338549 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [P.], seq 1:19, ack 8, win 115, length 18

12:31:47.338566 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [.], ack 19, win 115, length 0

12:31:47.338587 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [F.], seq 19, ack 8, win 115, length 0

12:31:47.338618 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [F.], seq 8, ack 20, win 115, length 0

12:31:47.338710 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:31:47.338805 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [.], ack 9, win 115, length 0

12:31:48.337371 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

12:31:50.341369 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

Whoa! What's this. First it looks like everything is there's traffic to port 4369 which goes fine, but what is this traffic back to 57459 that fails? This isn't listed in the port range! Halp! So we end up with the inevitable:

Node riak@10.8.2.6 is not reachable!

Now if you're like me, you'll go back to the page where the ports were listened and consider writing an angry email to Basho stating they've ruined your afternoon by not documenting their stuff properly, which is when you'll probably notice the talk about port mapping and app config before the ports were being listed.

Oh yes, enter the world of port mapping. The way RIAK behaves is much like any other program that requires portmapper. If no limits are set for the port range to use, it simply sends a port number 0 to the operating system, which in general returns the next available port and uses that. This is not a problem if your nodes have no trouble communicating through basically ports between 0-65536, but usually the security people start twitching if you even suggest this being the case. That's where the basho's app.config file steps in:

{ kernel, [
            {inet_dist_listen_min, 6000},
            {inet_dist_listen_max, 7999}
          ]},

There we go. You can restrict the port range to something reasonable yet big enough range for RIAK to do its job. Do this, set your firewall accordingly, restart your node and I think things turn better in no time :)

They didn't? Perhaps you'll need to write that email before, but just before you do and if this is an option for you: allow the whole port range (TCP) and try again. If this works out for you, you'll probably have a rule missing or not applied somewhere. And if you don't, it's time to, well bash the basho...

I'll get my coat.

Maven -version

"Use Maven, you silly old school grumpy developer!", they said. 

"It'll solve all your dependency problems", they also said.

I hate them.

It's not that I dislike maven as such. It does fantastic job what comes to dependencies - when it works. It also does fantastic job when it doesn't work, but only when it gives you reason why you failed. When it looks like it's doing it job, but it quite isn't, I'll get back to my hating game.

Not to pile on to your problems, here's what happened so perhaps I can make you hate me less at least. 

I was trying to compile a project in my OSX desktop and the tests started failing for no apparent reason. The very same project from the same source tree compiled nicely in a Linux desktop running Ubuntu and the tests ran fine.

First of all the poor mac was seemed to be missing a dependency that ought to have come transitively through the project pom dependencies. You might have an opinion on the transitive dependencies (I certainly do), so without going deeper into that subject let's just say the mvn depencency:tree (one of my favorite maven plugins) told me that it's not there, nor was the jar associated it ever finding its way to the target lib directory.

Maven still wasn't my first suspect. I thought it must be something that the OSX does differently, but my colleague did immediately say with a laconic voice that: "It's probably just one of those issues with your stupid mac and maven versions". 

He turned out to be right, at least on the maven part.

I checked the version on both workstations and in Ubuntu it showed version 3.0.4 whereas the mac showed 3.0.3, which is I guess the default that comes as bundled in. I did install latest maven with brew (http://mxcl.github.io/homebrew/) and it installed 3.0.5 out of the box.

Now, depending on your path settings, the /usr/local/bin might or might not be in the correct place, so check with "which maven" which is the one pointing to your binary. There's probably a symbolic link in /usr/bin/mvn pointing to your bundled one, so you want to either remove that or point it elsewhere.

The new maven package is probably located in: /usr/local/Cellar/maven/<mvn version>/libexec

This is probably information you want to tell your favorite IDE in case you'll ever need to point the maven home directory into it. 

What you also might want to do is to do what many do with java, which is to point another symbolic link into your current maven version, assuming you're changed into that directory and say for instance:

[/usr/local/Cellar/maven] tiltti@sofia$ sudo ln -s 3.0.5/libexec/ current

After which you can point to your maven home dir as: /usr/local/Cellar/maven/current in IDE and other places statically. Just remind to update your symlink accordingly in case you do upgrades :)

Anyhow, after all this maven brewing, the jar found its way back to the project and all the tests were executed without problems.

Yay.

It still left me thinking my relationship with maven. I certainly don't miss ant... or actually I do miss it. But it's more like missing 4DOS for more nostalgic reasons than a reason that it's better what's out there. 

Maybe it's because I've never really gotten kick out of being able to compile complex projects and do releases, and I hate everything that produces way much output than actual, useful information, that bugs me in general. It's like starting pretty much any J2EE container and looking at the puke it throws out to log files even at the default, preferred logging levels. And when it doesn't start, it's usually a missing class/jar file and a stacktrace anyhow that you need to start digging into.

What I wish maven would have is less verbose and more friendly output. And that people who do love the release would learn how not to make the project download ALL the dependencies in every single run without explicitly telling maven not to do so with -o option.

I'm rambling. 

TL;DR Maven works if you're less stupid than me.