tiistai 10. syyskuuta 2013

Bourne Again Fun

Some Quality Time

I've recently had an opportunity to get back familiar with my old friend, bash. I would have called him an acquaintance instead of friend since it's been so long since we last spent truly time together, but we're back to friend status. (Remind me to update our Facebook relationship!)

It all started from something that was supposed to a small script (aren't they always) to help bootstrap a VPC from scratch in Amazon AWS, since it involves quite a lot of nibbling around in the VPC configuration to get it ready for instance deployment. Most if not all of this could be scripted so I decided to spend a couple of days to get it done.

Turned out I got to spend several weeks with this and since I started it all with bash, I might as well finnish with it. Although many people will say, and for a reason, this might've not been the best decision, I considered it a good exercise to get to really know AWS more than just bash exercise. 

But without going into too much details into the whole AWS shenanigans, I decided to share some of bash intricacies I encountered on the way that are easily forgotten or that people seem to find confusing (saw quite a lot of links to Stack Overflow when researching for these).

Arrays - A String Concerto

Yes, bash has arrays and those are quite useful as well, but they are also provide loads of pitfalls to step into. I needed quite a lot of these for various reasons, so here are some that I think are useful tips for you.

$ friend="angry bird"
$ friends_array=($var)

That's a very basic array with space separated string in it. Let's echo the first element:
$ echo ${friends_array[0]}
angry

Great, just what you would expect and similar results on the index of 1. But what if we wrap the variable into double quotes and then echo the first element?

$ friends_array=("$var")
$ echo ${friends_array[0]}
angry bird

You'll get both words without the double quotes as expected, right? There are cases when you might want this, but in many cases you want the space separated word list in there, like so:

# birds_array =("red bird" "blue bird" "yellow bird")
# echo ${birds_array[1]}
blue bird

Meet the IFS

Then there are cases when you need to use arrays but want to use different separator. That where the IFS variable comes handy, which is easy enough to use, but again full of surprises for the unprepared.

First of all, if using default values for separation of: <space><tab><newline>

You can override it as follows:
$ friends="red bird, chuck"
$ IFS=","
$ friends_array=($friends)
$ echo ${friends_array[0]}
red bird

Also, note if you used space to separate the var as we did above, the echo will print that as well:
echo ${friends_array[1]}
 chuck

So even if your mind is used to neat separation by having space after each comma or colon, bash will take this quite literally. Remember to strip your variables or keep your lists clean.

You might also be clever (in your mind) and think to use the following (as I actually did):

$ IFS=" ,"
friends_array=($friends)

Oh, did I mention you need to redefine the array after changing the IFS? It does the separation into array when setting the array variable, not using it as a separator for output via echo for instance. I don't demonstrate that here, but try it and you will see for yourself.

$ echo ${friends_array[0]}
red

Wait, what? Let's see the rest:

$ echo ${friends_array[1]}
bird
$ echo ${friends_array[2]}
chuck

But that's not what I wanted! I thought I created a literal <space><,> as a separator, but instead I made a list of characters to use, much like the IFS has by default. Of course this is what the man pages state as well, but this is for all those brave souls that experiment before reading documentation ;)

So, you've met the IFS. Now don't forget him! You've altered his personality, now change it back if you don't want it to affect the rest of your scripts. Or, even better, use a sub shell when you need to change IFS value to make the change temporary.

Hoop-a-loop

Back to arrays. They're usually good for one purpose: looping through. But beware, what you learned above applies here as well with some exceptions.

Let's make a small shell script called bird.sh:

#!/bin/bash
arr=("red bird" "blue bird" "yellow bird")
for bird in ${arr[@]}; do
echo $bird
done

Running this will output:

$ sh bird.sh 
red
bird
blue
bird
yellow
bird

Exactly. Forgot the double quotes around the array, so instead of listing the elements we thought we setup in the array we just looped through the words using the default IFS separator, space tab and newline. Change the for clause to:
for bird in "${arr[@]}"; do

And your output is what you would expect:

$ sh bird.sh 
red bird
blue bird
yellow bird

Bracketeering

You've probably seen some if statements in bash and stepped into the first mine of every developer oriented soul of using something like:
if [ "foo" == $bar ]; then

Depending on what you want usually the first thing you encounter are that bash might not like space around the comparator or the equal signs need not (or can't) be double etc. But this is old news. What you might not have known is that [ is not only a syntax, it's an actual command. Don't believe me?

$ which [
/bin/[

Whaat, no way! Next, try checking the man page next. It's the condition evaluation utility. This makes some comparisons more interesting than others. See following example:

$ var="angry bird"
$ [ $var = "angry bird" ] && echo match
-bash: [: too many arguments

Bummer. This is caused by the comparison string having space in between. Bash no likey. But no worries, that's when we can use double brackets:

$ [[ $var = "angry bird" ]] && echo match
match

Now, why this is can be go deeper than I'd like to venture here, but as a general rule the world splitting in variables are usually not expected unless specifically prepared to do so. Also, should you have used some other IFS here than space, it would've also worked in the single bracket case since it considers the variable as one string.

There are cases like switch-case (or case-esac in bash, which I always find funny much like if-fi statements) that also in general don't like these kind of strings either, but work if you prepare for a split string explicitly by double (or single) quotes around the variable. 

Fun and Games

The syntax around bash arrays are sometimes something completely else than what an innocent developer is prepared for, but quite easy to remember once familiar with. The most useful I've found are the amount of elements in an array. 

Set the array as follows:

$ friends_array=("red bird" "blue bird" "yellow bird")

List the amount of elements:

$ echo ${#friends_array[*]}
3

Here lies another caveat. Remember IFS? Remember I told you it will affect only after variable is set into the array? In here I must emphasise the use of variables and not static strings. Let's do this again:

$ IFS=","
$ friends_array=("red bird","blue bird","yellow bird")
echo ${#friends_array[*]}
1

Aaargh! Didn't I just told it to use comma as a separator. You did, but like said, this applies only variables. Let's do it via a variable:

$ IFS=","
$ friends="red bird,blue bird,yellow bird"
$ friends_array($friends)
echo ${#friends_array[*]}
3

Voilá ;)

I could continue on the subject, but where I have ventured, many men have gone before me. Did some googling and this seems to cover quite nicely many of the bashiness.

Have fun.sh!


perjantai 30. elokuuta 2013

RIAK & Node is not reachable!


RIAK node is not reachable! 


This has happened a few times when setting up RIAK clusters and by doing a little googling it looks like somewhat common problem around RIAK users, especially new ones. So I decided to do a short blog post hopefully providing a little more understanding on why this might have happened to you :)

This is the scenario:

You've setup a new RIAK cluster on your preferred method and notice that the nodes are happily up but have no idea on each other.

root@riak-test-cluster-10-8-2-5:~# riak-admin ring-status
Attempting to restart script through sudo -H -u riak
================================== Claimant ===================================
Claimant:  'riak@10.8.2.5'
Status:     up
Ring Ready: true

============================== Ownership Handoff ==============================
No pending changes.

============================== Unreachable Nodes ==============================
All nodes are up and reachable

Sounds good when all nodes are up and reachable, but looks bad when there is just one of'em :) Your first option is to go and do the ping, if for no other reason, just to see the pong reply and have a smile:

root@riak-test-cluster-10-8-2-5:~# riak ping
Attempting to restart script through sudo -H -u riak
pong

Voilá! RIAK seems happy in this sense. Let's try to join another member into the cluster:

root@riak-test-cluster-10-8-2-5:~:~# riak-admin cluster join riak@10.8.2.6
Attempting to restart script through sudo -H -u riak
Node riak@10.8.2.6 is not reachable!

Oh noes :( So this is why you're here, let's get busy solving the problem! First of all: check your firewall. The checklist of doing that can be found here in the RIAK documentation:

The documentation states that following ports are needed to be opened:

Riak nodes in a cluster need to be able to communicate freely with one another on the following ports:
  • epmd listener: TCP:4369
  • handoff_port listener: TCP:8099
  • range of ports specified in app.config
Riak clients must be able to contact at least one machine in a Riak cluster on the following ports:
  • web_port: TCP:8098
  • pb_port: TCP:8087

So make sure this is indeed what you have in place. If you're using iptables, start from there. If you're more open about yourself and rely on, say, Amazon security groups, that's your next target. What ever you do, check that the ports are both open, being listened and reachable between nodes by using your favorite approach (and yes, use of telnet is fine, just delete it if it wasn't part of your install base ;)).

Depending on if any changes were made, give it another go.

Still no luck? Let's do the next step and go dumpster diving. Enter the world of tcpdump. Your usage may vary, but since this node has pretty much no other traffic than our connection to the ssh server in port 22, this does the trick (note the 'not' before port number or you'll immediately regret this decision):

root@riak-test-cluster-10-8-2-5:~# tcpdump -ni eth0 port not 22 &
[1] 31478

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:31:38.221485 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG kernel.info, length: 140

root@riak-test-cluster-10-8-2-5:~# riak-admin cluster join riak@10.8.2.612:31:43.225372 ARP, Request who-has 10.8.2.18 tell 10.8.2.5, length 28
12:31:43.225500 ARP, Reply 10.8.2.18 is-at 0a:c0:08:c4:92:6d, length 28
-admin cluster join riak@10.8.2.6
Attempting to restart script through sudo -H -u riak
12:31:46.138724 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG authpriv.notice, length: 170
12:31:46.139716 IP 10.8.2.5.43449 > 10.8.2.18.514: SYSLOG authpriv.info, length: 122
12:31:47.337759 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [S], seq 903580437, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:31:47.338163 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [S.], seq 3918144107, ack 903580438, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:31:47.338191 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [.], ack 1, win 115, length 0
12:31:47.338231 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [P.], seq 1:8, ack 1, win 115, length 7
12:31:47.338461 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [.], ack 8, win 115, length 0
12:31:47.338549 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [P.], seq 1:19, ack 8, win 115, length 18
12:31:47.338566 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [.], ack 19, win 115, length 0
12:31:47.338587 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [F.], seq 19, ack 8, win 115, length 0
12:31:47.338618 IP 10.8.2.5.57459 > 10.8.2.6.4369: Flags [F.], seq 8, ack 20, win 115, length 0
12:31:47.338710 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:31:47.338805 IP 10.8.2.6.4369 > 10.8.2.5.57459: Flags [.], ack 9, win 115, length 0
12:31:48.337371 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
12:31:50.341369 IP 10.8.2.5.50074 > 10.8.2.6.58491: Flags [S], seq 3771522001, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

Whoa! What's this. First it looks like everything is there's traffic to port 4369 which goes fine, but what is this traffic back to 57459 that fails? This isn't listed in the port range! Halp! So we end up with the inevitable:

Node riak@10.8.2.6 is not reachable!

Now if you're like me, you'll go back to the page where the ports were listened and consider writing an angry email to Basho stating they've ruined your afternoon by not documenting their stuff properly, which is when you'll probably notice the talk about port mapping and app config before the ports were being listed.

Oh yes, enter the world of port mapping. The way RIAK behaves is much like any other program that requires portmapper. If no limits are set for the port range to use, it simply sends a port number 0 to the operating system, which in general returns the next available port and uses that. This is not a problem if your nodes have no trouble communicating through basically ports between 0-65536, but usually the security people start twitching if you even suggest this being the case. That's where the basho's app.config file steps in:

{ kernel, [
            {inet_dist_listen_min, 6000},
            {inet_dist_listen_max, 7999}
          ]},

There we go. You can restrict the port range to something reasonable yet big enough range for RIAK to do its job. Do this, set your firewall accordingly, restart your node and I think things turn better in no time :)

They didn't? Perhaps you'll need to write that email before, but just before you do and if this is an option for you: allow the whole port range (TCP) and try again. If this works out for you, you'll probably have a rule missing or not applied somewhere. And if you don't, it's time to, well bash the basho...

I'll get my coat.




keskiviikko 10. heinäkuuta 2013

Maven -version

"Use Maven, you silly old school grumpy developer!", they said. 

"It'll solve all your dependency problems", they also said.

I hate them.

It's not that I dislike maven as such. It does fantastic job what comes to dependencies - when it works. It also does fantastic job when it doesn't work, but only when it gives you reason why you failed. When it looks like it's doing it job, but it quite isn't, I'll get back to my hating game.

Not to pile on to your problems, here's what happened so perhaps I can make you hate me less at least. 

I was trying to compile a project in my OSX desktop and the tests started failing for no apparent reason. The very same project from the same source tree compiled nicely in a Linux desktop running Ubuntu and the tests ran fine.

First of all the poor mac was seemed to be missing a dependency that ought to have come transitively through the project pom dependencies. You might have an opinion on the transitive dependencies (I certainly do), so without going deeper into that subject let's just say the mvn depencency:tree (one of my favorite maven plugins) told me that it's not there, nor was the jar associated it ever finding its way to the target lib directory.

Maven still wasn't my first suspect. I thought it must be something that the OSX does differently, but my colleague did immediately say with a laconic voice that: "It's probably just one of those issues with your stupid mac and maven versions". 

He turned out to be right, at least on the maven part.

I checked the version on both workstations and in Ubuntu it showed version 3.0.4 whereas the mac showed 3.0.3, which is I guess the default that comes as bundled in. I did install latest maven with brew (http://mxcl.github.io/homebrew/) and it installed 3.0.5 out of the box.

Now, depending on your path settings, the /usr/local/bin might or might not be in the correct place, so check with "which maven" which is the one pointing to your binary. There's probably a symbolic link in /usr/bin/mvn pointing to your bundled one, so you want to either remove that or point it elsewhere.

The new maven package is probably located in: /usr/local/Cellar/maven/<mvn version>/libexec

This is probably information you want to tell your favorite IDE in case you'll ever need to point the maven home directory into it. 

What you also might want to do is to do what many do with java, which is to point another symbolic link into your current maven version, assuming you're changed into that directory and say for instance:

[/usr/local/Cellar/maven] tiltti@sofia$ sudo ln -s 3.0.5/libexec/ current
After which you can point to your maven home dir as: /usr/local/Cellar/maven/current in IDE and other places statically. Just remind to update your symlink accordingly in case you do upgrades :)

Anyhow, after all this maven brewing, the jar found its way back to the project and all the tests were executed without problems.

Yay.

It still left me thinking my relationship with maven. I certainly don't miss ant... or actually I do miss it. But it's more like missing 4DOS for more nostalgic reasons than a reason that it's better what's out there. 

Maybe it's because I've never really gotten kick out of being able to compile complex projects and do releases, and I hate everything that produces way much output than actual, useful information, that bugs me in general. It's like starting pretty much any J2EE container and looking at the puke it throws out to log files even at the default, preferred logging levels. And when it doesn't start, it's usually a missing class/jar file and a stacktrace anyhow that you need to start digging into.

What I wish maven would have is less verbose and more friendly output. And that people who do love the release would learn how not to make the project download ALL the dependencies in every single run without explicitly telling maven not to do so with -o option.

I'm rambling. 

TL;DR Maven works if you're less stupid than me.