Today I was working on creating a logstash -> logstash -> elasticsearch -> kibana setup. As we run a dual stack ip v4/v6 environment both ipv4 and ipv6 addresses need to work. After installing logstash and adding the following config as an input filter on the indexer:
input {
amqp {
# get logs to the ‘rawlogs’ fanout queue.
type => “all”
host => “rabbitmq0.domb.net”
user => “logs”
password => “xxxx”
exchange => “rawlogs”
name => “rawlogs_consumer”
}
}
and output filter on the shipper:
output {
# Output events to stdout for debugging. Feel free to remove
# this output if you don’t need it.
#stdout { }
# Ship events to the amqp fanout queue named ‘rawlogs”
amqp {
host => “rabbitmq0.domb.net”
user => “logs”
password => “xxxx”
exchange_type => “fanout”
name => “rawlogs”
port => “5672”
}
}
I started logstash on the shipper and indexer. Boom!!! Logstash crashed. What happend? I looked at the logs and saw the following error message:
Unable to access jarfile /usr/local/logstash/logstash-1.1.1-monolithic.jar
{“message”:”AMQP connection error (during connect), will reconnect”,”exception”:”Connection refused – Connection refused”,”backtrace”:[“file:/usr/local/logstash/logstash-1.1.1-monolithic.jar!/gems/bunny-0.7.9/lib/qrack/client.rb:227:in `socket'”,”file:/usr/local/logstash/logstash-1.1.1-monolithic.jar!/gems/bunny-0.7.9/lib/bunny/client08.rb:369:in `start_session'”,”org/jruby/RubyKernel.java:1410:in `loop'”,”file:/usr/local/logstash/logstash-1.1.1-monolithic.jar!/gems/bunny-0.7.9/lib/bunny/client08.rb:367:in `start_session'”,”file:/usr/local/logstash/logstash-1.1.1-monolithic.jar!/logstash/outputs/amqp.rb:82:in `connect'”,”file:/usr/local/logstash/logstash-1.1.1-monolithic.jar!/logstash/outputs/amqp.rb:62:in `register'”,”file:/usr/local/logstash/logstash-1.1.1-monolithic.jar!/logstash/agent.rb:719:in `run_output'”,”file:/usr/local/logstash/logstash-1.1.1-monolithic.jar!/logstash/agent.rb:719:in `run_output'”,”file:/usr/local/logstash/logstash-1.1.1-monolithic.jar!/logstash/agent.rb:362:in `start_output'”],”level”:”error”}
That’s interesting because I know that I can access rabbitmq0.domb.net via ipv6 with bunny and the system ruby. Seems jruby and openjdk has a problem here. So I tcpdumpd it. Looking at the tcpdump I realized that my jruby bunny got the ipv4 address back when it tried to resolve rabbitmq0.domb.net. This did not happen with the system ruby. So It seems that openjdk prefers ipv4 over ipv6 when doing DNS lookups. So I researched if I can pass a parameter as a java option to prefer ipv6 over ipv4. To my surprise I found one. The solution is to add the following option to the Logstash startup script:
-Djava.net.preferIPv6Addresses=true
This option does the following: (http://docs.oracle.com/javase/1.4.2/docs/guide/net/properties.html)
“java.net.preferIPv6Addresses (default: false)
- If IPv6 is available on the operating system the default preference is to prefer an IPv4-mapped address over an IPv6 address. This is for backward compatibility reasons – for example applications that depend on access to an IPv4 only service or applications that depend on the %d.%d.%d.%d representation of an IP address. This property can be set to try to change the preferences to use IPv6 addresses over IPv4 addresses. This allows applications to be tested and deployed in environments where the application is expected to connect to IPv6 services.”
To add that option system wide you can set the variable
export _JAVA_OPTIONS=’-Djava.net.preferIPv6Addresses=true’
The variable _JAVA_OPTIONS will be picked up by java when starting logstash. If you tail the logstash.log you’ll see:
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv6Addresses=true
After that your all set to use DNS names which resolves to ipv6 addresses.