logstash-forwarder

http://www.logstashbook.com/TheLogstashBook_sample.pdf

kafka

http://kafka.apache.org/07/quickstart.html

https://github.com/kafka-dev/kafka/tree/master/contrib/hadoop-consumer

Consider using Samza which is an abstraction over KAfka!
http://samza.apache.org/learn/documentation/latest/introduction/background.html

logstash & integration with Kafka, hadoop, ElasticSearch, Storm, etc

http://blog.mmlac.com/how-to-pre-process-logs-with-logstash/

https://support.shotgunsoftware.com/entries/23566933-Installing-Logstash-Central-Server-Logstash-Installation