排查logstash3.4升级到5.0版本后kafka不兼容问题
参考文档：
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/CHANGELOG.md /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/DEVELOPER.md /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/README.md.md /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/lib/logstash/inputs/kafka.rb /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.0.4/lib/logstash/outputs/kafka.rb缘由：
之前对ELKB环境从2.4版本升级到最新的5.0稳定版本，主要升级步骤可以参考，后来发现kafka集群运行报错，现在把排查过程记录如下，仅供参考
之前环境：
logstash3.4
logstash-input-kafka-2.0.9
logstash-output-kafka-2.0.5
kafka_2.10-0.8.2.2.tgz
升级后环境：
logstash6.0
logstash-input-kafka-2.0.9
logstash-output-kafka-2.0.5
报错信息：
[2016-11-16T14:35:44,739][ERROR][logstash.inputs.kafka ] Unknown setting 'zk_connect' for kafka [2016-11-16T14:35:44,741][ERROR][logstash.inputs.kafka ] Unknown setting 'topic_id' for kafka [2016-11-16T14:35:44,741][ERROR][logstash.inputs.kafka ] Unknown setting 'reset_beginning' for kafka实施步骤：
1，根据错误查看程序哪里报错
grep "Unknown setting" /usr/share/logstash/ -R /usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb: self.logger.error("Unknown setting '#{name}' for #{@plugin_name}")2，查看程序相关代码，发现需要查看plugins的config定义文件等
def validate_check_invalid_parameter_names(params) invalid_params = params.keys # Filter out parameters that match regexp keys. # These are defined in plugins like this: # config /foo.*/ => ... @config.each_key do |config_key| if config_key.is_a?(Regexp) invalid_params.reject! { |k| k =~ config_key } elsif config_key.is_a?(String) invalid_params.reject! { |k| k == config_key } end end if invalid_params.size > 0 invalid_params.each do |name| self.logger.error("Unknown setting '#{name}' for #{@plugin_name}") end return false end # if invalid_params.size > 0 return true end # def validate_check_invalid_parameter_names3，进入插件总目录查看具体信息
cd /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5
发现重点查看如下文件
grep config ./* -R |awk '{print $1}' |uniq ./CHANGELOG.md: ./DEVELOPER.md:See ./lib/logstash/inputs/kafka.rb:# ./lib/logstash/inputs/kafka.rb: ./README.md:- Binary1）首先看CHANGELOG.md，就有发现logstash-input-3.0.0.beta1开始就不在向后兼容，且剔除了jruby-kafka，注意这里有个坑2）会讲到，4.0.0版本说开始支持kafka 0.9，5.0.0又说开始
支持0.10切不向后兼容，这破坏性更新也是够了。看来问题找到了我的kafka版本是kafka_2.10-0.8.2.2.tgz，kafka版本不兼容导致的。
CHANGELOG.md部分文档如下：
## 5.0.4 - Update to Kafka version 0.10.0.1 for bug fixes ## 5.0.0 - Support for Kafka 0.10 which is not backward compatible with 0.9 broker. ## 4.0.0 - Republish all the gems under jruby. - Update the plugin to the version 2.0 of the plugin api, this change is required for Logstash 5.0 compatibility. See https://github.com/elastic/logstash/issues/5141 - Support for Kafka 0.9 for LS 5.x ## 3.0.0.beta1 - Refactor to use new Java based consumer, bypassing jruby-kafka - Breaking: Change configuration to match Kafka's configuration. This version is not backward compatible郑重声明：本文版权归原作者所有，转载文章仅为传播更多信息之目的，如作者信息标记有误，请第一时间联系我们修改或删除，多谢。
