下面是3台电脑的配置 主机名 IP地址 硬件配置 kafka-1 192.168.1.42 4CPU、4G内存、128G存储 kafka-2 192.168.1.41 4CPU、4G内存、128G存储 kafka ,在本例中对应关系如下 主机名 IP地址 zookeeper myid kafka-1 192.168.1.42 server.1 1 kafka-2 192.168.1.41 server.2 2 kafka /data/myid #kafka-1主机myid 在kafka-2中执行下面命令 echo “2” > /usr/local/zookeeper/data/myid #kafka-2主机myid 在kafka -3中执行下面命令 echo “3” > /usr/local/zookeeper/data/myid #kafka-3主机myid 在每台电脑上启动zookeeper /usr/local/zookeeper /192.168.1.41:9092 #新增 zookeeper.connect=192.168.1.41:2181,192.168.1.42:2181,192.168.1.47:2181 #新增 在kafka
Kafka 共用统一虚拟机; 三台虚拟机信息: hostname:kafka-1,IP:10.0.0.1,ID:1 hostname:kafka-2,IP:10.0.0.2,ID:2 hostname:kafka $ cat /etc/hosts ...... 10.0.0.1 kafka-1 10.0.0.2 kafka-2 10.0.0.3 kafka-3 1. :3888 #server.3=zookeeper3:2888:3888 server.1=kafka-1:2888:3888 server.2=kafka-2:2888:3888 server.3=kafka myid # on kafka-1 1 $ cat /etc/zookeeper/conf/myid # on kafka-2 2 $ cat /etc/zookeeper/conf/myid # on kafka kafka.security.auth.SimpleAclAuthorizer allow.everyone.if.no.acl.found=true zookeeper.connect=kafka-1:2181,kafka-2:2181,kafka
kafka.bootstrap.servers=ip1:port,ip2:port,ip3:port \ --kafka_topic=testdynamic_%{database}_%{table} 测试脚本kafka 静态传参和动态传参 4、静态传参是--kafka_topic=参数值,手动指定的,maxwell会将变化数据发送到指定的topic中 5、动态传参 5.1 动态传参没有前缀 测试脚本kafka
=192.168.196.128:2181,192.168.196.131:2181,192.168.196.132:2181 offsets.topic.replication.factor=3 在kafka
尤其是key的维度比较高、value状态比较大的 object StateOperator { private val brokers = "kafka-1:9092,kafka-2:9092,kafka
部署 2.1 初始化环境 初始化系统,关闭防火墙修改主机名与ip名称 名称 HOSTNAME IP 1 kafka-1 172.17.10.207 2 kafka-2 172.17.10.208 3 kafka
org.apache.flume.sink.kafka.KafkaSink a1.sinks.sink1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka org.apache.flume.channel.kafka.KafkaChannel a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9092,kafka-2:9092,kafka org.apache.flume.channel.kafka.KafkaChannel a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka
org.apache.flume.channel.kafka.KafkaChannel a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9092,kafka-2:9092,kafka
name: kafka-service-3 app: kafka-service-3 spec: containers: - name: kafka
172.16.2.5 10.0.0.40 kafka-2 1/1 Running 0 1h 172.16.1.5 10.0.0.45 kafka
logstash.conf input { kafka { type => "kafka" bootstrap_servers => "kafka-1:9092,kafka-2:9092,kafka
192.168.110.93 kafka-1 broker,controller 2 192.168.110.94 kafka-2 broker,controller 3 192.168.110.94 kafka