SpringBoot整合Kafka实践

发表于 | 分类于 | 游览 次

飘了飘了，最近都没好好学习，加把劲吧😔

简介

Apache的Kafka™是一个分布式流平台(a distributed streaming platform)
有以下几个特性：（来自百度百科）

  1. 通过O(1)的磁盘数据结构提供消息的持久化，这种结构对于即使数以TB的消息存储也能够保持长时间的稳定性能。
  2. 高吞吐量 ：即使是非常普通的硬件Kafka也可以支持每秒数百万的消息。
  3. 支持通过Kafka服务器和消费机集群来分区消息。
  4. 支持Hadoop并行数据加载。

本文记录的是如何将kafka和SpringBoot整合使用，调用生产者Producer和消费者Consumer的API进行消息发送和消费，更加深入的点并未详细了解。

环境准备

代码整合之后，需要安装zookeeper作为注册中心，还有kafka服务。
我选择使用的是Centos 7服务器上进行环境配置的。

Zookeeper启动

首先去官网下载tar压缩包，选择合适的版本，然后进行解压

1
2
3
[root@VM_234_23_centos app]# wget 
[root@VM_234_23_centos app]# tar -zxvf zookeeper-3.4.10.tar
[root@VM_234_23_centos app]# cd zookeeper-3.4.10/

复制一份配置文件，文件名修改成zoo.cfg，由于是单机模式，所以使用默认的配置即可，不需要进行修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[root@VM_234_23_centos zookeeper-3.4.10]# cp conf/zoo_sample.cfg conf/zoo.cfg
[root@VM_234_23_centos zookeeper-3.4.10]# cat conf/zoo.cfg
# The number of milliseconds of each tick
# 服务器与客户端之间交互的基本事件单元（ms）
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
# 初始化可以连接的客户端数量
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
# 客户端与服务器之前请求和应答的时间间隔
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
# 客户端进行连接的端口号
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

启动zkServer服务

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@VM_234_23_centos zookeeper-3.4.10]# bin/zk
zkCleanup.sh  zkCli.cmd     zkCli.sh      zkEnv.cmd     zkEnv.sh      zkServer.cmd  zkServer.sh
[root@VM_234_23_centos zookeeper-3.4.10]# bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/app/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@VM_234_23_centos zookeeper-3.4.10]# jps
1122 QuorumPeerMain   ---- zk启动好的标志
1139 Jps
[root@VM_234_23_centos zookeeper-3.4.10]# bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/app/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: standalone --- 单机模式

Kafka安装

同样需要先去官网进行下载

1
[root@VM_234_23_centos app]# wget http://mirrors.hust.edu.cn/apache/kafka/1.1.0/kafka_2.12-1.1.0.tgz

接着解压和修改配置文件，同样也是按照默认即可（有个spring进行连接的坑，最后讲）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
[root@VM_234_23_centos app]# tar -zxvf kafka_2.12-1.1.0.tgz
[root@VM_234_23_centos app]# cd kafka_2.12-1.1.0/

root@VM_234_23_centos kafka_2.12-1.1.0]# cat config/server.properties
···
# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
# boker全局唯一的编号，不能重复
broker.id=0

############################# Socket Server Settings(Socket服务设置) #############################
# 用来监听链接的端口，producer和consumer将在此端口建立连接
#listeners=PLAINTEXT://:9092

#advertised.listeners=PLAINTEXT://your.host.name:9092

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
# 处理网络请求的线程数量
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
# 处理磁盘IO的线程数量
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
# 发送套接字的缓冲区大小
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
# 接收套接字缓冲区大小
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
# 请求套接字最大容量
socket.request.max.bytes=104857600
···
############################# Log Basics(打印设置) #############################

# A comma separated list of directories under which to store log files
# 消息存放的路径
log.dirs=/tmp/kafka-logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
# topic在当前broker上的分片数量
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
# 用来恢复和清理data下数据的线程数量
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
# 用于配置offset记录的topic的partition的副本个数
offsets.topic.replication.factor=1
# 事务主题的复制因子（设置更高以确保可用性）， 内部主题创建将失败，直到群集大小满足此复制因素要求
transaction.state.log.replication.factor=1
# 覆盖事务主题的min.insync.replicas配置
transaction.state.log.min.isr=1


############################# Log Retention Policy #############################

# The minimum age of a log file to be eligible for deletion due to age
# segment文件保留的最长时间，超时将被删除
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
# 日志文件中每个segment的大小
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
# 周期性检查文件大小的时间
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
# 链接zookeeper的端口号
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
# zookeeper链接超时的时间
zookeeper.connection.timeout.ms=6000

启动kafka

1
2
3
4
5
6
7
8
9
10
[root@VM_234_23_centos kafka]# bin/kafka-server-start.sh config/server.properties
[2018-07-11 17:10:24,315] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2018-07-11 17:10:25,645] INFO starting (kafka.server.KafkaServer)
[2018-07-11 17:10:25,647] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2018-07-11 17:10:25,693] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
···
[root@VM_234_23_centos ~]# jps
10002 Jps
1122 QuorumPeerMain
9673 Kafka --- 成功启动的标志

还有一些比较常用的客户端命令，例如查询topic列表，更多可以参考这里

代码进行整合

首先加入依赖，maven与gradle都可以

1
2
// https://mvnrepository.com/artifact/org.springframework.kafka/spring-kafka
compile group: 'org.springframework.kafka', name: 'spring-kafka', version: '1.3.5.RELEASE'

在application.properties进行配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#============== kafka ===================
# 指定kafka 代理地址，可以多个
spring.kafka.bootstrap-servers=localhost:9092

#=============== provider  =======================

spring.kafka.producer.retries=0
# 每次批量发送消息的数量
spring.kafka.producer.batch-size=16384
spring.kafka.producer.buffer-memory=33554432

# 指定消息key和消息体的编解码方式
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

#=============== consumer  =======================
# 指定默认消费者group id
spring.kafka.consumer.group-id=test-consumer-group

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=100

# 指定消息key和消息体的编解码方式
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer

发送消息&处理消息

发送消息需要使用到kafkaTemplate，还要指定发送的topic名称

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public class TestController {

    /**
     * kafka发送消息
     */
    @Autowired
    private KafkaTemplate kafkaTemplate;

    @RequestMapping("/test")
    public void testkafka(String message) {
        User user = new User();
        user.setName("hahaha");
        user.setAddress(message);
        // 第一个参数是topic名称
        kafkaTemplate.send("test", JSON.toJSONString(user));
    }

}

监听消息，进行处理，使用到@KafkaListener注解

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Service
public class TestKafkaMessageHandler{

    //  参数对应的是topic名称，可以监听多个topic，中间用逗号分开
    @KafkaListener(topics = {"test"})
    public void processMessage(String content) {
        try {
            User m = JSON.parseObject(content, User.class);
            System.out.println(m.toString());
        } catch (Exception e) {
            System.out.println(e.toString());
        }
    }

}

发送消息测试

正常启动之后，发送消息后，能够看到消息处理的结果

举个🌰，示例代码的执行结果如下所示：

Springboot配置简易化了许多，简单调用API就能进行消息的发送和消息处理，当然Kafka还有更强的功能和作用，可以深入去了解一下

遇到的坑

个人环境

Mac
IDEA 2017.03
SpringBoot 2.0.3.RELEASE
JDK 1.8
Centos 7 安装了 zookeeper-3.4.10 kafka_2.12

启动提示java.lang.NoSuchMethodError: org.springframework.util.Assert.state

kafka版本过高，springboot版本低不支持

原本打算引用最新的kafka依赖2.1.7.RELEASE，后来启动时报错，然后发现2.1.7版本的kafka需要依赖高版本的spring（5.0.6版本）

Group / Artifact Version Updates
org.springframework » spring-messaging 5.0.6.RELEASE 5.0.7.RELEASE
org.springframework » spring-context 5.0.6.RELEASE 5.0.7.RELEASE
org.springframework » spring-tx 5.0.6.RELEASE 5.0.7.RELEASE
········ ······· ·······

但是SpringBoot-2.0.3版本的spring使用的是spring-context是4.3.14，导致无法启动，所以需要调整依赖的版本号（我选择了kafka降级）

1
2
// https://mvnrepository.com/artifact/org.springframework.kafka/spring-kafka
compile group: 'org.springframework.kafka', name: 'spring-kafka', version: '1.3.5.RELEASE'

启动提示Connection to node 0 could not be established. Broker may not be available.

我是用Centos服务器搭建的kafka服务，然后使用本地环境进行连接kafka，消息订阅和发布失败，看到两篇博文，发现是解析服务器的hostname失败所导致的。

解决方法：修改kafka的server.properties配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@VM_234_23_centos kafka]# vim config/server.properties
····
############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
# 找到该行，进行修改
advertised.listeners=PLAINTEXT://个人服务器的ip:9092

然后重新启动解决问题

1
[root@VM_234_23_centos kafka]# bin/kafka-server-start.sh config/server.properties

参考资料

  1. Kafka配置文档
  2. 被版本更新坑到哭系列:SpringBoot整合Kafka
  3. 如何通过外部电脑访问kafka服务器
  4. SpringBoot Kafka 整合使用