background

for a fairly performance compare between pulsar and kafka，we need to modify kafka configuration to align with pulsar，consider kafka did‘nt have an WAL mechanism to make the data safety， the data will first save in memory then respond to the client，and the data in memory will async flush to the persistent disk，so the data have the chance to lost ， but pulsar use journal disk as WAL mechanism，so it can make sure every message written to pulsar will not lost。

configuration need to tune in kafka

• we want every message written to kafka will be persistent to disk to make sure data safety

  log.flush.interval.messages=1   # means every single message need to flush to disk,so it will impact the performance,need to tradeoff
  

  • some other parameter relate to the log flush(no need to change this time, just for reference)

• we want when produce message to kafka，at least write to two brokers（same as pulsar‘s 2 write quorum）

  min.insync.replicas=2 # make sure that when write data, there is 2 broker have recieved the data.
  

  Attention: this configuration need to combine with producer configure ack=all

conclusion

If we want to achieve the same safety level as pulsar provided in kafka, we can modify the above configuration in kafka broker and producer, but it will impact the performance of kafka.

• consider the test is running on customer’s datacenter, and the pulsar cluster, kafka cluster deployed on different infrastructure, so the test result will not give accurate performance number, just give a roughly trend in different scenario for reference

supplement some best practice

• important configurations when deploy cluster in production
• producer callback API
• consumer offset commit（sync vs async）

some related tech detail

• broker rebalance（preferred replica rebalance）
• consumer group rebalance(some core concepts)
• broker replication mechanism