Apache Kafka 元数据获取需要 40 秒以上的时间来读取和获取为什么？

问题描述 投票：0回答：1

我有三个集群，每个集群有 7 个主题和 2 个代理。每个主题都配置有一个分区。每个集群有 1 个生产者和 2 个消费者，消费者连接到其他集群。这导致总共有 6 个消费者（每个集群 2 个 × 3 个集群）和 3 个生产者。我还使用 

ACL authentication
 设置了简单的
kafka.security.authorizer.AclAuthorizer.

我面临的问题是其中一个集群的问题。当我尝试使用以下代码获取元数据时，

List<PartitionInfo> partitions = consumer.partitionsFor(topic);

花了40多秒，我无法确定原因。完全相同的代码和配置在其他集群上完美运行，只需 

300-500ms
。所有操作（包括代理和 ZooKeeper 设置）均使用 Java API 进行管理。

我不明白为什么这种延迟只发生在这个特定的集群上。 任何有助于确定可能导致此问题的原因的帮助将不胜感激。

为了帮助排除故障，请在回答之前考虑以下事项：配置、代码和整体设置在运行 

Linux
的两个集群上完美运行，而问题仅出现在 
Windows
上。这表明问题可能不在于代码或设置，而在于主机环境。

在我的调查过程中，我确保以下潜在原因不负有责任：

  1. 缓慢的元数据获取：我相信这是问题的根源，因为在日志记录期间，需要太多时间来获取元数据并获取以下行（
    2024-11-23 20:16:54 INFO  org.apache.kafka.clients.Metadata - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] Cluster ID: bss9f46kS_qHYiScR08meA
    ）。消费者需要有关主题和分区的元数据来确定获取位置。我还观察到元数据获取很慢，并且使用 Jprofiler 我观察到 updateFetchPositions 出现延迟，但我不知道如何解决它
  2. 偏移查找延迟：我验证没有延迟，一切都运行顺利。
  3. 重新平衡开销：日志确认没有发生重新平衡过程。
  4. 大量分区：在此设置中这不是问题，因为我的主题数量少于 10。
  5. 经纪商节流或负载：我相信这不是问题，因为只有两个消费者连接。
  6. DNS 问题：这不太可能，因为 bootstrap.servers 配置使用 IP 地址而不是主机名。
  7. Windows Defender：我已从防病毒软件中排除了 JetBrains 文件夹和 Kafka、zookeeper 路径，这可能会导致不稳定，但我不知道背后是否有任何防火墙导致延迟。

对于可能导致该问题的原因，您还有其他建议或想法吗？

这是消费者配置：

   Properties props = new Properties();
            props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, ip + ":" + KafkaConfiguration.KAFKA_PORT);
            props.put(ConsumerConfig.GROUP_ID_CONFIG, groupID);
            props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, groupID);
            props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
            props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer.class.getName());
            props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, "13107200");
            props.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, "536870912");
            props.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, "-1");
            props.put(ConsumerConfig.SEND_BUFFER_CONFIG, "-1");
            props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
            props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, "1");
            props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "10");
            props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "60000");
            props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "600000");
            props.put(ConsumerConfig.AUTO_INCLUDE_JMX_REPORTER_CONFIG, "false");
            props.put(ConsumerConfig.ENABLE_METRICS_PUSH_CONFIG, "FALSE");
            props.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, "1000");
            props.put(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG, "300");
            props.put(ConsumerConfig.RETRY_BACKOFF_MAX_MS_CONFIG, "7000");
            props.put(ConsumerConfig.RECONNECT_BACKOFF_MS_CONFIG, "300");
            props.put(ConsumerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, "7000");
            props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "11000");
            props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "8000");
            props.put(ConsumerConfig.SOCKET_CONNECTION_SETUP_TIMEOUT_MS_CONFIG, "20000");
            props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
            props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, "org.apache.kafka.clients.consumer.StickyAssignor");
            props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
            props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
            props.put(SaslConfigs.SASL_JAAS_CONFIG, "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"" + "consumer" + "-" + position + "-" + current_ip + "\" password=\"consumer-secret\";");

这是生产者配置：

   props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, KafkaConfiguration.KAFKA_HOST + ":" + KafkaConfiguration.KAFKA_PORT);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, KafkaConfiguration.PRODUCER_GROUP_ID + "-" + KafkaConfiguration.KAFKA_HOST);
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());
    props.put(ProducerConfig.ACKS_CONFIG, "1");
    props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
    props.put(ProducerConfig.BATCH_SIZE_CONFIG, "56384");
    props.put(ProducerConfig.LINGER_MS_CONFIG, "0");
    props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "1000");
    props.put(ProducerConfig.METADATA_MAX_AGE_CONFIG, "1000");
    props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, "300");
    props.put(ProducerConfig.RETRY_BACKOFF_MAX_MS_CONFIG, "7000");
    props.put(ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG, "300");
    props.put(ProducerConfig.RECONNECT_BACKOFF_MAX_MS_CONFIG, "7000");
    props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, "134217728");
    props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, "2024000000");
    props.put(ProducerConfig.RECEIVE_BUFFER_CONFIG, "-1");
    props.put(ProducerConfig.SEND_BUFFER_CONFIG, "-1");
    props.put(ProducerConfig.AUTO_INCLUDE_JMX_REPORTER_CONFIG, "false");
    props.put(ProducerConfig.ENABLE_METRICS_PUSH_CONFIG, "false");
    props.put(ProducerConfig.RETRIES_CONFIG, "0");
    props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, "600000");
    props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_PLAINTEXT");
    props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
    props.put(SaslConfigs.SASL_JAAS_CONFIG, "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"producer\" password=\"producer-secret\";");
    props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, RoundRobinPartitioner.class.getName());
    props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);

最后这是 kafka 代理配置：

kafkaProps.setProperty("broker.id", "0");
kafkaProps.setProperty("log.dirs", zkDir);
kafkaProps.setProperty("zookeeper.connection.timeout.ms", "15000");
kafkaProps.setProperty("zookeeper.max.in.flight.requests", "10");
//kafkaProps.setProperty("zookeeper.set.acl","true");
kafkaProps.setProperty("zookeeper.connect", KafkaConfiguration.ZOOKEEPER_HOST + ":" + KafkaConfiguration.ZOOKEEPER_PORT);
kafkaProps.setProperty("offsets.topic.replication.factor", "1");
kafkaProps.setProperty("delete.topic.enable", "true");
kafkaProps.setProperty("num.network.threads", String.valueOf(this.cores));
kafkaProps.setProperty("num.io.threads", String.valueOf(this.cores * 2));
kafkaProps.setProperty("auto.delete.topics.enable", "true");
kafkaProps.setProperty("auto.create.topics.enable", "false");
kafkaProps.setProperty("auto.leader.rebalance.enable", "false");
kafkaProps.setProperty("background.threads", String.valueOf(this.cores / 2));
kafkaProps.setProperty("metrics.reporters", "");
kafkaProps.setProperty("default.api.timeout.ms", "15000");
kafkaProps.setProperty("log.flush.interval.ms", "15000");
kafkaProps.setProperty("max.connections.per.ip,", "250");
kafkaProps.setProperty("remote.fetch.max.wait.ms,", "10");
kafkaProps.setProperty("message.max.bytes", "2024000000");
kafkaProps.setProperty("socket.request.max.bytes", "2024000000");
kafkaProps.setProperty("fetch.max.bytes", "536870912");
kafkaProps.setProperty("group.consumer.heartbeat.interval.ms", "15000");
kafkaProps.setProperty("group.consumer.min.heartbeat.interval.ms", "15000");
kafkaProps.setProperty("group.consumer.max.heartbeat.interval.ms", "45000");
kafkaProps.setProperty("group.consumer.max.session.timeout.ms", "65000");
kafkaProps.setProperty("group.consumer.session.timeout.ms", "60000");
kafkaProps.setProperty("replica.fetch.wait.max.ms", "10");
kafkaProps.setProperty("replica.fetch.max.bytes", "13107200");
kafkaProps.setProperty("auto.include.jmx.reporter", "false");
kafkaProps.setProperty("request.timeout.ms", "25");
kafkaProps.setProperty("socket.receive.buffer.bytes", "-1");
kafkaProps.setProperty("compression.type", "lz4");
kafkaProps.setProperty("socket.send.buffer.bytes", "-1");
kafkaProps.setProperty("socket.connection.setup.timeout.ms", "15000");
kafkaProps.setProperty("group.initial.rebalance.delay.ms", "0");
kafkaProps.setProperty("metadata.max.idle.interval.ms", "1500");
kafkaProps.setProperty("metadata.max.retention.ms", "60480000");
kafkaProps.setProperty("session.timeout.ms", "9000");
kafkaProps.setProperty("heartbeat.interval.ms", "14000");
kafkaProps.setProperty("producer.id.expiration.ms", String.valueOf(ConsensusConfiguration.EPOCH_TRANSITION * 2 * ConsensusConfiguration.CONSENSUS_TIMER));
kafkaProps.setProperty("log.retention.ms", String.valueOf(ConsensusConfiguration.EPOCH_TRANSITION * 2 * ConsensusConfiguration.CONSENSUS_TIMER));
kafkaProps.setProperty("authorizer.class.name", "kafka.security.authorizer.AclAuthorizer");
kafkaProps.setProperty("listeners", "SASL_PLAINTEXT://" + KafkaConfiguration.KAFKA_HOST + ":" + KafkaConfiguration.KAFKA_PORT);
kafkaProps.setProperty("advertised.listeners", "SASL_PLAINTEXT://" + KafkaConfiguration.KAFKA_HOST + ":" + KafkaConfiguration.KAFKA_PORT);
kafkaProps.setProperty("security.inter.broker.protocol", "SASL_PLAINTEXT");
kafkaProps.setProperty("sasl.mechanism.inter.broker.protocol", "PLAIN");
kafkaProps.setProperty("sasl.enabled.mechanisms", "PLAIN");
kafkaProps.setProperty("allow.everyone.if.no.acl.found", "false");
kafkaProps.setProperty("listener.name.sasl_plaintext.plain.sasl.jaas.config", getJaasConfigString());
kafkaProps.setProperty("super.users", getSuperUsers());

这是来自有问题的 Windows 机器的 kafka-log 示例：

2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from __consumer_offsets-13 for epoch 0
2024-11-23 20:16:46 INFO  k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Elected as the group coordinator for partition 28 in epoch 0
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from __consumer_offsets-28 for epoch 0
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-3 in 4 milliseconds for epoch 0, of which 0 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-18 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-41 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-10 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-33 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-48 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-19 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-34 in 6 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-4 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-11 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-26 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-49 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-39 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-9 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-24 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-31 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-46 in 5 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-1 in 6 milliseconds for epoch 0, of which 5 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-16 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-2 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-25 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-40 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-47 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-17 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-32 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-37 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-7 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-22 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-29 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-44 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-14 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-23 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-38 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-8 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-45 in 6 milliseconds for epoch 0, of which 6 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-15 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-30 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-0 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-35 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-5 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-20 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-27 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-42 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-12 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-21 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-36 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-6 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-43 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-13 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.c.group.GroupMetadataManager - [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-28 in 7 milliseconds for epoch 0, of which 7 milliseconds was spent in the scheduler.
2024-11-23 20:16:46 INFO  k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Static member with groupInstanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 and unknown member id joins group CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 in Empty state. Created a new member id CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116-58f4af9b-d6c1-4998-ad87-04561bcdeab0 for this member and add to the group.
2024-11-23 20:16:46 INFO  k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Preparing to rebalance group CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 in state PreparingRebalance with old generation 0 (__consumer_offsets-7) (reason: Adding new member CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116-58f4af9b-d6c1-4998-ad87-04561bcdeab0 with group instance id Some(CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116); client reason: )
2024-11-23 20:16:46 INFO  k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Stabilized group CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 generation 1 (__consumer_offsets-7) with 1 members
2024-11-23 20:16:46 INFO  k.coordinator.group.GroupCoordinator - [GroupCoordinator 0]: Assignment received from leader CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116-58f4af9b-d6c1-4998-ad87-04561bcdeab0 for group CONSUMER_PRIVATE_GROUP_ID-192.168.1.106-192.168.1.116 for generation 1. The group has 1 members, 1 of which are static.
2024-11-23 20:16:54 INFO  org.apache.kafka.clients.Metadata - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] Cluster ID: bss9f46kS_qHYiScR08meA
2024-11-23 20:16:54 INFO  org.apache.kafka.clients.Metadata - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106] Cluster ID: GYgEyPiCTzGEuSJvs9r9xA
2024-11-23 20:17:04 INFO  o.a.k.c.c.i.ClassicKafkaConsumer - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] Subscribed to topic(s): PREPARE_PHASE, COMMITTEE_PHASE, ANNOUNCE_PHASE, DISPERSE_PHASE2
2024-11-23 20:17:04 INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] Discovered group coordinator 192.168.1.115:9092 (id: 2147483647 rack: null)
2024-11-23 20:17:04 INFO  o.a.k.c.c.i.ClassicKafkaConsumer - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106] Subscribed to topic(s): PREPARE_PHASE, COMMITTEE_PHASE, ANNOUNCE_PHASE, DISPERSE_PHASE2
2024-11-23 20:17:04 INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106] Discovered group coordinator 192.168.1.116:9092 (id: 2147483647 rack: null)
2024-11-23 20:17:13 INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.115-192.168.1.106] (Re-)joining group
Consumer 192.168.1.115 2 started and subscribed to topics: [PREPARE_PHASE, COMMITTEE_PHASE, ANNOUNCE_PHASE, DISPERSE_PHASE2]
2024-11-23 20:17:13 INFO  o.a.k.c.c.i.ConsumerCoordinator - [Consumer instanceId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, clientId=consumer-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106-CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106, groupId=CONSUMER_PRIVATE_GROUP_ID-192.168.1.116-192.168.1.106] (Re-)joining group
Consumer 192.168.1.116 1 started and subscribed to topics: [PREPARE_PHASE, COMMITTEE_PHASE, ANNOUNCE_PHASE, DISPERSE_PHASE2]
All tasks completed
java scala performance apache-kafka publish-subscribe
1个回答
0
投票

终于，我找到了问题的答案。很难弄清楚到底发生了什么。我很挣扎，花费了我数百个小时的搜索和调试。问题不在于 Kafka 配置或 API，而是网络问题。具体来说，Ubuntu VM IP 地址的主机名解析是本地的，因此 Windows 无法从 /etc/hosts 文件解析正确的主机名。 Kafka 处理了异常，但是该方法一直运行到超时，导致获取主题元数据出现延迟。

以下是快速解决问题的详细步骤：

确保 ipv4 已启用或以编程方式使用它

 -Djava.net.preferIPv4Stack=true

更改主机名的步骤：

打开终端：按 Ctrl + Alt + T 打开终端。检查当前主机名：要查看当前主机名，请键入：

 hostnamectl

这将显示当前主机名以及其他系统信息。设置新主机名：要更改主机名，请使用以下命令：

sudo hostnamectl set-hostname new-hostname

将 new-hostname 替换为所需的主机名。

验证更改：要确认更改，请运行：

sh hostnamectl

您应该会看到“静态主机名”部分下列出的新主机名。

转到 Windows 并执行以下操作： 打开 Hosts 文件：在记事本中，单击文件 > 打开。

导航至

C:\Windows\System32\drivers\etc
。

选择主机文件。如果您没有看到它，请确保将文件类型过滤器更改为右下角的“所有文件”1。

添加 IP 地址和主机名：

在文件末尾添加一个新行，其中包含虚拟机的 IP 地址，后跟要映射的主机名。例如：

192.168.1.100    myvmhostname

按 Ctrl + S 或单击“文件”>“保存”保存更改

然后在终端中运行以下命令：

ipconfig /flushdns
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.