Ini 容器未初始化 Kafka 来创建主题 Helm 图表

Question

我遇到初始化容器无法创建主题的问题。 Kafka pod 可以在没有 init 容器的情况下启动，但是添加后会出现以下错误。我使用了 localhost:9092，它仍然产生相同的结果。

kubectl logs datapipeline-kafka-5c45bfcdfb-sfbzq    -c init-kafka
[2023-08-11 16:19:08,945] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (my-kafka-service/10.101.182.60:9092) could not be established. Broker may not be available. 

values.yaml：

kafka:
  serviceName: my-kafka-service
  kafkaPort: 9092
  topicsToCreate:
    - stream-a
  dependsOn:
    - name: zookeeper
  fullnameOverride: kafka
  image:
    repository: confluentinc/cp-enterprise-kafka
    tag: 7.4.0
  brokerId: 1
  zookeeperConnect: "{{ .Values.zookeeper.serviceName }}:2181"  # Use zookeeperServiceName
  interBrokerListenerName: PLAINTEXT
  listenerSecurityProtocolMap:
    PLAINTEXT: PLAINTEXT
    PLAINTEXT_HOST: PLAINTEXT
  advertisedListeners: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092"
  offsetTopicReplicationFactor: 1
  groupInitialRebalanceDelayMs: 100
  autoCreateTopicsEnable: "true"
  metricsReporterBootstrapServers: kafka:29092
  metricsReporterZookeeperConnect: "{{ .Values.zookeeper.serviceName }}:2181"  # Use zookeeperServiceName
  metricsReporterTopicReplicas: 1
  metricsEnable: 'true'
  supportCustomerId: 'anonymous'
  metricReporters: io.confluent.metrics.reporter.ConfluentMetricsReporter

pvc:
  enabled: true
  name: "{{ .Release.Name }}-kafka-pvc"
  accessMode: ReadWriteOnce
  storage: 1Gi

部署和服务是：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-kafka
  labels:
    app: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec: 
      containers:
        - name: kafka
          image: "{{ .Values.kafka.image.repository }}:{{ .Values.kafka.image.tag }}"
          env:
            - name: KAFKA_BROKER_ID
              value: "{{ .Values.kafka.brokerId }}"
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: "{{ .Values.zookeeper.serviceName }}:{{ .Values.zookeeper.clientPort }}"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: "{{ .Values.kafka.interBrokerListenerName }}"
            - name: KAFKA_ADVERTISED_LISTENERS
              value: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092"
            - name: KAFKA_METRIC_REPORTERS
              value: "{{ .Values.kafka.metricReporters }}"
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "{{ .Values.kafka.offsetTopicReplicationFactor }}"
            - name: KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS
              value: "{{ .Values.kafka.groupInitialRebalanceDelayMs }}"
            - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
              value: "{{ .Values.kafka.autoCreateTopicsEnable }}"
            - name: CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS
              value: "{{ .Values.kafka.serviceName }}:{{ .Values.kafka.kafkaPort }}" 
            - name: CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT
              value: "{{ .Values.zookeeper.serviceName }}:{{ .Values.zookeeper.clientPort }}"
            - name: CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS
              value: "{{ .Values.kafka.metricsReporterTopicReplicas }}"
            - name: CONFLUENT_METRICS_ENABLE
              value: "{{ .Values.kafka.metricsEnable }}"
            - name: CONFLUENT_SUPPORT_CUSTOMER_ID
              value: "{{ .Values.kafka.supportCustomerId }}"
          ports:
            - containerPort: 9092
            - containerPort: 9093
      initContainers:
        - name: init-kafka
          image: confluentinc/cp-kafka:7.4.0  # Use the appropriate Kafka image
          command: ["/bin/sh", "-c", "kafka-topics --bootstrap-server my-kafka-service:9092 --create --topic stream-a --partitions 1 --replication-factor 1"]
          volumeMounts:
            - name: kafka-storage
              mountPath: /var/lib/kafka/data
          resources:
            requests:
              memory: "64Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
              cpu: "100m"
          restartPolicy: OnFailure
      volumes:
        - name: kafka-storage
          persistentVolumeClaim:
            claimName: "{{ .Release.Name }}-kafka-pvc"  
---
apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.kafka.serviceName }} # Use the provided kafkaServiceName from values.yaml
spec:
  selector:
    app: kafka
  ports:
    - protocol: TCP
      port: 9092
      targetPort: 9092

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "{{ .Release.Name }}-kafka-pvc"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi  # Adjust the storage size as needed

我希望在 kafka 容器的开头创建主题。 init 容器将如何从 kafka 的 create-topics 开始。

Answer 1

为什么要推出自己的 Kafka 图表？这是一个已解决的问题。从这里开始 - http://strimzi.io 查看 KafkaTopic CRD 上的文档；当操作员工作得更好时，你不需要一个初始化容器