使用远程 Kafka 集群在本地设置 KSQLDB 的问题

Question

我正在尝试在本地设置 KSQLDB 并将其连接到在 upstash 上运行的远程 kafka 服务器。但是，配置凭据后，我无法连接到 kafka 服务器。

我按照官方文档中给出的说明使用docker安装了KSQLDB。 （https://docs.ksqldb.io/en/latest/operate-and-deploy/installation/installing/）

为了连接到外部 kafka 服务器，我创建了一个 .properties 文件并将其作为卷安装在 ksql-server 容器中。

下面是我正在使用的 docker-compose.yml 文件。

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:${CP_VERSION}
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-enterprise-kafka:${CP_VERSION}
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100

  schema-registry:
    image: confluentinc/cp-schema-registry:${CP_VERSION}
    depends_on:
      - zookeeper
      - kafka
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092


  primary-ksqldb-server:
    image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-server:${KSQL_VERSION}
    hostname: primary-ksqldb-server
    container_name: primary-ksqldb-server
    depends_on:
      - kafka
      - schema-registry
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: kafka:9092
      KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
    volumes:
      - ./ksql-server.properties:/etc/ksqldb/ksql-server.properties

  additional-ksqldb-server:
    image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-server:${KSQL_VERSION}
    hostname: additional-ksqldb-server
    depends_on:
      - primary-ksqldb-server
    ports:
      - "8090"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8090
      KSQL_BOOTSTRAP_SERVERS: kafka:9092
      KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
    volumes:
      - ./ksql-server.properties:/etc/ksqldb/ksql-server.properties

  # Access the cli by running:
  # > docker-compose exec ksqldb-cli  ksql http://primary-ksqldb-server:8088
  ksqldb-cli:
    image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-cli:${KSQL_VERSION}
    container_name: ksqldb-cli
    depends_on:
      - primary-ksqldb-server
    entrypoint: /bin/sh
    tty: true

下面是我的 ksql-server.properties 文件。

bootstrap.servers=sensible-condor-6169-us1-kafka.upstash.io:9092
sasl.mechanism=SCRAM-SHA-256
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="***" \
  password="***";

现在，当我打开 KSQLDB CLI 时，出现以下错误。

WARNING: Could not identify server version.
         Non-matching CLI and server versions may lead to unexpected errors.

Server Status: <unknown>

Having trouble? Type 'help' (case-insensitive) for a rundown of how things work!


*************************************ERROR**************************************
Remote server at http://primary-ksqldb-server:8088 does not appear to be a valid
KSQL server. Please ensure that the URL provided is for an active KSQL server.

The server responded with the following error:
Error issuing GET to KSQL server. path:/info
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException:
        Connection refused: primary-ksqldb-server/172.24.0.5:8088
Caused by: Could not connect to the server. Please check the server details are
        correct and that the server is running.
********************************************************************************

容器日志中也没有任何内容。

Answer 1

问题是您将主 ksqldb 服务器引用到 docker-compose.yml 中的新 Kafka 实例。如果您已经有一个现有的远程 Kafka 集群，您所需要做的就是在 docker-compose.yml 中引用它，这样

KSQL_BOOTSTRAP_SERVERS: sensible-condor-6169-us1-kafka.upstash.io:9092

您的 docker-compose.yml 应该如下所示（请仔细检查我的缩进）

---
version: '2'
services:
  primary-ksqldb-server:
   image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-server:${KSQL_VERSION}
   hostname: primary-ksqldb-server
   container_name: primary-ksqldb-server
   depends_on:
    - schema-registry
   ports:
    - "8088:8088"
   environment:
     KSQL_LISTENERS: http://0.0.0.0:8088
     KSQL_BOOTSTRAP_SERVERS: sensible-condor-6169-us1-kafka.upstash.io:9092 #consider checking if it is reachable over this name/port
     KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
     KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
     KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
   volumes:
    - ./ksql-server.properties:/etc/ksqldb/ksql-server.properties

  additional-ksqldb-server:
   image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-server:${KSQL_VERSION}
   hostname: additional-ksqldb-server
   depends_on:
    - primary-ksqldb-server
   ports:
    - "8090"
   environment:
    KSQL_LISTENERS: http://0.0.0.0:8090
    KSQL_BOOTSTRAP_SERVERS: sensible-condor-6169-us1-kafka.upstash.io:9092 #consider checking if it is reachable over this name/port
    KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
   volumes:
    - ./ksql-server.properties:/etc/ksqldb/ksql-server.properties

  ksqldb-cli:
   image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-cli:${KSQL_VERSION}
   container_name: ksqldb-cli
   depends_on:
     - primary-ksqldb-server
   entrypoint: /bin/sh
   tty: true