使用 debezium 的 Mariadb docker 容器连接失败

问题描述 投票：0回答：1

所以我尝试使用 debezium 创建 mariadb docker 容器连接

起初连接运行良好，但两分钟后我尝试再次检查状态，但出现错误。创建 postgre docker 容器连接时这不是问题，有人知道如何解决这个问题吗？

这是我收到的错误。

C:\Users\User>curl -X POST -H "Content-Type: application/json" --data @D:\\new_kafka\\connectors\\mariadb.json http://localhost:8083/connectors
{"name":"exampledb2-connector","config":{"connector.class":"io.debezium.connector.mysql.MySqlConnector","tasks.max":"1","database.hostname":"mariadb","database.port":"3306","database.user":"root","database.password":"root_password","database.server.id":"184054","database.server.name":"mariadb","database.whitelist":"exampledb","database.history.kafka.bootstrap.servers":"localhost:9092","database.history.kafka.topic":"dbhistory.exampledb","include.schema.changes":"true","database.connectionTimeZone":"UTC","name":"exampledb2-connector"},"tasks":[],"type":"source"}
C:\Users\User>curl -X GET http://localhost:8083/connectors/exampledb2-connector/status
{"name":"exampledb2-connector","connector":{"state":"RUNNING","worker_id":"172.21.0.8:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"172.21.0.8:8083"}],"type":"source"}
C:\Users\User>curl -X GET http://localhost:8083/connectors/exampledb2-connector/status
{"name":"exampledb2-connector","connector":{"state":"RUNNING","worker_id":"172.21.0.8:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"172.21.0.8:8083","trace":"org.apache.kafka.connect.errors.ConnectException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata\n\tat io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:287)\n\tat io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:106)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:232)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata\n"}],"type":"source"}

以下是docker日志错误

debezium 容器日志：

2024-07-11 09:49:07 debezium-1         | 2024-07-11 02:49:07,664 WARN   MySQL|mariadb|task  [Producer clientId=exampledb2-connector-dbhistory] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected   [org.apache.kafka.clients.NetworkClient]
2024-07-11 09:49:08 debezium-1         | 2024-07-11 02:49:08,819 WARN   MySQL|mariadb|task  [Producer clientId=exampledb2-connector-dbhistory] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]

MariaDB 容器日志

2024-07-12 11:05:26 2024-07-12 04:05:26+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.4.34+maria~ubu2004 started.
2024-07-12 11:05:27 2024-07-12 04:05:27+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-07-12 11:05:27 2024-07-12 04:05:27+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.4.34+maria~ubu2004 started.
2024-07-12 11:05:28 2024-07-12 04:05:28+00:00 [Note] [Entrypoint]: Initializing database files
2024-07-12 11:05:36 
2024-07-12 11:05:36 
2024-07-12 11:05:36 PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
2024-07-12 11:05:36 To do so, start the server, then issue the following command:
2024-07-12 11:05:36 
2024-07-12 11:05:36 '/usr/bin/mysql_secure_installation'
2024-07-12 11:05:36 
2024-07-12 11:05:36 which will also give you the option of removing the test
2024-07-12 11:05:36 databases and anonymous user created by default.  This is
2024-07-12 11:05:36 strongly recommended for production servers.
2024-07-12 11:05:36 
2024-07-12 11:05:36 See the MariaDB Knowledgebase at http://mariadb.com/kb
2024-07-12 11:05:36 
2024-07-12 11:05:36 Please report any problems at http://mariadb.org/jira
2024-07-12 11:05:36 
2024-07-12 11:05:36 The latest information about MariaDB is available at http://mariadb.org/.
2024-07-12 11:05:36 
2024-07-12 11:05:36 Consider joining MariaDB's strong and vibrant community:
2024-07-12 11:05:36 https://mariadb.org/get-involved/
2024-07-12 11:05:36 
2024-07-12 11:05:36 2024-07-12 04:05:36+00:00 [Note] [Entrypoint]: Database files initialized
2024-07-12 11:05:36 2024-07-12 04:05:36+00:00 [Note] [Entrypoint]: Starting temporary server
2024-07-12 11:05:36 2024-07-12 04:05:36+00:00 [Note] [Entrypoint]: Waiting for server startup
2024-07-12 11:05:37 2024-07-12 04:05:37+00:00 [Note] [Entrypoint]: Temporary server started.
2024-07-12 11:05:49 2024-07-12 04:05:49+00:00 [Note] [Entrypoint]: Creating database exampledb
2024-07-12 11:05:49 2024-07-12 04:05:49+00:00 [Note] [Entrypoint]: Creating user user
2024-07-12 11:05:49 2024-07-12 04:05:49+00:00 [Note] [Entrypoint]: Giving user user access to schema exampledb
2024-07-12 11:05:49 2024-07-12 04:05:49+00:00 [Note] [Entrypoint]: Securing system users (equivalent to running mysql_secure_installation)
2024-07-12 11:05:49 
2024-07-12 11:05:49 2024-07-12 04:05:49+00:00 [Note] [Entrypoint]: Stopping temporary server
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] Starting MariaDB 10.4.34-MariaDB-1:10.4.34+maria~ubu2004 source revision 16394f1aa1b4097f897b8ab01ea2064726cca059 as process 118
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Using Linux native AIO
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Uses event mutexes
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Number of pools: 1
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Using SSE2 crc32 instructions
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Completed initialization of buffer pool
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] InnoDB: 10.4.34 started; log sequence number 60943; transaction id 20
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Warning] 'user' entry 'root@07acf519475e' ignored in --skip-name-resolve mode.
2024-07-12 11:05:36 2024-07-12  4:05:36 0 [Warning] 'proxies_priv' entry '@% root@07acf519475e' ignored in --skip-name-resolve mode.
2024-07-12 11:05:37 2024-07-12  4:05:37 0 [Note] Reading of all Master_info entries succeeded
2024-07-12 11:05:37 2024-07-12  4:05:37 0 [Note] Added new Master_info '' to hash table
2024-07-12 11:05:37 2024-07-12  4:05:37 0 [Note] mysqld: ready for connections.
2024-07-12 11:05:37 Version: '10.4.34-MariaDB-1:10.4.34+maria~ubu2004'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
2024-07-12 11:05:49 2024-07-12  4:05:49 0 [Note] mysqld (initiated by: unknown): Normal shutdown
2024-07-12 11:05:49 2024-07-12  4:05:49 0 [Note] Event Scheduler: Purging the queue. 0 events
2024-07-12 11:05:49 2024-07-12  4:05:49 0 [Note] InnoDB: FTS optimize thread exiting.
2024-07-12 11:05:50 2024-07-12  4:05:50 0 [Note] InnoDB: Starting shutdown...
2024-07-12 11:05:50 2024-07-12  4:05:50 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2024-07-12 11:05:50 2024-07-12  4:05:50 0 [Note] InnoDB: Buffer pool(s) dump completed at 240712  4:05:50
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] InnoDB: Shutdown completed; log sequence number 60952; transaction id 21
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] mysqld: Shutdown complete
2024-07-12 11:05:52 
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] Starting MariaDB 10.4.34-MariaDB-1:10.4.34+maria~ubu2004 source revision 16394f1aa1b4097f897b8ab01ea2064726cca059 as process 1
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] InnoDB: Using Linux native AIO
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] InnoDB: Uses event mutexes
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] InnoDB: Number of pools: 1
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] InnoDB: Using SSE2 crc32 instructions
2024-07-12 11:05:52 2024-07-12  4:05:52 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: Completed initialization of buffer pool
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: 10.4.34 started; log sequence number 60952; transaction id 20
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] Server socket created on IP: '::'.
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] InnoDB: Buffer pool(s) load completed at 240712  4:05:53
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] Reading of all Master_info entries succeeded
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] Added new Master_info '' to hash table
2024-07-12 11:05:53 2024-07-12  4:05:53 0 [Note] mysqld: ready for connections.
2024-07-12 11:05:53 Version: '10.4.34-MariaDB-1:10.4.34+maria~ubu2004'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2024-07-12 11:05:52 2024-07-12 04:05:52+00:00 [Note] [Entrypoint]: Temporary server stopped
2024-07-12 11:05:52 
2024-07-12 11:05:52 2024-07-12 04:05:52+00:00 [Note] [Entrypoint]: MariaDB init process done. Ready for start up.
2024-07-12 11:05:52

这是我的码头工人撰写：

version: "3.7"
services:
  postgres:
    image: debezium/postgres:13
    restart: always
    ports:
      - 5432:5432
    environment:
      - POSTGRES_USER=docker
      - POSTGRES_PASSWORD=docker
      - POSTGRES_DB=exampledb

  zookeeper:
    image: confluentinc/cp-zookeeper:5.5.3
    restart: always
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-enterprise-kafka:5.5.3
    restart: always
    depends_on: [zookeeper]
    environment:
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_BROKER_ID: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9991
    ports:
      - 9092:9092

  debezium:
    image: debezium/connect:1.4
    restart: always
    environment:
      BOOTSTRAP_SERVERS: kafka:9092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: connect_configs
      OFFSET_STORAGE_TOPIC: connect_offsets
      KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
      VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8087
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8087
      PLUGIN_PATH: /kafka/connect
      CONNECT_PLUGIN_PATH: /kafka/connect 
    volumes:
      - ./connect-distributed.properties:/etc/kafka/connect-distributed.properties
      - ./mysql-connector:/kafka/connectors 
    depends_on: [kafka]
    ports:
      - 8083:8083

  schema-registry:
    image: confluentinc/cp-schema-registry:5.5.3
    restart: always
    environment:
      - SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2181
      - SCHEMA_REGISTRY_HOST_NAME=schema-registry
      - SCHEMA_REGISTRY_LISTENERS=http://schema-registry:8087,http://localhost:8087
    ports:
      - 8087:8087
    depends_on: [zookeeper, kafka]

  mariadb:
    image: mariadb:10.4
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: root_password
      MYSQL_DATABASE: exampledb
      MYSQL_USER: user
      MYSQL_PASSWORD: password
    volumes:
      - D:/mariadb-config/debezium_fix.cnf:/etc/mysql/mariadb.conf.d/debezium_fix.cnf
    ports:
      - 3306:3306
    healthcheck:
      test: ["CMD", "/bin/bash", "-c", "/path/to/healthcheck.sh"]
      interval: 10s
      timeout: 10s
      retries: 5

  phpmyadmin:
    image: phpmyadmin/phpmyadmin
    restart: always
    environment:
      PMA_HOST: mariadb
      PMA_USER: user
      PMA_PASSWORD: password
    ports:
      - 8080:80
    depends_on:
      - mariadb

下面是 mariadb 连接的curl JSON：

{
  "name": "exampledb2-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",
    "database.hostname": "mariadb",
    "database.port": "3306",
    "database.user": "root",
    "database.password": "root_password",
    "database.server.id": "184054",
    "database.server.name": "mariadb",
    "database.whitelist": "exampledb",
    "database.history.kafka.bootstrap.servers": "localhost:9092",
    "database.history.kafka.topic": "dbhistory.exampledb",
    "include.schema.changes": "true",
    "database.connectionTimeZone": "UTC"
  }
}
docker apache-kafka docker-compose mariadb debezium
1个回答
0
投票

我需要编辑我的 mariadb docker 配置 '''

  mariadb:
    image: mariadb:10.5
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: mydatabase
      MYSQL_USER: root
      MYSQL_PASSWORD: password
    volumes:
      - ./mariadb-config:/etc/mysql/mariadb.conf.d
      - D:\mariakafka\my.cnf:/etc/mysql/my.cnf
      - ./mysql:/var/lib/mysql
    ports:
      - "3306:3306"

'''

这是 my.cnf 里面的内容 '''

[mysqld]
log_bin = /var/lib/mysql/mysql-bin.log
binlog_format = ROW

'''

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.