kafka connect 不加载连接器

Question

我正在使用以下 dockerfile 在我的 kafka-connect 映像中加载 mysql 连接器。

FROM eclipse-temurin
RUN mkdir -p /opt/kafka/bin /opt/kafka/libs /opt/kafka/config /opt/kafka/logs
COPY ./kafka/bin /opt/kafka/bin/
COPY ./kafka/libs /opt/kafka/libs/
COPY ./kafka/config/connect-distributed.properties /opt/kafka/config/
COPY ./kafka/config/connect-log4j.properties /opt/kafka/config/
COPY ./jdbc-source-bulk.json /opt/kafka
RUN mkdir -p /opt/kafka/connectors
COPY ./mysql-connector-j-8.3.0/* /opt/kafka/connectors/
WORKDIR /opt/kafka
EXPOSE 8083
ENTRYPOINT ["./bin/connect-distributed.sh","config/connect-distributed.properties"]

运行容器时得到以下信息

[2024-04-21 20:31:45,252] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.AbstractConnectCli:120)
[2024-04-21 20:31:45,304] INFO Loading plugin from: /opt/kafka/connectors/mysql-connector-j-8.3.0.jar (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,529] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/mysql-connector-j-8.3.0.jar} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,537] INFO Loading plugin from: /opt/kafka/connectors/build (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,546] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/build/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,546] INFO Loading plugin from: /opt/kafka/connectors/demo (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,554] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/demo/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,554] INFO Loading plugin from: /opt/kafka/connectors/generated (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,562] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/generated/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,563] INFO Loading plugin from: /opt/kafka/connectors/legacy (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,570] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/legacy/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,570] INFO Loading plugin from: /opt/kafka/connectors/main (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,577] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/main/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,578] INFO Loading plugin from: /opt/kafka/connectors/test (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,585] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/test/} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,585] INFO Loading plugin from: classpath (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,600] INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@c387f44 (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,601] INFO Scanning plugins with ServiceLoaderScanner took 297 ms (org.apache.kafka.connect.runtime.isolation.PluginScanner:70)
[2024-04-21 20:31:45,602] INFO Loading plugin from: /opt/kafka/connectors/mysql-connector-j-8.3.0.jar (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)
[2024-04-21 20:31:45,913] INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/connectors/mysql-connector-j-8.3.0.jar} (org.apache.kafka.connect.runtime.isolation.PluginScanner:80)
[2024-04-21 20:31:45,913] INFO Loading plugin from: /opt/kafka/connectors/build (org.apache.kafka.connect.runtime.isolation.PluginScanner:75)

结果：curl -X PUT -H“内容类型：application/json”
-d @jdbc-source-bulk.json
本地主机：8083/connectors/mysql-source-bulk/config

{"error_code":500,"message":"Failed to find any class that implements Connector and which name matches io.confluent.connect.jdbc.JdbcSourceConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='3.7.0', encodedVersion=3.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='3.7.0', encodedVersion=3.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='3.7.0', encodedVersion=3.7.0, type=source, typeName='source', location='classpath'}"}

有什么线索我在这里做错了吗？

尝试加载连接器，但 kafka-connect 无法识别它们。

Answer 1

mysql 连接器 jar 是一个 JDBC 驱动程序，没有 kafka 连接插件。您需要单独将 Confluence JDBC 连接器添加到插件路径