我已经在本地环境中部署了 strimzi Kafka 集群。我还在本地环境中部署了 strimzi connect 。我想运行 JDBC Source Connector 从 Postgresql 读取数据并将数据写入主题。 当我运行代码时,JDBC 连接器 jar 文件不会复制到 /opt/kafka/plugins 文件夹。当我登录到 connect pod 时,发现部署 connect 集群时未创建插件文件夹。如果我手动创建它并尝试复制 jar 文件,则需要重新启动,当我重新启动时,所有内容都会重置。
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
namespace: strimzi
spec:
replicas: 1
bootstrapServers: my-kafka-cluster-kafka-bootstrap:9092
config:
group.id: "connect-cluster"
offset.storage.topic: "connect-cluster-offsets"
config.storage.topic: "connect-cluster-configs"
status.storage.topic: "connect-cluster-status"
plugin.path: "/opt/kafka/external-configuration/plugins" # Kafka Connect plugin path
image: strimzi/kafka-connect:latest
template:
pod:
securityContext:
runAsUser: 0 # Run as root to grant necessary permissions
externalConfiguration:
volumes:
- name: plugins
configMap:
name: my-connector-plugins # Use a ConfigMap to store the connectors
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: postgres-source-connector
namespace: strimzi
labels:
strimzi.io/cluster: my-connect-cluster
spec:
class: io.confluent.connect.jdbc.JdbcSourceConnector
tasksMax: 1
config:
connector.class: io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max: "1"
topic.prefix: "postgres-"
connection.url: "jdbc:postgresql://postgres:5432/exampledb"
connection.user: "postgres" # Username for PostgreSQL
connection.password: "examplepassword" # Password for PostgreSQL
table.whitelist: "users"
mode: "bulk"
poll.interval.ms: "5000"
timestamp.column.name: "last_modified"
numeric.mapping: "best_fit"
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: "false"
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
labels:
app: postgres
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:13
env:
- name: POSTGRES_PASSWORD
value: "examplepassword" # Change this to a more secure password
- name: POSTGRES_DB
value: "exampledb"
ports:
- containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
name: postgres
spec:
selector:
app: postgres
ports:
- protocol: TCP
port: 5432
targetPort: 5432
Strimzi 不允许您配置
plugin.path
和/或从 ConfigMap 加载插件(无论如何都不适合插件)。顾名思义,externalConfiguration
字段是为加载自定义配置而设计的。
如果您想向 Strimzi Connect 集群添加插件,请按照文档操作:https://strimzi.io/docs/operators/latest/full/deploying.html#using-kafka-connect-with-plug-ins -str