我正在尝试在名为:llm_workflow_dag.py 的 Airflow Dag 中使用 OpenAI 库,每当我尝试使用除此之外的任何库时,它都会在 UI 上出现错误:
损坏的 DAG:[/opt/airflow/dags/llm_workflow_dag.py] 回溯(最近一次调用):文件“”,第 488 行, 在 _call_with_frames_removed 文件中 “/opt/airflow/dags/llm_workflow_dag.py”,第 10 行,位于 从 langchain_openai 导入 AzureOpenAI ModuleNotFoundError:没有名为“langchain_openai”的模块
我已经使用requirements.txt安装了这个并检查了网络服务器并尝试在其上安装它的pip,它显示我的要求已经满足
对于网络服务器,我的 docker-compose.yml 看起来像这样:
airflow-webserver:
image: apache/airflow:2.10.2
container_name: airflow-webserver
restart: always
depends_on:
- postgres
- airflow-scheduler
environment:
- AIRFLOW__CORE__EXECUTOR=LocalExecutor
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
- AIRFLOW__CORE__LOAD_EXAMPLES=False
- _AIRFLOW_WWW_USER_USERNAME=admin
- _AIRFLOW_WWW_USER_PASSWORD=admin
# - PYTHONPATH=/opt/airflow/modules # Add this line to set PYTHONPATH
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./modules:/opt/airflow/modules # Mapping the modules folder
- ./airflow.cfg:/opt/airflow/airflow.cfg
- ./requirements.txt:/requirements.txt
- "C:/Users/navthars/Desktop/PENV/.venv:/usr/local/airflow/.venv"
# - ./modules:/opt/airflow/modules # Correct volume mapping for modules
ports:
- "8080:8080"
command: >
bash -c "pip install -r /requirements.txt && airflow webserver"
airflow-init:
image: apache/airflow:2.10.2
container_name: airflow-init
depends_on:
- postgres
environment:
- AIRFLOW__CORE__EXECUTOR=LocalExecutor
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
# - ./modules:/opt/airflow/modules # Mapping the modules folderd
entrypoint: >
/bin/bash -c
"airflow db init"
command: "echo Initializing Airflow Database"
我创建了一个像这样的dockerfile,它解决了这个问题:
# Dockerfile for custom Airflow image
# Use the base Airflow image
FROM apache/airflow:2.10.2
# Switch to root user to install system-level packages (if needed)
USER root
# Copy requirements.txt
COPY requirements.txt /requirements.txt
# Switch to airflow user before installing Python packages
USER airflow
# Install Python packages from requirements.txt
RUN pip install --no-cache-dir -r /requirements.txt
我在单个映像中创建了多个容器,并且没有在导致此问题的所有容器上安装所有库。