在 Apache Airflow Dag 中导入库时出错

Question

我正在尝试在名为：llm_workflow_dag.py 的 Airflow Dag 中使用 OpenAI 库，每当我尝试使用除此之外的任何库时，它都会在 UI 上出现错误：

损坏的 DAG：[/opt/airflow/dags/llm_workflow_dag.py] 回溯（最近一次调用）：文件“”，第 488 行， 在 _call_with_frames_removed 文件中 “/opt/airflow/dags/llm_workflow_dag.py”，第 10 行，位于 从 langchain_openai 导入 AzureOpenAI ModuleNotFoundError：没有名为“langchain_openai”的模块

我已经使用requirements.txt安装了这个并检查了网络服务器并尝试在其上安装它的pip，它显示我的要求已经满足

对于网络服务器，我的 docker-compose.yml 看起来像这样：

      airflow-webserver:
    image: apache/airflow:2.10.2
    container_name: airflow-webserver
    restart: always
    depends_on:
      - postgres
      - airflow-scheduler
    environment:
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
      - AIRFLOW__CORE__LOAD_EXAMPLES=False
      - _AIRFLOW_WWW_USER_USERNAME=admin        
      - _AIRFLOW_WWW_USER_PASSWORD=admin        
      # - PYTHONPATH=/opt/airflow/modules  # Add this line to set PYTHONPATH

    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      - ./modules:/opt/airflow/modules  # Mapping the modules folder
      - ./airflow.cfg:/opt/airflow/airflow.cfg  
      - ./requirements.txt:/requirements.txt  
      - "C:/Users/navthars/Desktop/PENV/.venv:/usr/local/airflow/.venv"  
      # - ./modules:/opt/airflow/modules  # Correct volume mapping for modules

    ports:
      - "8080:8080"
    command: >
      bash -c "pip install -r /requirements.txt && airflow webserver"
  airflow-init:
    image: apache/airflow:2.10.2
    container_name: airflow-init
    depends_on:
      - postgres
    environment:
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    volumes:
      - ./dags:/opt/airflow/dags
      - ./logs:/opt/airflow/logs
      # - ./modules:/opt/airflow/modules  # Mapping the modules folderd
    entrypoint: >
      /bin/bash -c
      "airflow db init"
    command: "echo Initializing Airflow Database"

我的文件夹结构如下所示。

Answer 1

我创建了一个像这样的dockerfile，它解决了这个问题：

# Dockerfile for custom Airflow image

# Use the base Airflow image
FROM apache/airflow:2.10.2

# Switch to root user to install system-level packages (if needed)
USER root

# Copy requirements.txt
COPY requirements.txt /requirements.txt

# Switch to airflow user before installing Python packages
USER airflow

# Install Python packages from requirements.txt
RUN pip install --no-cache-dir -r /requirements.txt

我在单个映像中创建了多个容器，并且没有在导致此问题的所有容器上安装所有库。