完美调度不适用于 docker-compose

问题描述 投票：0回答：2

运行代码时出现此错误。它说找不到文件。

日志：

prefect-docker-compose-database-1  | 
prefect-docker-compose-database-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
prefect-docker-compose-database-1  | 
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  starting PostgreSQL 14.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.293 UTC [1] LOG:  listening on IPv6 address "::", port 5432
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.296 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.300 UTC [21] LOG:  database system was shut down at 2022-11-25 11:53:37 UTC
prefect-docker-compose-database-1  | 2022-11-25 11:54:21.305 UTC [1] LOG:  database system is ready to accept connections
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     |  ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
prefect-docker-compose-orion-1     | | _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
prefect-docker-compose-orion-1     | |  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
prefect-docker-compose-orion-1     | |_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | Configure Prefect to communicate with the server with:
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     |     prefect config set PREFECT_API_URL=http://0.0.0.0:4200/api
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | View the API reference documentation at http://0.0.0.0:4200/docs
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | Check out the dashboard at http://0.0.0.0:4200
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | 
prefect-docker-compose-orion-1     | 
prefect-docker-compose-agent-1     | Starting v2.6.8 agent connected to http://orion:4200/api...
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     |   ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
prefect-docker-compose-agent-1     |  | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
prefect-docker-compose-agent-1     |  |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
prefect-docker-compose-agent-1     |  |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     | 
prefect-docker-compose-agent-1     | Agent started! Looking for work from queue(s): demo...
prefect-docker-compose-orion-1     | INFO:     Started server process [7]
prefect-docker-compose-orion-1     | INFO:     Waiting for application startup.
prefect-docker-compose-orion-1     | INFO:     Application startup complete.
prefect-docker-compose-orion-1     | INFO:     Uvicorn running on http://0.0.0.0:4200 (Press CTRL+C to quit)
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55756 - "GET /deployments/name/greetings/get_weather_docker_example HTTP/1.1" 404 Not Found
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55770 - "POST /flows/ HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "GET /block_types/slug/process HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "PATCH /block_types/9003f2ea-3325-4191-a0a3-67497f162165 HTTP/1.1" 204 No Content
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "GET /block_schemas/checksum/sha256%3A47c4ac364708f4a6f27fb10b18086ad92c0a53fbbdd9ba07c030467067979f84?version=2.6.8 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55780 - "POST /block_documents/ HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.5:55770 - "POST /deployments/ HTTP/1.1" 201 Created
prefect-docker-compose-cli-1 exited with code 0
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /assets/index.85f05129.css HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /assets/index.f501c99b.js HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /ui-settings HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "GET /health HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /flow_runs/count HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /flow_runs/count HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46162 - "POST /saved_searches/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "POST /flow_runs/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46132 - "POST /ui/flow_runs/history HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46172 - "POST /deployments/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46162 - "POST /work_queues/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46164 - "POST /flows/filter HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /flows/91f4a98a-877d-4c40-93c7-41236a23d2e5 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46148 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.1:46164 - "GET /work_queues/name/demo HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /work_queues/name/demo HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "POST /work_queues/31d37abe-6aee-4674-beaf-4bc8d2cae7c4/get_runs HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:55.107 | INFO    | prefect.agent - Submitting flow run '2f6ba8ff-364c-4ddf-974b-f49712fede9b'
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "POST /flow_runs/2f6ba8ff-364c-4ddf-974b-f49712fede9b/set_state HTTP/1.1" 201 Created
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /flows/91f4a98a-877d-4c40-93c7-41236a23d2e5 HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33890 - "GET /block_documents/7f22b67b-4947-46a1-b0d0-5ae260c71b62?include_secrets=true HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:55.162 | INFO    | prefect.infrastructure.process - Opening process 'heavenly-macaw'...
prefect-docker-compose-agent-1     | 11:54:55.164 | INFO    | prefect.agent - Completed submission of flow run '2f6ba8ff-364c-4ddf-974b-f49712fede9b'
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33902 - "GET /flow_runs/2f6ba8ff-364c-4ddf-974b-f49712fede9b HTTP/1.1" 200 OK
prefect-docker-compose-orion-1     | INFO:     172.21.0.4:33902 - "GET /deployments/c9398a97-4ecc-4b57-8acb-0a50d5a7b328 HTTP/1.1" 200 OK
prefect-docker-compose-agent-1     | 11:54:56.745 | ERROR   | Flow run 'heavenly-macaw' - Flow could not be retrieved from deployment.
prefect-docker-compose-agent-1     | Traceback (most recent call last):
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 255, in retrieve_flow_then_begin_flow_run
prefect-docker-compose-agent-1     |     flow = await load_flow_from_flow_run(flow_run, client=client)
prefect-docker-compose-agent-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/contextlib.py", line 222, in __aexit__
prefect-docker-compose-agent-1     |     await self.gen.athrow(typ, value, traceback)
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 218, in asyncnullcontext
prefect-docker-compose-agent-1     |     yield
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
prefect-docker-compose-agent-1     |     return await fn(*args, **kwargs)
prefect-docker-compose-agent-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/deployments.py", line 163, in load_flow_from_flow_run
prefect-docker-compose-agent-1     |     await storage_block.get_directory(from_path=deployment.path, local_path=".")
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/site-packages/prefect/filesystems.py", line 144, in get_directory
prefect-docker-compose-agent-1     |     shutil.copytree(from_path, local_path, dirs_exist_ok=True)
prefect-docker-compose-agent-1     |   File "/usr/local/lib/python3.11/shutil.py", line 558, in copytree
prefect-docker-compose-agent-1     |     with os.scandir(src) as itr:
prefect-docker-compose-agent-1     |          ^^^^^^^^^^^^^^^
prefect-docker-compose-agent-1     | FileNotFoundError: [Errno 2] No such file or directory: '/root/flows'

Flow.py：这是负责部署调度的流程文件

from prefect import flow, task
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule, RRuleSchedule, CronSchedule
from datetime import datetime, timedelta



@task
def say_hello(name):
    print(f"hello {name}")

@task
def say_goodbye(name):
    print(f"goodbye {name}")

@flow
def greetings(names=["arthur", "trillian", "ford", "marvin"]):
    for name in names:
        say_hello(name)
        say_goodbye(name)

if __name__ == "__main__":
    deployment = Deployment.build_from_flow(
        name="get_weather_docker_example",  
        schedule = IntervalSchedule(interval=timedelta(seconds=30)),
        flow=greetings,        
        work_queue_name="demo"
    )
    deployment.apply()

docker-compose.yml

version: "3.9"
services:

  ### Prefect Database
  database:
    image: postgres:14.1-alpine
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=orion
    expose:
      - 5432
    volumes: 
      - db:/var/lib/postgresql/data
    profiles: ["orion"]
  

  ### Prefect Orion API
  orion:
    image: prefecthq/prefect:2.6.8-python3.11
    restart: always
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["prefect", "orion", "start"]
    environment:
      - PREFECT_ORION_API_HOST=0.0.0.0
      - PREFECT_ORION_DATABASE_CONNECTION_URL=postgresql+asyncpg://postgres:postgres@database:5432/orion
    ports:
      - 4200:4200
    depends_on:
      - database
    profiles: ["orion"]

  ## Prefect Agent
  agent:
    image: prefecthq/prefect:2.6.8-python3.11
    restart: always
    entrypoint: ["prefect", "agent", "start", "-q", "demo"]
    environment:
      - PREFECT_API_URL=http://orion:4200/api
    depends_on:
      - orion
    profiles: ["agent"]


  ### Prefect CLI
  cli:
    image: prefecthq/prefect:2.6.8-python3.11
    entrypoint: ["python", "flow.py"]
    working_dir: "/root/flows"
    volumes:
      - "./flows:/root/flows"
    environment:
      - PREFECT_API_URL=http://orion:4200/api
    profiles: ["cli"]


volumes:
  prefect:
  db:
  minio:
networks:
  default:
    name: prefect-network

目录结构：这是目录结构的样子，flows 目录和 docker-compose.yml 位于同一目录中。


- flows
  - flow.py 
- docker-compose.yml are in same directory

运行 docker-compose 的命令：

docker compose --profile orion --profile agent --profile cli up
docker-compose scheduling prefect
2个回答
0
投票

您是否尝试过在存储上部署流程代码？ https://docs.prefect.io/concepts/storage/?h=storage

您需要将流程代码上传到存储上并将其部署到您的 docker 容器上。

block_prod = RemoteFileSystem(
    basepath="your storage choice path",
    settings=dict(
        token=KEY,
    ),
)
block_prod.save(flow_name, overwrite=True)

deployment = Deployment.build_from_flow(
    name=flow_name,
    flow=flow,
    storage=RemoteFileSystem.load(flow_name),
    infrastructure=DockerContainer(
        image=IMAGE,
        image_pull_policy="IF_NOT_PRESENT",
        networks=["prefect"],
    ),
)
deployment.apply()
0
投票

发生这种情况是因为您的 Prefect Agent 无权访问流程文件。要解决此问题，只需在代理容器内创建适当的卷即可：

  agent:
    image: prefecthq/prefect:2.6.8-python3.11
    restart: always
    entrypoint: ["prefect", "agent", "start", "-q", "demo"]
    volumes:
      - "./flows:/root/flows"
    environment:
      - PREFECT_API_URL=http://orion:4200/api
    depends_on:
      - orion
    profiles: ["agent"]
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.