如何在使用 DataprocInstantiateInlineWorkflowTemplateOperator Airflow Operator 时启用 GPU？

Question

我正在使用

airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator

将作业从 Airflow 提交到 GCP Dataproc。

现在它接受以下格式的模板：

{
    "placement": {
      "managed_cluster": {
        "cluster_name": ,
        "config": cluster_config
      }
    },
    "jobs": [
      {
        "step_id": task_id,
        "pyspark_job": {
          "args": [<listOfArguments>],
          "main_python_file_uri": <PythonFileToExecute>,
          "jar_file_uris": "<ListOfJarsAsAString>"
        }
      }
    ]
}

如何使用此模板启用加速器（例如）添加 GPU：

--master-accelerator type=nvidia-tesla-t4,count=2 \
--worker-accelerator type=nvidia-tesla-t4 \

---

一些参考资料：

1. https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dataproc_cluster
2. https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig#GceClusterConfig
3. https://cloud.google.com/dataproc/docs/concepts/workflows/workflow-parameters#parameterizing_a_workflow_template
4. https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/dataproc.html#DataprocInstantiateInlineWorkflowTemplateOperator

Answer 1

你找到解决办法了吗？