我正在尝试使用标志在Google Cloud Dataflow(作业ID:2020-06-08_23_39_43-14062032727466654144
)上运行Apache Beam Job
--experiment=beam_fn_api
--worker_harness_container_image=gcr.io/PROJECT_NAME/apachebeamp3.7_imageconversion:latest
[不幸的是,工作人员停留在开始状态。完全相同的配置的工作在今年年初(二月?)进行了,我想知道此后发生了什么变化,并且我需要进行哪些更改才能使其重新运行。
如果我使用]在本地运行作业,则>
--runner=PortableRunner \ --job_endpoint=embed \ --environment_config=PROJECT_NAME/apachebeamp3.7_imageconversion:latest
运行完美。
在数据流日志中,我看到以下错误消息:
getPodContainerStatuses for pod "dataflow-beamapp-sam-0609063936-65-06082339-h464-harness-zzpb_default(a65b24a783afd25920bf29ff27d7baf8)" failed: rpc error: code = Unknown desc = Error: No such container: 586554fec1cf2942c7d2f45589db02b217c90c2ea96982041fc3f12b4b6595ff"
和
ContainerStatus "1647b951d266b4b1d318317b1836002eb4731a510dffa38ba6b58b45a7710784" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 1647b951d266b4b1d318317b1836002eb4731a510dffa38ba6b58b45a7710784
由于
gcr.io/PROJECT_NAME/apachebeamp3.7_imageconversion:latest
当前为8bdf43f9cdcd20d4c258a7810c81cb5214ecc984e534117ef8ba1a4cab2a3dae
,所以我对容器ID感到有点困惑。
问题:
编辑
感谢指针。我看了dataflow.googleapis.com/kubelet
日志。我看到的唯一错误是
while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
ContainerStatus "55271a8a1af2a90d6162eda03bd8924aad502fd32f09ca50bf35af58e428cf59" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 55271a8a1af2a90d6162eda03bd8924aad502fd32f09ca50bf35af58e428cf59
Error syncing pod a65b24a783afd25920bf29ff27d7baf8 ("dataflow-beamapp-sam-0609063936-65-06082339-h464-harness-7056_default(a65b24a783afd25920bf29ff27d7baf8)"), skipping: [failed to "StartContainer" for "sdk0" with CrashLoopBackOff: "Back-off 10s restarting failed container=sdk0 pod=dataflow-beamapp-sam-0609063936-65-06082339-h464-harness-7056_default(a65b24a783afd25920bf29ff27d7baf8)"
。 奇怪的是,我在日志查看器中没有看到类别worker-startup
。我需要怎么做才能查看这些日志条目并能够进行此调试过程的下一步:-)?
我正在尝试使用标志--experiment = beam_fn_api --worker_harness_container_image = gcr.io / ...在Google Cloud Dataflow(Job-ID:2020-06-08_23_39_43-14062032727466654144)上运行Apache Beam Job
结果我犯了多个错误:
FROM apachebeam/python3.7_sdk:latest
更改为FROM apache/beam_python3.7_sdk:latest
。根据https://hub.docker.com/r/apachebeam/python3.7_sdk,从2.20.0版本开始进行了切换。