错误“4 DEADLINE_EXCEEDED：102.893秒后超出截止日期”

Question

尝试使用 Cloud Run 上托管的 Node.js 应用程序将大型数据集上传到 Firestore 时，遇到“4 DEADLINE_EXCEEDED：102.893 秒后超出截止日期”错误。该错误发生在上传过程的中间，尽管数据最终写入 Firestore，但该过程花费了过多的时间，并且始终导致超出截止日期的错误。

云运行配置为： 资源： 内存8GIB 中央处理器：2个 要求： 超时1800秒 每个实例最大并发请求数 80 仅在请求处理期间分配CPU

最小实例数 0 最大实例数 25

import { firestore } from "firebase-admin";
import { db } from "../firebase/config";
import { areAllValuesNullOrEmpty } from "./areAllValuesNullOrEmpty";
import { removeNullValues } from "./filterNullValues";

const BATCH_SIZE = 450;
const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

export const uploadToFirestore = async (
  userRefId: string,
  collectionName: string,
  docId: string,
  data: object,
  retries: number = 3,
  delayMs: number = 1000
): Promise<void> => {
  try {
    const filteredData = removeNullValues(data);
    if (areAllValuesNullOrEmpty(filteredData)) {
      return;
    }

    const patientRef = db.collection("patients").doc(userRefId);
    const summaryRef = patientRef.collection(collectionName).doc(docId);

    await uploadInBatches(summaryRef, filteredData);

    console.log(`Data uploaded to ${collectionName}/${docId}`);
  } catch (error) {
    console.error(`Failed to upload data to ${collectionName}/${docId}`, error);

    if (retries > 0) {
      console.log(
        `Retrying upload to ${collectionName}/${docId} (${retries} retries left)...`
      );
      await delay(delayMs);
      await uploadToFirestore(
        userRefId,
        collectionName,
        docId,
        data,
        retries - 1,
        delayMs
      );
    } else {
      console.error(
        `Exhausted retries for uploading data to ${collectionName}/${docId}`
      );
    }
  }
};

const uploadInBatches = async (
  docRef: firestore.DocumentReference,
  data: object
): Promise<void> => {
  const batch = db.batch();
  batch.set(docRef, data);

  await batch.commit();
};

export const uploadActivityToFirestore = async (
  userRefId: string,
  collectionName: string,
  docId: string,
  data: { summary_id: string; [key: string]: any },
  retries: number = 3,
  delayMs: number = 1000
): Promise<void> => {
  try {
    const filteredData = removeNullValues(data);
    if (areAllValuesNullOrEmpty(filteredData)) {
      return;
    }

    const patientRef = db.collection("patients").doc(userRefId);
    const summaryRef = patientRef.collection(collectionName).doc(docId);
    const docSnapshot = await summaryRef.get();

    if (docSnapshot.exists) {
      const existingData = docSnapshot.data();
      const activities = existingData?.activities || {};

      activities[filteredData.summary_id] = filteredData;

      await uploadInBatches(summaryRef, { activities });
    } else {
      await uploadInBatches(summaryRef, {
        activities: {
          [filteredData.summary_id]: filteredData,
        },
      });
    }

    console.log(`Data uploaded to ${collectionName}/${docId}`);
  } catch (error) {
    console.error(`Failed to upload data to ${collectionName}/${docId}`, error);

    if (retries > 0) {
      console.log(
        `Retrying upload to ${collectionName}/${docId} (${retries} retries left)...`
      );
      await delay(delayMs);
      await uploadActivityToFirestore(
        userRefId,
        collectionName,
        docId,
        data,
        retries - 1,
        delayMs
      );
    } else {
      console.error(
        `Exhausted retries for uploading data to ${collectionName}/${docId}`
      );
    }
  }
};

这是我调用该函数时的示例：

const handleDaily = async (payload: TerraPayload) => {
  const data = payload.data as unknown as Daily[];
  const user = payload.user as unknown as TerraUser;
  const userRefId = user.reference_id;

  for (const dailyData of data) {
    const startTime = dailyData.metadata.start_time;
    const endTime = dailyData.metadata.end_time;
    const collectionId = generateCollectionId(startTime, endTime);

    const collectionsData = generateDailyCollections(dailyData);

    for (const [collectionName, collectionData] of Object.entries(
      collectionsData
    )) {
      await uploadToFirestore(
        userRefId,
        collectionName,
        collectionId,
        collectionData
      );
    }
  }
};

• 我尝试过批量写入器、批量写入和事务。
• 设置请求之间的延迟。
• 将数据分解成更小的块。
• 上传失败时重试，重试次数有限。
• 我尝试了之前帖子中的一些解决方案，但都给出了相同的结果。

正如我提到的，由于此错误造成的延迟，该过程花费了很多时间，而且我可以看到，如果此错误出现一次，则之后的所有批次都会因相同的错误而失败。 但所有数据都会进入 Firestore，这意味着尽管出现错误，它还是上传到了 Firestore。

我希望能够没有任何错误地上传数据，但我不知道是什么导致了这个问题。

Answer 1

常见问题是缺少等待或达到限制。

一种可能性是，当请求仍在处理时，您的云运行正在扩展到 0。