使用 Formidable 将内容直接（不保存在 /tmp 上）流式传输到 Azure Blob 存储

Question

Formidable 在这里提供了一个示例：https://github.com/node-formidable/formidable/blob/master/examples/store-files-on-s3.js 有关如何将流上传到 s3 存储桶而不保存内容到 tmp 文件（顺便说一句，这是默认行为）。但是，我在使用 TypeScript 和 @azure/storage-blob 复制它时遇到了麻烦。

如有任何帮助，我们将不胜感激。

注意：这是 NextJS 项目，我不使用应用程序文件夹结构。

如果我按照以下步骤进行，我将必须创建一个流来从 /tmp/{someRandomId} 读取，这是我试图避免的。

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse,
) {
  if (req.method == "POST") {
       const form = formidable({ 
        multiples: true
      });
      const [fields, files] = await form.parse(req);
      const formType = Array.isArray(fields?.formType) ? fields.formType[0] : null;
      const file: File | undefined = Array.isArray(files?.file) ? files.file[0] : undefined;
      console.log(files);
      if (!formType || !file?.originalFilename || !file.filepath) {
        res.status(400).send("Invalid request");
        return;
      }
      const blobServiceClient = BlobServiceClient.fromConnectionString(
          env.AZURE_STORAGE_CONNECTION_STRING,
      );
      const containerClient = blobServiceClient.getContainerClient(env.AZURE_CONTAINER_NAME);
      const dynamicPath = createDynamicPath(
          formType,
          file.originalFilename,
      );
      const blockBlobClient = containerClient.getBlockBlobClient(dynamicPath);

      // // upload the file
      **await blockBlobClient.uploadData(file.filepath);** // I do not want this
      const blobUrl: string = blockBlobClient.url;
...

Answer 1

我尝试使用 Formidable 使用以下代码将文件内容直接流式传输到 Azure Blob 存储，而不将其保存到临时文件。

代码：

pages/api/postStream.ts：

import { NextApiRequest, NextApiResponse } from 'next';
import { IncomingForm, File } from 'formidable';
import * as azureStorage from '../../lib/azureConnection';
import { PassThrough } from 'stream';
import fs from 'fs';
export const config = {
  api: {
    bodyParser: false,
  },
};
export default async function postStream(req: NextApiRequest, res: NextApiResponse) {
  const form = new IncomingForm({ multiples: true });
  const uploadPromises: Promise<string>[] = [];
  form.on('file', (formName: string, file: File) => {
    if (file && file.filepath) {
      const passThrough = new PassThrough();
      const readableStream = fs.createReadStream(file.filepath);
      readableStream.pipe(passThrough);
      uploadPromises.push(
        azureStorage.uploadStreamToAzureStorage(passThrough, file.originalFilename || 'defaultFilename', file.mimetype || 'application/octet-stream')
      );
    }
  });
  form.parse(req, async (err: any) => {
    if (err) {
      console.error('Error parsing form:', err);
      res.status(500).json({ error: 'Error parsing form data' });
      return;
    }
    try {
      const uploadedUrls = await Promise.all(uploadPromises);
      res.status(200).json({ fileUrls: uploadedUrls });
    } catch (error) {
      console.error('Error uploading stream:', error);
      res.status(500).json({ error: 'Stream upload failed' });
    }
  });
}

组件/FileUploadForm.tsx：

import { useState, ChangeEvent } from 'react';
const FileUploadForm: React.FC = () => {
  const [file, setFile] = useState<File | null>(null);

  const handleFileChange = (e: ChangeEvent<HTMLInputElement>) => {
    if (e.target.files && e.target.files[0]) {
      setFile(e.target.files[0]);
    }
  };
  const uploadFile = async () => {
    if (!file) return;

    const formData = new FormData();
    formData.append('file', file);
    try {
      const response = await fetch('/api/postStream', {
        method: 'POST',
        body: formData,
      });
      if (response.ok) {
        const data = await response.json();
        console.log('File uploaded successfully!', data);
      } else {
        console.error('File upload failed.');
      }
    } catch (error) {
      console.error('Error uploading file:', error);
    }
  };
  return (
    <>
      <input type="file" onChange={handleFileChange} />
      <button onClick={uploadFile}>Upload File</button>
    </>
  );
};
export default FileUploadForm;

lib/azureConnection.ts：

import { BlobServiceClient } from '@azure/storage-blob';
import { Readable } from 'stream';
const connectionString = process.env.AZURE_CONNECTION || '';
const blobServiceClient = BlobServiceClient.fromConnectionString(connectionString);
const containerName = process.env.AZURE_CONTAINER_NAME || '';
export async function uploadStreamToAzureStorage(stream: Readable, fileName: string, contentType: string): Promise<string> {
  const containerClient = blobServiceClient.getContainerClient(containerName);
  const blockBlobClient = containerClient.getBlockBlobClient(fileName);
  try {
    await blockBlobClient.uploadStream(
      stream,
      4 * 1024 * 1024, 
      20, 
      {
        blobHTTPHeaders: { blobContentType: contentType },
      }
    );
    return blockBlobClient.url;
  } catch (error) {
    console.error('Error uploading stream:', error);
    throw error;
  }
}

类型/formidable.d.ts：

declare module 'formidable' {
  import { IncomingMessage } from 'http';
  import { EventEmitter, Writable } from 'events';
  import { Stream } from 'stream';
  export interface FormOptions {
    encoding?: string;
    uploadDir?: string;
    keepExtensions?: boolean;
    maxFileSize?: number;
    maxFields?: number;
    maxFieldsSize?: number;
    hash?: boolean | 'sha1' | 'md5';
    multiples?: boolean;
  }
  export class IncomingForm extends EventEmitter {
    constructor(options?: FormOptions);
    parse(req: IncomingMessage, callback?: (err: any, fields: Fields, files: Files) => void): void;
    on(event: 'fileBegin', listener: (formName: string, file: File) => void): this;
    on(event: string, listener: Function): this;
  }
  export interface File {
    size: number;
    path: string;
    name: string;
    type: string;
    hash?: string;
    lastModifiedDate?: Date;
    mimetype?: string;
    originalFilename?: string;
    newFilename?: string;
    filepath: string;
    _writeStream: Writable;
  }
  export interface Fields {
    [key: string]: string[];
  }
  export interface Files {
    [key: string]: File[];
  }
}

pages/index.tsx：

import FileUploadForm from '../components/FileUploadForm';
const Home: React.FC = () => {
  return (
    <div>
      <h1>Stream Upload to Azure Blob Storage</h1>
      <FileUploadForm />
    </div>
  );
};
export default Home;

.env.local：

AZURE_CONNECTION='<storage_connection>'
AZURE_CONTAINER_NAME='<container_name>'

浏览器输出：

我点击 Choose File 选择一个文件，然后点击 Upload File 按钮，如下所示。

本地输出：

Azure 门户：

blob 已成功上传，如下所示。