我正在尝试使用golang sdk将对象上传到AWS S3，而无需在我的系统中创建文件（尝试仅上传字符串）。但我很难实现这一目标。任何人都可以举例说明如何在不需要创建文件的情况下上传到AWS S3？
AWS如何上传文件的示例：
// Creates a S3 Bucket in the region configured in the shared config
// or AWS_REGION environment variable.
//
// Usage:
// go run s3_upload_object.go BUCKET_NAME FILENAME
func main() {
if len(os.Args) != 3 {
exitErrorf("bucket and file name required\nUsage: %s bucket_name filename",
os.Args[0])
}
bucket := os.Args[1]
filename := os.Args[2]
file, err := os.Open(filename)
if err != nil {
exitErrorf("Unable to open file %q, %v", err)
}
defer file.Close()
// Initialize a session in us-west-2 that the SDK will use to load
// credentials from the shared credentials file ~/.aws/credentials.
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-west-2")},
)
// Setup the S3 Upload Manager. Also see the SDK doc for the Upload Manager
// for more information on configuring part size, and concurrency.
//
// http://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#NewUploader
uploader := s3manager.NewUploader(sess)
// Upload the file's body to S3 bucket as an object with the key being the
// same as the filename.
_, err = uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(bucket),
// Can also use the `filepath` standard library package to modify the
// filename as need for an S3 object key. Such as turning absolute path
// to a relative path.
Key: aws.String(filename),
// The file to be uploaded. io.ReadSeeker is preferred as the Uploader
// will be able to optimize memory when uploading large content. io.Reader
// is supported, but will require buffering of the reader's bytes for
// each part.
Body: file,
})
if err != nil {
// Print the error and exit.
exitErrorf("Unable to upload %q to %q, %v", filename, bucket, err)
}
fmt.Printf("Successfully uploaded %q to %q\n", filename, bucket)
}
我已经尝试以编程方式创建文件，但它正在我的系统上创建文件，然后将其上传到S3。
Body结构的UploadInput字段只是一个io.Reader。所以传递你想要的任何io.Reader - 它不需要是一个文件。
在这个答案中，我将发布与我有关的所有与此问题相关的事情。非常感谢@ThunderCat和@Flimzy提醒我上传请求的body参数已经是io.Reader。我将发布一些示例代码，评论我从这个问题中学到了什么以及它如何帮助我解决这个问题。也许这会帮助像我这样的人和@AlokKumarSingh。
案例1：您已经拥有内存中的数据（例如，从Kafka，Kinesis或SQS等流/消息服务接收数据）
func main() {
if len(os.Args) != 3 {
fmt.Printf(
"bucket and file name required\nUsage: %s bucket_name filename",
os.Args[0],
)
}
bucket := os.Args[1]
filename := os.Args[2]
// this is your data that you have in memory
// in this example it is hard coded but it may come from very distinct
// sources, like streaming services for example.
data := "Hello, world!"
// create a reader from data data in memory
reader := strings.NewReader(data)
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
uploader := s3manager.NewUploader(sess)
_, err = uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(bucket),
Key: aws.String(filename),
// here you pass your reader
// the aws sdk will manage all the memory and file reading for you
Body: reader,
})
if err != nil {.
fmt.Printf("Unable to upload %q to %q, %v", filename, bucket, err)
}
fmt.Printf("Successfully uploaded %q to %q\n", filename, bucket)
}
案例2：您已经有一个持久文件，并且您想要上传它，但您不想将整个文件保存在内存中：
func main() {
if len(os.Args) != 3 {
fmt.Printf(
"bucket and file name required\nUsage: %s bucket_name filename",
os.Args[0],
)
}
bucket := os.Args[1]
filename := os.Args[2]
// open your file
// the trick here is that the method os.Open just returns for you a reader
// for the desired file, so you will not maintain the whole file in memory.
// I know this might sound obvious, but for a starter (as I was at the time
// of the question) it is not.
fileReader, err := os.Open(filename)
if err != nil {
fmt.Printf("Unable to open file %q, %v", err)
}
defer fileReader.Close()
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-1")},
)
uploader := s3manager.NewUploader(sess)
_, err = uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(bucket),
Key: aws.String(filename),
// here you pass your reader
// the aws sdk will manage all the memory and file reading for you
Body: fileReader,
})
if err != nil {
fmt.Printf("Unable to upload %q to %q, %v", filename, bucket, err)
}
fmt.Printf("Successfully uploaded %q to %q\n", filename, bucket)
}
案例3：这是我在系统的最终版本上实现它的方式，但要理解我为什么这样做，我必须给你一些背景知识。
我的用例发展了一下。上传代码将成为Lambda中的一个函数，文件变得非常庞大。这种变化意味着什么：如果我通过连接到Lambda函数的API网关中的入口点上传文件，我将不得不等待整个文件在Lambda中完成上传。由于lambda是根据调用的持续时间和内存使用量来定价的，因此这可能是一个非常大的问题。
因此，为了解决这个问题，我使用了预先签名的帖子URL进行上传。这对架构/工作流程有何影响？
我只是创建并验证URL，以便在后端将对象发布到S3并将此URL发送到前端，而不是从我的后端代码上传到S3。有了这个，我刚刚实现了一个分段上传到该URL。我知道这比问题更具体，但发现这个解决方案并不容易，所以我认为在这里为其他人记录它是个好主意。
以下是如何在nodejs中创建预签名URL的示例。
const AWS = require('aws-sdk');
module.exports.upload = async (event, context, callback) => {
const s3 = new AWS.S3({ signatureVersion: 'v4' });
const body = JSON.parse(event.body);
const params = {
Bucket: process.env.FILES_BUCKET_NAME,
Fields: {
key: body.filename,
},
Expires: 60 * 60
}
let promise = new Promise((resolve, reject) => {
s3.createPresignedPost(params, (err, data) => {
if (err) {
reject(err);
} else {
resolve(data);
}
});
})
return await promise
.then((data) => {
return {
statusCode: 200,
body: JSON.stringify({
message: 'Successfully created a pre-signed post url.',
data: data,
})
}
})
.catch((err) => {
return {
statusCode: 400,
body: JSON.stringify({
message: 'An error occurred while trying to create a pre-signed post url',
error: err,
})
}
});
};
如果你想使用go它是相同的想法，你只需要改变de sdk。
这是我最后写的
func (s *S3Sink) upload() {
now := time.Now()
key := s.getNewKey(now)
_, err := s.uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(s.bucket),
Key: aws.String(key),
Body: s.bodyBuf,
})
if err != nil {
glog.Errorf("Error uploading %s to s3, %v", key, err)
}
glog.Infof("Uploaded at %s", key)
s.lastUploadTimestamp = now.UnixNano()
s.bodyBuf.Truncate(0)
}