boto 相关问题

boto是Amazon Web Services的开源Python接口

如何使用Python Boto？我正在使用Boto，Python和Amazon S3。如果我使用 [键。然后，我获得了所有文件的所有键。 mybucket/files/pdf/abc.pdf mybucket/files/pdf/abc2.pdf 我的...

如果我使用

python amazon-s3 boto

回答 6 投票 0

Python 3.12 中没有名为“boto.vendored.six.moves”的模块

我正在将我的项目从python3.9升级到python 3.12。我在我的项目中使用 boto 来连接 s3 和所有我不断收到以下错误从 boto.vendored.six.moves 导入过滤器，http_client，m...

python python-3.x boto3 boto python-3.12

回答 1 投票 0

使用 MinIO 调用 MLflow 中的 PutObject 时，由于 MissingContentLength 导致 S3UploadFailedError

当尝试使用 mlflow.log_artifact() 保存/上传文件到 MinIO 时，我们的 MLflow 用户突然在之前工作的代码中遇到此错误，该错误是在 boto3 包中引发的（minio 的依赖项

python amazon-s3 boto minio mlflow

回答 1 投票 0

Boto + Mechanical Turk：如何获得所有字段？

目前，我想手动创建 HIT，并使用 mturk 的 API 检索它们。我正在使用 Boto 检索作业，但问题是我只能看到

python boto mechanicalturk

回答 1 投票 0

boto.exception.S3ResponseError：S3ResponseError：403禁止

我试图让 django 将静态文件上传到 S3，但我收到了 403 禁止错误。堆栈跟踪：回溯（最近一次调用最后一次）：文件“manage.py”，第 14 行，位于我试图让 django 将静态文件上传到 S3，但我收到了 403 禁止错误。堆栈跟踪： Traceback (most recent call last): File "manage.py", line 14, in <module> execute_manager(settings) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv self.execute(*args, **options.__dict__) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute output = self.handle(*args, **options) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 351, in handle return self.handle_noargs(**options) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 89, in handle_noargs self.copy_file(path, prefixed_path, storage, **options) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 184, in copy_file if not self.delete_file(path, prefixed_path, source_storage, **options): File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 115, in delete_file if self.storage.exists(prefixed_path): File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 209, in exists return k.exists() File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 391, in exists return bool(self.bucket.lookup(self.name)) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 143, in lookup return self.get_key(key_name, headers=headers) File "/home/levi/Projects/DoneBox/.virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 208, in get_key response.status, response.reason, '') boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden 设置.py： import os DIRNAME = os.path.dirname(__file__) # Django settings for DoneBox project. DEBUG = True TEMPLATE_DEBUG = DEBUG ADMINS = ( # ('Your Name', '[email protected]'), ) MANAGERS = ADMINS DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'. 'NAME': os.path.join(DIRNAME, "box.sqlite"), # Or path to database file if using sqlite3. 'USER': '', # Not used with sqlite3. 'PASSWORD': '', # Not used with sqlite3. 'HOST': '', # Set to empty string for localhost. Not used with sqlite3. 'PORT': '', # Set to empty string for default. Not used with sqlite3. } } # Local time zone for this installation. Choices can be found here: # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name # although not all choices may be available on all operating systems. # On Unix systems, a value of None will cause Django to use the same # timezone as the operating system. # If running in a Windows environment this must be set to the same as your # system time zone. TIME_ZONE = 'America/Denver' # Language code for this installation. All choices can be found here: # http://www.i18nguy.com/unicode/language-identifiers.html LANGUAGE_CODE = 'en-us' SITE_ID = 1 # If you set this to False, Django will make some optimizations so as not # to load the internationalization machinery. USE_I18N = True # If you set this to False, Django will not format dates, numbers and # calendars according to the current locale USE_L10N = True # Absolute filesystem path to the directory that will hold user-uploaded files. # Example: "/home/media/media.lawrence.com/media/" MEDIA_ROOT = '' # URL that handles the media served from MEDIA_ROOT. Make sure to use a # trailing slash. # Examples: "http://media.lawrence.com/media/", "http://example.com/media/" MEDIA_URL = "d1eyn4cjl5vzx0.cloudfront.net" # Absolute path to the directory static files should be collected to. # Don't put anything in this directory yourself; store your static files # in apps' "static/" subdirectories and in STATICFILES_DIRS. # Example: "/home/media/media.lawrence.com/static/" STATIC_ROOT = os.path.join(DIRNAME, "static") # URL prefix for static files. # Example: "http://media.lawrence.com/static/" STATIC_URL = "d280kzug7l5rug.cloudfront.net" # URL prefix for admin static files -- CSS, JavaScript and images. # Make sure to use a trailing slash. # Examples: "http://example.com/static/admin/", "/static/admin/". ADMIN_MEDIA_PREFIX = '/static/admin/' # Additional locations of static files STATICFILES_DIRS = ( # Put strings here, like "/home/html/static" or "C:/www/django/static". # Always use forward slashes, even on Windows. # Don't forget to use absolute paths, not relative paths. os.path.join(DIRNAME, "main", "static"), ) # List of finder classes that know how to find static files in # various locations. STATICFILES_FINDERS = ( 'django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 'django.contrib.staticfiles.finders.DefaultStorageFinder', ) # Make this unique, and don't share it with anybody. SECRET_KEY = '<snip>' # List of callables that know how to import templates from various sources. TEMPLATE_LOADERS = ( 'django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader', 'django.template.loaders.eggs.Loader', ) MIDDLEWARE_CLASSES = ( 'django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', ) ROOT_URLCONF = 'DoneBox.urls' TEMPLATE_DIRS = ( # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates". # Always use forward slashes, even on Windows. # Don't forget to use absolute paths, not relative paths. os.path.join(DIRNAME, "main", "templates"), os.path.join(DIRNAME, "templates"), os.path.join(DIRNAME, "basic", "blog", "templates"), ) INSTALLED_APPS = ( 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.sitemaps', # Uncomment the next line to enable the admin: 'django.contrib.admin', # Uncomment the next line to enable admin documentation: 'storages', 'django.contrib.admindocs', 'main', 'contacts', 'piston', 'registration', # 'contact_form', 'basic', 'basic.blog', ) # A sample logging configuration. The only tangible logging # performed by this configuration is to send an email to # the site admins on every HTTP 500 error. # See http://docs.djangoproject.com/en/dev/topics/logging for # more details on how to customize your logging configuration. LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'handlers': { 'mail_admins': { 'level': 'ERROR', 'class': 'django.utils.log.AdminEmailHandler' } }, 'loggers': { 'django.request': { 'handlers': ['mail_admins'], 'level': 'DEBUG', 'propagate': True, }, 'django.db.backends': { 'handlers': ['mail_admins'], 'level': 'DEBUG', 'propagate': True, } } } DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' AWS_ACCESS_KEY_ID = '<snip>' AWS_SECRET_ACCESS_KEY = '<snip>' STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage' AWS_STORAGE_BUCKET_NAME = "donebox-static" STATIC_FILES_BUCKET = "donebox-static" MEDIA_FILES_BUCKET = "donebox-media" ACCOUNT_ACTIVATION_DAYS = 7 EMAIL_HOST = "email-smtp.us-east-1.amazonaws.com" EMAIL_HOST_USER = '<snip>' EMAIL_HOST_PASSWORD = '<snip>' EMAIL_PORT = 587 EMAIL_USE_TLS = True TEMPLATE_CONTEXT_PROCESSORS = ( "django.contrib.auth.context_processors.auth", "django.core.context_processors.debug", "django.core.context_processors.i18n", "django.core.context_processors.media", "django.core.context_processors.static", "django.contrib.messages.context_processors.messages", "DoneBox.main.context_processors_PandC", ) 要求.pip： django==1.3 django-storages==1.1.4 django-registration==0.8 django-piston==0.2.3 django-tagging==0.3.1 django-extensions==0.8 BeautifulSoup==3.2.1 boto==2.4.1 mysql-python==1.2.3 tweepy==1.9 feedparser==5.1.2 pycrypto==2.6 我正在使用 Amazon IAM 作为特定密钥 ID 和访问密钥，但刚刚遇到了相同的 403 Forbidden...事实证明，您需要授予针对存储桶根及其子对象两者的权限： { "Statement": [ { "Principal": { "AWS": "*" }, "Effect": "Allow", "Action": "s3:*", "Resource": ["arn:aws:s3:::bucket-name/*", "arn:aws:s3:::bucket-name"] } ] } 我建议您尝试单独测试您的 AWS 凭证，以验证凭证是否确实具有在 S3 存储桶中读取和写入数据的权限。以下应该有效： >>> import boto >>> s3 = boto.connect_s3('<access_key>', '<secret_key>') >>> bucket = s3.lookup('donebox-static') >>> key = bucket.new_key('testkey') >>> key.set_contents_from_string('This is a test') >>> key.exists() >>> key.delete() 您应该对另一个存储桶（“donebox-media”）尝试相同的测试。如果有效，则权限正确，问题出在 Django 存储代码或配置中。如果失败并返回 403，则： access_key/secret_key 字符串不正确 access_key/secret_key 正确，但该帐户没有写入存储桶所需的权限希望有帮助。请报告您的发现。我也遇到了同样的问题，最后发现真正的问题是服务器时间。它配置错误，AWS 以 403 FORBIDDEN 响应。使用 Debian，您可以使用 NTP 自动配置： ntpdate 0.pool.ntp.org 如果您的机器时间设置不正确也会发生这种情况如果这对任何人都有帮助，我必须添加以下配置条目才能使 collectstatic 正常工作而不返回 403： AWS_DEFAULT_ACL = '' 也有可能使用了错误的凭据。验证： import boto s3 = boto.connect_s3('<your access key>', '<your secret key>') bucket = s3.get_bucket('<your bucket>') # does this work? s3 = boto.connect_s3() s3.aws_access_key_id # is the same key being used by default? 如果没有，请查看 ~/.boto、~/.aws/config 和 ~/.aws/credentials。这是一个具有最小权限的改进。在所有情况下，正如所讨论的，其他地方s3:ListAllMyBuckets对于所有存储桶都是必要的。在默认配置中，django-storages 会将文件上传到具有公共读取权限的 S3 - 请参阅 django-storages Amazon S3 后端试验和错误表明，在此默认配置中，仅需要两个权限：首先上传文件，然后将该对象的权限设置为公共。无需执行任何其他操作，因为从那时起，无论如何，该对象的读取都是公开的。 IAM 用户策略 - 公开读取（默认）： s3:PutObject 让对象公开可读并不总是可取的。这是通过在设置文件中设置相关属性来实现的。 Django 设置.py： s3:PutObjectAcl 然后不再需要{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::bucketname/*" } ] } ，最低权限如下： IAM 用户策略 - 私有： ... AWS_DEFAULT_ACL = "private" ... 另一种避免自定义策略并使用 AWS 预定义策略的解决方案：为您的 S3 用户添加 S3 完全访问权限。 IAM / 用户 / 权限和附加策略添加策略 “AmazonS3FullAccess” 也许您实际上无权访问您尝试查找/获取/创建的存储桶.. 请记住：存储桶名称在整个 S3 生态系统中必须是唯一的，因此，如果您尝试访问（查找/获取/创建）名为“test”的存储桶，您将无法访问它。

django amazon-s3 boto django-storage

回答 9 投票 0

如何向 boto (Amazon S3) 中的密钥添加标签？

我正在尝试标记已上传到 S3 的密钥。在下面的内容中，我只是从字符串创建一个文件。一旦我有了密钥，我不知道如何标记文件。我尝试过 Tag 和 TagSet。夫...

python amazon-s3 boto boto3

回答 5 投票 0

如何在 Python 3 上将文件上传到 Google Cloud Storage？

如何从 Python 3 将文件上传到 Google Cloud Storage？如果 Python 3 不可行，最终会是 Python 2。我看了又看，但还没有找到真正有效的解决方案。我...

python python-3.x google-cloud-storage boto google-cloud-platform

回答 5 投票 0

s3 - 如何快速获取文件的行数？ wc -l 太慢了

有人有快速获取 S3 中托管文件的行数的方法吗？最好使用 CLI、s3api，但我也对 python/boto 持开放态度。注意：解决方案必须以非交互方式运行，即......

amazon-web-services amazon-s3 boto boto3 aws-cli

回答 5 投票 0

Boto AWS Glacier - 检索档案

我实际上正在使用 python boto 将数据存储在我的冰川库上并启动检索 jov 和库存作业。这非常有效。但我没有找到任何有关下载方法的示例...

amazon-web-services boto amazon-glacier

回答 2 投票 0

我可以匿名使用boto3吗？

使用 boto，我可以通过传递 anon= 关键字参数来连接到公共 S3 存储桶，而无需凭据。 s3 = boto.connect_s3(anon=True) boto3 可以吗？

python amazon-web-services amazon-s3 boto boto3

回答 3 投票 0

botocore >= 1.28.0 在多线程应用程序中速度较慢

Boto3 官方文档建议为每个线程创建一个新资源：https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-or-multiprocessing-with-resources

python amazon-web-services amazon-s3 boto botocore

回答 1 投票 0

您可以列出 S3 存储桶中的所有文件夹吗？

我有一个包含多个文件夹的存储桶，每个文件夹包含多个图像。是否可以列出所有文件夹而不迭代存储桶中的所有键（文件夹和图像）...

python amazon-s3 boto

回答 2 投票 0

AWS CloudFormation：一个大模板文件还是许多小模板文件？

我即将重写大量 AWS 部署代码，以使用 boto 控制的 CloudFormation 启动所有内容，而不是使用 boto 单独启动每个元素。有谁知道它是否&...

amazon-web-services boto aws-cloudformation

回答 5 投票 0

双重SSLEOFError：EOF发生违反协议

我口是心非运行多年，稳定无问题。现在，当我将文件备份到 Amazon S3 时，我遇到了以下问题。所有文件的格式为“口是心非”。