目录中不存在胶水表

Question

我正在将 json 文件转换为 parquet 格式，但收到错误“Glue Table 不存在”，这是代码：

 

  import awswrangler as wr
  import pandas as pd
  import urllib.parse
  import os
 

  os_input_s3_file_path = os.environ['s3_cleaned_layer']
  os_input_glue_catalog_db_name = os.environ['glue_catalog_db_name']
  os_input_glue_catalog_table_name = os.environ['glue_catalog_table_name']
  os_input_write_data_operation = os.environ['write_data_operation']

  def lambda_handler(event, context):
    
     #Get object and key name
     bucket = event['Records'][0]['s3']['bucket']['name']
     key= urllib.parse.unquote_plus(event['Records'][0]['s3']['object']
                 ['key'], encoding = 'utf-8')             
    
     try:
        
         #creating df from content
         df_raw = wr.s3.read_json('s3://{}/{}'.format(bucket, key))
        
         #get item from raw df
         df_step_1 = pd.json_normalize(df_raw['items'])
        
         #write to s3
         wr_response = wr.s3.to_parquet(
             df=df_step_1,
             dataset=True,
             database=os_input_glue_catalog_db_name,
             table=os_input_glue_catalog_table_name,
             mode=os_input_write_data_operation
             )
        
         return wr_response
        
     except Exception as e:
         print(e)
         print('Error getting object {}  {}'.format(key, bucket))
         raise e

我已经在配置选项卡中添加了环境变量。

附加政策：Amazons3fullaccess、Amazonglueservicerole

期望在新存储桶中创建镶木地板格式文件

Answer 1

检查 Amazon Athena 中是否存在所有必需的目录数据库/表。另请查看 https://docs.aws.amazon.com/athena/latest/ug/security-iam-athena.html 特别是 https://docs.aws.amazon.com/athena/latest /ug/fine-grained-access-to-glue-resources.html