使用AWS Glue将CSV转换为ORC时如何排除分区？

Question

我在S3中有一堆CSV文件，我试图使用AWS Glue中的ETL作业转换为ORC。我有一个爬虫程序，它爬行包含CSV的目录并生成一个表。该表如下所示：

Column name | Data type | Partition key
---------------------------------------
field1      | string    |
field2      | string    |
field3      | string    |
partition_0 | string    | Partition (0)
partition_1 | string    | Partition (1)

接下来，我尝试将CS​​V转换为ORC文件。这是我正在使用的类似ETL脚本：

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'database', 'table_name', 'output_dir'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
partition_predicate = '(partition_0 = "val1") AND (partition_1 = "val2")'
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = args['database'], table_name = args['table_name'], push_down_predicate = partition_predicate, transformation_ctx = "datasource0")
final = glueContext.write_dynamic_frame.from_options(frame = datasource0, connection_type = "s3", connection_options = { "path": args['output_dir'] }, format = "orc")
job.commit()

我有另一个爬虫爬行我的包含ORC文件的输出目录。当它生成表时，它看起来像这样：

Column name | Data type | Partition key
---------------------------------------
field1      | string    |
field2      | string    |
field3      | string    |
partition_0 | string    |
partition_1 | string    |
partition_0 | string    | Partition (0)
partition_1 | string    | Partition (1)

看起来它认为分区是ORC文件中的字段（它们不应该是）。如何修改我的脚本以便CSV到ORC的转换不会将分区键包含在架构列中？

Answer 1

如果您需要保留分区，请将选项partitionKeys添加到编写器：

final = glueContext.write_dynamic_frame.from_options(frame = datasource0, connection_type = "s3", connection_options = { "path": args['output_dir'], "partitionKeys" -> Seq("partition_0", "partition_1") }, format = "orc")

否则只需删除分区列：

cleanDyf = datasource0.dropFields(Seq("partition_0", "partition_1"))
final = glueContext.write_dynamic_frame.from_options(frame = cleanDyf, connection_type = "s3", connection_options = { "path": args['output_dir'] }, format = "orc")