如何在转换数据类型时使“applymapping”正常工作

Question

我的问题是我的ETL作业中的特定列没有转换为搜索到的数据类型，这意味着每次运行作业并稍后爬网（每天）时，数据类型为“string”，应为“int” （整数）。

在数据源中，数据类型是“bigint”，我希望它在最终表中是“bigint”或“int”。

但是，这并没有发生，列“tester”的数据类型最终成为字符串。

这是在许多查询中使用的列，因此必须将其转换为整数（因为它之前是在更改之前）。

我尝试过多种源数据类型和目标数据类型的组合，但似乎没有一种工作，最终都是字符串。

我希望您可以帮助我，因为我不是AWS的专家，也无法获得任何进一步的支持。

下面我的ETL作业代码被粘贴（审查）

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('XX-bi-team')
bucket.objects.filter(Prefix="Tables_after_jobs/XX/tester/").delete()


## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "mysql_XX", table_name = "XX_XX_XX", transformation_ctx = "datasource0"]
## @return: datasource0
## @inputs: []
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "mysql_XX", table_name = "XX_XX_XX", transformation_ctx = "datasource0")
## @type: ApplyMapping
## @args: [mapping = [("v1", "string", "v1", "string"), ("XX_threshold_XX", "string", "XX_threshold_XX", "string"), ("limit_XX", "string", "limit_XX", "string"), ("XX_threshold", "decimal(19,2)", "XX_threshold", "decimal(19,2)"), ("created", "timestamp", "created", "timestamp"), ("gee_XX", "string", "gee_XX", "string"), ("type", "string", "type", "string"), ("uuid", "binary", "uuid", "binary"), ("v2", "int", "v2", "int"), ("geeXX", "decimal(19,2)", "geeXX", "decimal(19,2)"), ("limitXX", "decimal(29,2)", "limitXX", "decimal(29,2)"), ("XX", "int", "XX", "int"), ("id", "long", "id", "long"), ("XX", "decimal(29,2)", "XX", "decimal(29,2)"), ("XX_XX_id", "long", "XX_XX_id", "long"), ("replaces_id", "long", "tester", "int")], transformation_ctx = "applymapping1"]
## @return: applymapping1
## @inputs: [frame = datasource0]
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("v1", "string", "v1", "string"), ("XX_threshold_XX", "string", "XX_threshold_XX", "string"), ("limit_XX", "string", "limit_XX", "string"), ("XX_threshold", "decimal(19,2)", "XX_threshold", "decimal(19,2)"), ("created", "timestamp", "created", "timestamp"), ("gee_XX", "string", "gee_XX", "string"), ("type", "string", "type", "string"), ("uuid", "binary", "uuid", "binary"), ("v2", "int", "v2", "int"), ("geeXX", "decimal(19,2)", "geeXX", "decimal(19,2)"), ("limitXX", "decimal(29,2)", "limitXX", "decimal(29,2)"), ("XX", "int", "XX", "int"), ("id", "long", "id", "long"), ("XX", "decimal(29,2)", "XX", "decimal(29,2)"), ("XX_XX_id", "long", "XX_XX_id", "long"), ("tester", "bigint", "tester", "int")], transformation_ctx = "applymapping1")
## @type: DataSink
## @args: [connection_type = "s3", connection_options = {"path": "s3://XX-bi-team/Tables_after_jobs/XX/tester"}, format = "json", transformation_ctx = "datasink2"]
## @return: datasink2
## @inputs: [frame = applymapping1]
datasink2 = glueContext.write_dynamic_frame.from_options(frame = applymapping1, connection_type = "s3", connection_options = {"path": "s3://XX-bi-team/Tables_after_jobs/XX/tester"}, format = "json", transformation_ctx = "datasink2")
job.commit()

如上所述，我需要列“tester”成为整数而不是字符串，就像在当前状态中一样。

我希望你帮助我。

提前致谢！

Answer 1

首先尝试打印datasource0（datasource0.printSchema( )）的模式以查看实际数据类型。如果它是一个选择类型，那么你需要使用ResolveChoice来解决它。

另见here。