如何使用pyspark从aws glue中的时间戳中提取Year

Question

我需要从时间戳中获取年份，同时在aws glue中转换原始数据。以下是我正在尝试但不起作用的内容。

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import *

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "default", table_name = "xxx", transformation_ctx = "datasource0")
def AddDateYearForPartition(rec):
  rec["year"] = year(rec.date_entered);
  return rec

mapped_dyF =  Map.apply(frame = datasource0, f = AddDateYearForPartition)

Answer 1

据我所知，您需要先将数据源转换为数据帧，我这样做：

spark_df = dropnullfields0.toDF()
spark_df = spark_df.withColumn('year', year(spark_df.sessionstarttime).cast("string"))

Answer 2

这是Scala中的代码（如果有人需要它）：

import org.apache.spark.sql.functions._

val sourceDf = datasource0.toDF
val resultDf = sourceDf.withColumn("year", year(col("date_entered")))
val resultDyf = DynamicFrame(resultDf, glueContext)