如何在pyspark做年龄段？

Question

假设我有一个带有数字列Age的数据框。

我想根据Age中的值生成一个新列

像这样的东西：

   Age       |    New Age Range
-----------------------------------
    4        |        0-18
   25        |        19-65
   90        |        65+
  ....       |        ....

我想过使用一个函数......

def AgeByRange(Age):
  for i in Age:
    if i < 19:
      return '0-18'
    elif ......

我怀疑有更好的方法来做到这一点。

Answer 1

嗨，您只需创建带范围的DataFrame并使用连接，例如

from pyspark.sql.types import *
df = sqlctx.createDataFrame([(1),(25),(90)],IntegerType()).withColumnRenamed('value','age')

df2 = sqlctx.createDataFrame([(0,18,'0-18'),(19,65,'19-65'),(65,99999,'65+')],['age_f','age_to','range'])

df_joined =df.join(df2,(df.age>= df2.age_f) & (df.age<=df2.age_to),'inner').select(df.age,df2.range).show()

Answer 2

嗨，你用sql函数udf和withColumn来做

from pyspark.sql.functions import udf
 age_range = udf(lambda age: '< 20' if age < 20 else 
                       '20-25' if (age >= 20 and age < 25) else
                       '25-30' if (age >= 25 and age < 30) else
                       '30-35' if (age >= 30 and age < 35) else
                       '35-40' if (age >= 35 and age < 40) else
                       '40-45' if (age >= 40 and age < 45) else
                       '45-50' if (age >= 45 and age < 50) else
                       '50-55' if (age >= 50 and age < 55) else
                       '55-60' if (age >= 55 and age < 60) else
                       '60-65' if (age >= 60 and age < 65) else
                       '65-70' if (age >= 65 and age < 70) else
                        '75+'  if (age >= 70) else '')

 df = df.withColumn('age_range', age_range(df.Age))