Pyspark Getting Weeknumber Of Month

I am stuckup with getting weeknumber from month in pyspark from a datafrme column , For Examples consider my dataframe as WeekID,DateField,WeekNUM 1,01/JAN/2017 2,15/Feb/2017 My O

Solution 1:

You can combine to_date and date_format:

from pyspark.sql.functions import to_date, date_format

df = spark.createDataFrame(
    [(1, "01/JAN/2017"), (2, "15/FEB/2017")], ("id", "date"))

df.withColumn("week", date_format(to_date("date", "dd/MMM/yyyy"), "W")).show()
+---+-----------+----+| id|date|week|+---+-----------+----+|1|01/JAN/2017|1||2|15/FEB/2017|3|+---+-----------+----+

If you want week-of-year please replace format with w:

date_format(to_date("date", "dd/MMM/yyyy"), "w")

Solution 2:

Since spark 3.0 flag w has been deprecated. So, Simply you can use PySpark inbuilt function weekofyear as follows -

import pyspark.sql.functions as funcs

(df
 .withColumn(
    'week_of_year', 
    funcs.weekofyear(funcs.col('date_announced'))).select('date_announced', 'week_of_year')
).show(5)

+-------------------+------------+
|     date_announced|week_of_year|
+-------------------+------------+
|2020-01-3000:00:00|           5|
|2020-02-02 00:00:00|           5|
|2020-02-03 00:00:00|           6|
|2020-03-02 00:00:00|          10|
|2020-03-02 00:00:00|          10|
+-------------------+------------+