Skip to content Skip to sidebar Skip to footer
Showing posts with the label Pyspark
Pivot Row To Column Level

Pivot Row To Column Level

Post a Comment
I have a spark dataframe t which is the result of a spark.sql('...') query. Here is the fir… Read more Pivot Row To Column Level
Flatten Nested Array In Spark Dataframe

Flatten Nested Array In Spark Dataframe

Post a Comment
I'm reading in some JSON on the from: {'a': [{'b': {'c': 1, 'd'… Read more Flatten Nested Array In Spark Dataframe
Unable To Write Pyspark Dataframe Created From Two Zipped Dataframes

Unable To Write Pyspark Dataframe Created From Two Zipped Dataframes

Post a Comment
I am trying to follow the example given here for combining two dataframes without a shared join key… Read more Unable To Write Pyspark Dataframe Created From Two Zipped Dataframes
Gcp Dataproc Custom Image Python Environment

Gcp Dataproc Custom Image Python Environment

Post a Comment
I have an issue when I create a DataProc custom image and Pyspark. My custom image is based on Data… Read more Gcp Dataproc Custom Image Python Environment
Spark-submit With Specific Python Librairies

Spark-submit With Specific Python Librairies

Post a Comment
I have a pyspark code depending on third party librairies. I want to execute this code on my cluste… Read more Spark-submit With Specific Python Librairies
Selecting Empty Array Values From A Spark Dataframe

Selecting Empty Array Values From A Spark Dataframe

Post a Comment
Given a DataFrame with the following rows: rows = [ Row(col1='abc', col2=[8], col3=[18]… Read more Selecting Empty Array Values From A Spark Dataframe