Python Programming Language

I have a spark dataframe t which is the result of a spark.sql('...') query. Here is the fir… Read more Pivot Row To Column Level

I'm reading in some JSON on the from: {'a': [{'b': {'c': 1, 'd'… Read more Flatten Nested Array In Spark Dataframe

I am trying to follow the example given here for combining two dataframes without a shared join key… Read more Unable To Write Pyspark Dataframe Created From Two Zipped Dataframes

I have an issue when I create a DataProc custom image and Pyspark. My custom image is based on Data… Read more Gcp Dataproc Custom Image Python Environment

I have a pyspark code depending on third party librairies. I want to execute this code on my cluste… Read more Spark-submit With Specific Python Librairies

Given a DataFrame with the following rows: rows = [ Row(col1='abc', col2=[8], col3=[18]… Read more Selecting Empty Array Values From A Spark Dataframe

In Scala I can do get(#) or getAs[Type](#) to get values out of a dataframe. How should I do it in … Read more Get Value Out Of Dataframe

I have a spark frame with two columns which looks like: +------------------------------------------… Read more How To Use Pandas Udf Functionality In Pyspark

I have a script with the below setup. I am using: 1) Spark dataframes to pull data in 2) Converting… Read more Convert A Pandas Dataframe To A Pyspark Dataframe

I am trying to wrap the constructor for pyspark Pipeline.init constructor, and monkey patch in the … Read more Wrapping Pyspark Pipeline.__init__ And Decorators

How to create a new column in PySpark and fill this column with the date of today? This is what I t… Read more Assertionerror: Col Should Be Column

I am trying to split a dataframe in pyspark This is the data i have df = sc.parallelize([[1, '… Read more Splitting A Column In Pyspark

In pyspark I have a column called test_time. This is a timestamp column. The column has a records l… Read more Adding 1 Hours To Time Stamp Columns In Pyspark Data Frame

I need to extract the integers only from url stings in the column 'Page URL' and append tho… Read more 'column' Object Is Not Callable With Regex And Pyspark

Below is my code. I have tried many different select variations, and yet the app runs, but without … Read more Pyspark Structured Streaming From Kafka Does Not Output To Console For Debugging

I am trying to solve the following error (I am using the databricks platform and spark 2.0) tweets_… Read more Pyspark Error With Udf: Py4j.py4jexception: Method __getnewargs__([]) Does Not Exist Error

I have a DataFrame: +-----+--------+---------+ | usn|log_type|item_code| +-----+--------+--------… Read more How To Make An Integer Index Row?

In the spark, I understand how to use wholeTextFiles and textFiles, but I'm not sure which to u… Read more How Does The Number Of Partitions Affect `wholetextfiles` And `textfiles`?

Pyspark: Split and select part of the string column values How can I select the characters or file … Read more Pyspark: Select Part Of The String(file Path) Column Values

I want to find the cleanest way to apply the describe function to a grouped DataFrame (this questio… Read more How To Apply The Describe Function After Grouping A Pyspark Dataframe?