How To Fix "MissingHeaders" Error While Appending Where Clause With S3 Select

I have a csv file in the format IDATE_TIMESTAMP,OPEN,HIGH,LOW,CLOSE,VOLUME 1535535060,94.36,94.36,94.36,94.36,1 1535535120,94.36,94.36,93.8,93.8,1 1535535180,93.8,93.8,93.8,93.8,0

Solution 1:

I know this is a bit late and might not be the solution to your issue, but I was having a similar one.

My issue turned out to be that I was attempting to perform an S3 Select on an object with UTF-8-BOM encoding, rather than just UTF-8. It turns out that the 3 byte BOM header was being interpreted as part of the first field in the CSV object, essentially corrupting the first column name.

As a result, rather than "IDATE_TIMESTAMP", the first column would be seen by the S3 Select call as "xxxIDATE_TIMESTAMP", causing an error when your expected column is "missing".

Solution 2:

The timestamp columns need to be casted as int. The following query:

fromTs = 1535535480
toTs = 1535535480

SELECT * FROM s3object s 
where cast(s.IDATE_TIMESTAMP as int) >= {} 
AND cast(s.IDATE_TIMESTAMP as int) <= {}".format(fromTs, toTs) 

would work in python 3.