Data format

[sakshamg94]

among the pms_pm_01_0 and similarly for other radiation and PM fields, the numbers have commas in them such that they land as strings when i import data from the CSVs
like 1,000 instead of 1000
commas are not supported in float dtype. so i think if json to csv results in figures with commas in them, then irrespective of the platform (I have checked on Python), we should see those inconsistencies

Here's how I fix it using Pandas library (Py):
df['pms_pm01_0'] = df['pms_pm01_0'].astype(str).str.replace(',', '').astype(float)

[matschaffer]

@sakshamg94 can you post a sample CSV with a comma in it? Could be something going on in the exporting step.

[sakshamg94]

see row. 1348 (pms payloads). Converted csv to xlsx for posting on github (csv not supported)
output-2020-07-01T00_00_00+00_00 2.xlsx

[matschaffer]

I’d recommend zipping the file for attachment to avoid the possibility of excel changing the formatting on its own (this happened with @nokton’s investigation)

[sakshamg94]

I see. good idea. Here (should be row 1347 or 1348)
output-2020-07-01T00_00_00+00_00.csv.zip