fix(python,rust): read_csv preserve whitespace and newlines

[Julian-J-S]

fixes #13933 and partly fix for #12763

addresses a few issues of read_csv (adding issue links later)

Fix 1: preserve whitespace at start of line (currently trimmed)

Currently read_csv trims whitespace at the start of each line (only for the first value). This is a bug. Whitespace should be preserved as is belongs to the value. Also this only happens for the first value.

DATA = """col
a
    b
        c
d"""
pl.read_csv(source=StringIO(DATA))

# Current result:
shape: (4, 1)
┌─────┐
│ col │
│ --- │
│ str │
╞═════╡
│ a   │
│ b   │
│ c   │
│ d   │
└─────┘

# NEW result (preserve whitespace, it belongs to the value!)
shape: (4, 1)
┌───────────┐
│ col       │
│ ---       │
│ str       │
╞═══════════╡
│ a         │
│     b     │
│         c │
│ d         │
└───────────┘

Fix 2: preserve newlines (see #13933 )

Currently read_csv removed empty lines if the csv has multiple columns and keeps empty lines as null for csv with a single column. All whitespace in csv files belongs to the values and should be preserved. Also this is currently inconsistent as described. In the future I can also add an option like pandas has to optionally skip empty lines.

if/when this is merged I can also add an parameter skip_empty_lines (like pandas). But default should be to preserve any form of whitespace as it belongs to the format.

# Single column (stays the same)
DATA = """col
a

c"""
pl.read_csv(source=StringIO(DATA))
shape: (3, 1)
┌──────┐
│ col  │
│ ---  │
│ str  │
╞══════╡
│ a    │
│ null │ <<<<<<<<<<<<<<<<<<< already correct for single column csv
│ c    │
└──────┘

# Multiple columns
DATA = """col1,col2
a,1

c,3"""
pl.read_csv(source=StringIO(DATA))

# Current (newline skipped)
shape: (2, 2)
┌──────┬──────┐
│ col1 ┆ col2 │
│ ---  ┆ ---  │
│ str  ┆ i64  │
╞══════╪══════╡
│ a    ┆ 1    │
│ c    ┆ 3    │
└──────┴──────┘

# New result (newline preserved)
shape: (3, 2)
┌──────┬──────┐
│ col1 ┆ col2 │
│ ---  ┆ ---  │
│ str  ┆ i64  │
╞══════╪══════╡
│ a    ┆ 1    │
│ null ┆ null │  <<<<<<<<<<<<<<<<<<< this is new! consistent with single column csv
│ c    ┆ 3    │
└──────┴──────┘

[ritchie46]

Yes, I agree. Thank you for the fix. Can you add those examples above as tests? We don't want to regress in the future.

[Julian-J-S]

Yes, I agree. Thank you for the fix. Can you add those examples above as tests? We don't want to regress in the future.

sure, done!

Added test for preserving whitespace at the start of the line (fix1)

There were already tests with empty lines. I had already adjusted the expected result to be consistent and treat empty lines as null.