Behaviour of resource.write after steps.table_normalize

[fjuniorr]

I'm trying to use frictionless transform to standardize data sources that use different representations for the same data (think dates in different formats).

For example, for the csv file described by a proper table schema:

I would like to generate the following csv file:

I can achieve this by exporting the csv with petl:

from frictionless import Resource, transform, steps

url = 'https://raw.githubusercontent.com/splor-mg/datapackage-reprex/main/20230711T100325/resource.yaml'
resource = Resource(url)
transform(resource, steps=[steps.table_normalize()])
resource.to_petl().tocsv('output-petl.csv')

but I'm wondering if I shoudn't be able to do the same with resource.write

from frictionless import Resource, transform, steps

url = 'https://raw.githubusercontent.com/splor-mg/datapackage-reprex/main/20230711T100325/resource.yaml'
resource = Resource(url)
transform(resource, steps=[steps.table_normalize()])
resource.write('output-frictionless.csv')

which currently generates the following csv file:

Note that resource.write did converted the missingValue from NA to a empty char in the memo field without adjusting the tableschema.

[fjuniorr]

Just to note that

transform(resource, steps=[steps.table_normalize(), steps.table_write(path='table_write.csv')])

also does not output the normalized data.

[roll]

Thanks! I created an issue to investigate frictionlessdata/frictionless-py#1567

[fjuniorr]

Hi @roll! Could you expand a little more your thinking into way this should be the behaviour of resource.write?

For me, if the in-memory representation of the data has changed from

[['id', 'valid', 'amount', 'memo', 'valid_from', 'valid_to', 'updated_at'],
 ['1',
  'no',
  '120',
  'some information',
  '01/02/1995',
  '01/02/1995',
  '30/11/1994 09:45:00'],
 ['2',
  'yes',
  'R$ 2.000,5',
  'NA',
  '01/02/2002',
  '01/02/2002',
  '30/09/2002 10:45:00']]

to (after the call to table_normalize)

[['id', 'valid', 'amount', 'memo', 'valid_from', 'valid_to', 'updated_at'],
 [1,
  False,
  Decimal('120'),
  'some information',
  datetime.date(1995, 1, 2),
  datetime.date(1995, 2, 1),
  datetime.datetime(1994, 11, 30, 9, 45)],
 [2,
  True,
  Decimal('2000.5'),
  None,
  datetime.date(2002, 1, 2),
  datetime.date(2002, 2, 1),
  datetime.datetime(2002, 9, 30, 10, 45)]]

why would resource.write not export Decimal and datetime as a standard csv python export:

data = resource.read_cells()
import csv
with open('data.csv', 'w') as csvfile:
    # creating a csv writer object
    csvwriter = csv.writer(csvfile)
    csvwriter.writerow(data[0])
    csvwriter.writerows(data[1:])

[roll]

@fjuniorr
It uses the source schema, so we need to provide a target schema to change the format (steps.table_write(path=path, schema=new_schema))

[fjuniorr]

I don't think I follow @roll. steps.table_write does't look like it takes a schema argument and indeed I get:

from frictionless import Resource, transform, steps, Schema

resource = Resource('https://raw.githubusercontent.com/splor-mg/datapackage-reprex/main/reprex/20230711T100325/resource.yaml')
target_schema = Schema('https://raw.githubusercontent.com/splor-mg/datapackage-reprex/main/reprex/20230711T100325/target.yaml')

transform(resource, steps=[steps.table_normalize(), steps.table_write(path='output.csv', schema=target_schema)])

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: table_write.__init__() got an unexpected keyword argument 'schema'

with frictionless 5.15.10.

[roll]

@shashigharti Can you please take a look?