PartitionedDataSet not picking up load_args

[SwePalm]

Hi,
i am new to Kedro, so still exploring all options, but now i found something that i am not able to google :-)

My use-case involves data that are partitioned by year/month/day/hour.
Where every hour includes multiple files.

I started to explore PartitionedDataSet for this.

My first test was to read a single hour.
The catalog looks like this:
json_from_dir:
type: PartitionedDataSet
path: data/01_raw/hour=01
dataset: pandas.JSONDataSet
load_args:
lines: True
convert_dates: False
filename_suffix: ".gz"

I have been using the load_args in my "old" code, and works there.
I test the catalog from kedro ipython and
catalog.load("json_from_dir")
This works great.

But when building and run a simple pipeline, with the dataset as input i get an ERROR, where i can see that the file is loaded with load_args={}
(and the error is thrown because it is json-lines, so lines=True has to be included).

Anyone with a suggestion on what to do?

A dataset with a single json-file works as input to a node. So pandas.JSONDataSet picks up the load_args correct.
So i suspect PartitionedDataSet, but could of course be wrong :-)

[datajoely]

Hi @SwePalm ! Would you mind taking a screenshot of how it looks in your IDE?

json_from_dir:
    type: PartitionedDataSet
    path: data/01_raw/hour=01
    dataset: pandas.JSONDataSet
    load_args:
        lines: True
        convert_dates: False
        filename_suffix: ".gz"

Does it look like this? It's a bit hard to see how you've pasted it.

[datajoely]

Also - it may be quicker to answer this on Discord

[SwePalm]

thanks for the pointer!

[SwePalm]

sorry, I am not able the take a screenshot, my path info are a bit revealing...what do you want to see more specific?
Final message is where i see that load_args is not included:
kedro.io.core.DataSetError: Failed while loading data from data set JSONDataSet(filepath=/...stuff_deleted.../data/01_raw/hour=01/my_file_name.gz, load_args={}, protocol=file, save_args={}).
Trailing data

[SwePalm]

OK, adding info from second test, this time with parquet files (instead of json). Did try to pass columns as load_args.
Works well for reading single file, but not when passed while using PartitionDataSet

And learnt that the error is thrown when trying the partition_load_func() even in kedro ipython (sorry for the initial confusion, me learning!)

[datajoely]

Hi @SwePalm - good that you're making progress. It would be easier to chat on Discord if possible. Otherwise could you please post the full stack trace?

[SwePalm]

thanks, will ping you when getting onboard Discord :-)

[datajoely]

Sweet - I'm there under the same username there :)

[SwePalm]

got a temp workaround, made a node reading files from generated list with pandas and strore concat dataset, DYI partitioneddataset :-)