Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grass.script: Automatically parse JSON and CSV in parse_command #3687

Merged
merged 4 commits into from
May 8, 2024

Conversation

petrasovaa
Copy link
Contributor

@petrasovaa petrasovaa commented May 3, 2024

Given several tools have options format, this adds automatic parsing to parse_command:

>>> data = gs.parse_command("r.what", map='elevation@PERMANENT', coordinates=[641567, 223962], format='json')
[{'easting': 641567, 'northing': 223962, 'site_name': '', 'elevation@PERMANENT': {'value': 75.82444763183594}}]

>>> data = gs.parse_command("v.db.select", map='busroute1@PERMANENT', format='csv')
[{'cat': '1', 'ROUTE': '1'}, {'cat': '2', 'ROUTE': '1'}]
>>> pd.DataFrame(data)
  cat ROUTE
0   1     1
1   2     1

For csv, I use DictReader, which makes sense only when the csv has a header, but that seems like reasonable assumption, and then it works nicely with pandas.

Option delimiter does not fit well into it conceptually, so I deprecated it. Also, it's now taken into account only when parameter parse is not specified, it assumed that anyway.

@petrasovaa petrasovaa added this to the 8.4.0 milestone May 3, 2024
@petrasovaa petrasovaa requested a review from wenzeslaus May 3, 2024 19:39
@petrasovaa petrasovaa self-assigned this May 3, 2024
@github-actions github-actions bot added Python Related code is in Python libraries labels May 3, 2024
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great update overall. The heuristic should work well the format parameter which in combination with the parse_command function will almost always refer to output text format.

A short change for what it accomplishes!

Thanks for changing to csv.DictReader, the purpose here is convenience, so DictReader seems more appropriate than plain reader.

I agree with converting the generator to a list. json.loads also returns plain objects, so it is consistent with that.

>>> type(json.loads("""{"a": 1}"""))
<class 'dict'>
>>> type(json.loads("""[1,2]"""))
<class 'list'>

The delimiter change is a good clean up.

Please, add tests.

@github-actions github-actions bot added the tests Related to Test Suite label May 6, 2024
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the tests! They nicely show how powerful this is.

@petrasovaa petrasovaa requested a review from wenzeslaus May 7, 2024 18:49
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome. Looking forward to using this. It will work great together with the JSON GSoC project.

@wenzeslaus wenzeslaus changed the title pythonlib/parse_command: automatically parse json and csv output grass.script: Automatically parse JSON and CSV output in parse_command May 8, 2024
@wenzeslaus wenzeslaus changed the title grass.script: Automatically parse JSON and CSV output in parse_command grass.script: Automatically parse JSON and CSV in parse_command May 8, 2024
@petrasovaa petrasovaa merged commit 79cf5fe into OSGeo:main May 8, 2024
26 checks passed
@petrasovaa petrasovaa deleted the improve-parse-command branch May 8, 2024 01:46
HuidaeCho pushed a commit to HuidaeCho/grass that referenced this pull request May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
libraries Python Related code is in Python tests Related to Test Suite
Projects
Development

Successfully merging this pull request may close these issues.

2 participants