Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimizing read_geopkg function #819

Merged
merged 4 commits into from
Aug 14, 2024

Conversation

AminTorabi-NOAA
Copy link
Contributor

This PR improves the speed of read_geopkg function, with below changes. :
(It improve the Duration by 20% and CPU Time by 50%)

  1. Optimized Parallel Processing.
    Used min(cpu_pool, len(layers)) to ensure we don't create more parallel jobs than layers, which avoids unnecessary overhead.

  2. Reduced Repetition:
    Removed repeated code for reading layers by creating a read_layer function.

  3. Ensured Efficiency:
    Removed unnecessary list creation and directly accessed the table_dict.

Additions

Removals

Changes

Originally it was taking 2.47 sec
image

after the changes it takes 1.96 sec which is 20% improvement:
image

Testing

Screenshots

Notes

Todos

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Visually tested in supported browsers and devices (see checklist below 👇)
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

  • Windows
  • Linux
  • Browser

Accessibility

  • Keyboard friendly
  • Screen reader friendly

Other

  • Is useable without CSS
  • Is useable without JS
  • Flexible from small to large screens
  • No linting errors or warnings
  • JavaScript tests are passing

Comment on lines 39 to 40
'flowpaths': r'flow[-_]?paths?',
'flowpath_attributes': r'flow[-_]?path[-_]?attributes?',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to drop the "paths" here. I believe these can be called "flowlines" which this would not find.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if we drop the "paths" and make it to just look for "flow" it will find both "flowpaths" layer and "flowpath_attribute" layer. I just added "flowlines" too, that should solve the problem

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great. Can you do the same for flowline-attributes? I'm not sure if that's a potential table name, but we might as well cover our bases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Just added

@shorvath-noaa shorvath-noaa merged commit 7a6e25d into NOAA-OWP:master Aug 14, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants