Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor to use polars #244

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open

Refactor to use polars #244

wants to merge 39 commits into from
+3,534 −2,862

Commits on Jun 14, 2024

  1. polars testing

    @jcadam14
    jcadam14 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    267153f View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. More polars updates

    @jcadam14
    jcadam14 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    a873e2e View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. More polars updates

    @jcadam14
    jcadam14 committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    e83221e View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. More polars testing

    @jcadam14
    jcadam14 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    cfe754a View commit details
    Browse the repository at this point in the history

  2. Rewrite of function for better readability.

    @jcadam14
    jcadam14 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    ba37110 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. Updates to data formatters

    @jcadam14
    jcadam14 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    92c32d5 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Switch to polars batching and generators

    @jcadam14
    jcadam14 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    06bd9a8 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Final updates for the polars conversion. Comments, cleaned up code, etc

    @jcadam14
    jcadam14 committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    de78658 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into test_polars

    @jcadam14
    jcadam14 committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    eff3884 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. Merge branch 'main' into test_polars

    @jcadam14
    jcadam14 committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    d4e066f View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. Testing new mac commits

    @jcadam14
    jcadam14 committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    1cdf338 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Removed psutil

    @jcadam14
    jcadam14 committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    b76995d View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Updated to latest polars

    @jcadam14
    jcadam14 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    c2001b1 View commit details
    Browse the repository at this point in the history

  2. Added polars-lts-cpu for ARM support

    @jcadam14
    jcadam14 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    a8c4f81 View commit details
    Browse the repository at this point in the history

  3. Removed polars in favor of polars-lts-cpu

    @jcadam14
    jcadam14 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    2a73355 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. Updated report writer from sink_csv to write_csv

    @jcadam14
    jcadam14 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    186a7e9 View commit details
    Browse the repository at this point in the history

  2. Removed memory resource testing code

    @jcadam14
    jcadam14 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    7ce9068 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into test_polars

    @jcadam14
    jcadam14 committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    c179b60 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Changed to pull severity value, not enum

    @jcadam14
    jcadam14 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    4324951 View commit details
    Browse the repository at this point in the history

  2. Trying enum again

    @jcadam14
    jcadam14 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f427134 View commit details
    Browse the repository at this point in the history

  3. Using string instead of enum in dataframe

    @jcadam14
    jcadam14 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    5dfe595 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Trying out boto3 over fsspec

    @jcadam14
    jcadam14 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    82a36a2 View commit details
    Browse the repository at this point in the history

  2. S3 bucket updated

    @jcadam14
    jcadam14 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    a85667e View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Added truncating errors for json processing

    @jcadam14
    jcadam14 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    df6e246 View commit details
    Browse the repository at this point in the history

  2. Fixing row error

    @jcadam14
    jcadam14 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    37b03d3 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Updated to try and shrink down the memory footprint of the dataframes

    @jcadam14
    jcadam14 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    452fbd9 View commit details
    Browse the repository at this point in the history

  2. Revert boto3 to 1.34

    @jcadam14
    jcadam14 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    2f0493d View commit details
    Browse the repository at this point in the history

  3. Added phase to dataframe

    @jcadam14
    jcadam14 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    0089588 View commit details
    Browse the repository at this point in the history

  4. Fixed phase bug

    @jcadam14
    jcadam14 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    1ced8ce View commit details
    Browse the repository at this point in the history

  5. Added scope back to dataframe

    @jcadam14
    jcadam14 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    d81ff0d View commit details
    Browse the repository at this point in the history

  6. missing scope in final frame

    @jcadam14
    jcadam14 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    e66a4e1 View commit details
    Browse the repository at this point in the history

  7. Trying to use dicts instead of dataframes

    @jcadam14
    jcadam14 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    3efa1bf View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. switched back to returning a dataframe

    @jcadam14
    jcadam14 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    515bca6 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Reworked validator to allow for correct real and max errors

    @jcadam14
    jcadam14 committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    ca9b9c7 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Added default max size per json group

    @jcadam14
    jcadam14 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    1277923 View commit details
    Browse the repository at this point in the history

  2. Linting, and fixed pytests

    @jcadam14
    jcadam14 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    98f1160 View commit details
    Browse the repository at this point in the history

  3. Had to add s3fs

    @jcadam14
    jcadam14 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    bd7a2e6 View commit details
    Browse the repository at this point in the history

  4. Updated poetry.lock

    @jcadam14
    jcadam14 committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    f2c9412 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2024

  1. Fixed ordering of records

    @jcadam14
    jcadam14 committed Oct 5, 2024
    Configuration menu
    Copy the full SHA
    566e63c View commit details
    Browse the repository at this point in the history