Version 5.0.0

[jpivarski]

Uproot version 5.0.0

Uproot version 5 has a few major new features, one removal (uproot.lazy), and is based on Awkward Array version 2 instead of version 1.

uproot.lazy → uproot.dask

@kkothari2001 upgraded Uproot from Awkward version 1 to version 2, the major part of which was replacing uproot.lazy, which is based on Awkward 1's virtual and partitioned lazy arrays, with the new Dask collection, dask-awkward. The entry point for this function is uproot.dask.

@kkothari2001 also simplified Uproot's Pandas backend, which used to "explode" ragged arrays from ROOT into Pandas DataFrames with a non-trivial MultiIndex. Now, it takes advantage of awkward-pandas to put ragged (and more complex) Awkward Arrays directly into Pandas columns.

If you want the old behavior, you can read data using library="ak" to get an Awkward Array, and use ak.to_dataframe to "explode" the data into a MultiIndex.

TTree-reading with AwkwardForth

@aryan26roy added a new code path to the TTree-reading routines to read them with AwkwardForth instead of pure Python. Users won't see any interface changes due to this code, but the performance of reading TBranches with AsObject or AsStrings Interpretations should be orders of magnitude faster. For example, std::vector<std::vector<float>> reading is now 400× faster.

Reading RNTuples

@Moelf added a complete reader of RNTuple data with most of an RNTuple-writer in an unmerged pull request (#705). Although the RNTuple format is still in development, this is a very good start at reading RNTuple data, whose structure is a close match to Awkward Arrays (so the translation is more one-to-one than it is for TTrees, for instance).

New features

• feat: move to hatchling by @henryiii in feat: move to hatchling #688
• feat: from_map like optimization for dask arrays by @kkothari2001 in feat: from_map like optimization for dask arrays #679
• feat: Finalizing AwkwardForth reader for Uproot by @aryan26roy in feat: Finalizing AwkwardForth reader for Uproot #644
• feat: implemented NON-memberwise deserialization for AsMap. by @jpivarski in feat: implemented NON-memberwise deserialization for AsMap. #746
• feat: Added column_projection optimization by @kkothari2001 in feat: Added column_projection optimization #755
• feat: support categorical axes on boost histograms by @lobis in feat: support categorical axes on boost histograms #764
• feat: warn about TBranch name, alias name conflict. by @jpivarski in feat: warn about TBranch name, alias name conflict. #776
• feat: any Mapping assigned to a WritableDirectory is interpreted as a TTree or failure, no fall-through. by @jpivarski in feat: any Mapping assigned to a WritableDirectory is interpreted as a TTree or failure, no fall-through. #779
• feat: add 'interp_options' mechanism and ak_add_doc. by @jpivarski in feat: add 'interp_options' mechanism and ak_add_doc. #784
• feat: Use awkward pandas, instead of the existing code that explodes Pandas Dataframes by @kkothari2001 in feat: Use awkward pandas, instead of the existing code that explodes Pandas Dataframes #734
• feat: made 'very optional' arguments keyword-only by @jpivarski in feat: made 'very optional' arguments keyword-only #787
• feat: adjust for name change in feat: rename toXXX methods awkward#1919. by @jpivarski in feat: adjust for name change in scikit-hep/awkward#1919. #788

Bug-fixes and performance

• fix: depend on packaging, not setuptools vendored packaging by @henryiii in fix: depend on packaging, not setuptools vendored packaging #684
• fix: Avoid triggering temporary dask-awkward/awkward incompatibility. by @jpivarski in fix: Avoid triggering temporary dask-awkward/awkward incompatibility. #694
• fix: Do not write incorrect fSumw2 in histograms (v5). by @jpivarski in fix: Do not write incorrect fSumw2 in histograms (v5). #698
• fix: Fixes uproot.dask bug with empty branches by @kkothari2001 in fix: Fixes uproot.dask bug with empty branches #700
• fix: Use from_map optimization for delayed numpy arrays and add tests with empty branches for the same by @kkothari2001 in fix: Use from_map optimization for delayed numpy arrays and add tests with empty branches for the same #703
• fix: use ctx manager to ensure resources are freed by @agoose77 in fix: use ctx manager to ensure resources are freed #713
• fix: ReadOnlyDirectory should provide the largest abs(cycle) when cycle is unspecified, not the largest cycle. by @jpivarski in fix: ReadOnlyDirectory should provide the largest abs(cycle) when cycle is unspecified, not the largest cycle. #715
• fix: regularize ROOT type aliases to C fundamental type names. by @jpivarski in fix: regularize ROOT type aliases to C fundamental type names. #717
• fix: avoid empty TBasket issue in embedded TBasket by @jpivarski in fix: avoid empty TBasket issue in embedded TBasket #751
• fix: don't use Awkward in test_0751 that doesn't need it by @jpivarski in fix: don't use Awkward in test_0751 that doesn't need it #753
• fix: working TList serialization by @lobis in fix: working TList serialization #763
• fix: histogram weights not handled correctly in hist / boost conversion by @lobis in fix: histogram weights not handled correctly in hist / boost conversion #774
• perf: streamline metadata handling for TBranch name lookup and uproot.dask by @jpivarski in perf: streamline metadata handling for TBranch name lookup and uproot.dask #772
• fix: ensure AwkwardForth fallback path is tested without history. by @jpivarski in fix: ensure AwkwardForth fallback path is tested without history. #780
• fix: all AwkwardForth Forms now agree with awkward_form method output. by @jpivarski in fix: all AwkwardForth Forms now agree with awkward_form method output. #790
• fix: Uproot tests now work with Awkward 2.0.0. by @jpivarski in fix: Uproot tests now work with Awkward 2.0.0. #795

Other

• Manually add a Model for TMatrixTSym. by @jpivarski in Manually add a Model for TMatrixTSym<double>. #484
• Updating docs and test in response to the removal of uproot.lazy in Uproot5 by @kkothari2001 in Updating docs and test in response to the removal of uproot.lazy in Uproot5 #615
• changed arguments for awkward_form by @aryan26roy in changed arguments for awkward_form #617
• Completed the Forth based AsStrings reader. by @aryan26roy in Completed the Forth based AsStrings reader. #616
• Actually pass user-specified 'awkward_form' arguments into context. by @jpivarski in Actually pass user-specified 'awkward_form' arguments into context. #622
• Fix annoying gaps in test files. by @jpivarski in Fix annoying gaps in test files. #625
• Cleaning up string generation in streamers.py by @aryan26roy in Cleaning up string generation in streamers.py #629
• chore: add dependabot for actions by @henryiii in chore: add dependabot for actions #631
• Bump actions/checkout from 2 to 3 by @dependabot in Bump actions/checkout from 2 to 3 #632
• Bump actions/upload-artifact from 2 to 3 by @dependabot in Bump actions/upload-artifact from 2 to 3 #633
• Bump pypa/gh-action-pypi-publish from 1.4.2 to 1.5.0 by @dependabot in Bump pypa/gh-action-pypi-publish from 1.4.2 to 1.5.0 #634
• Bump actions/download-artifact from 2 to 3 by @dependabot in Bump actions/download-artifact from 2 to 3 #635
• Forth based ROOT reader (revised) by @aryan26roy in Forth based ROOT reader (revised) #636
• Set up tests for AsObjects, for the AwkwardForth reader by @jpivarski in Set up tests for AsObjects, for the AwkwardForth reader #637
• Iterate over objects in TDirectory in linear time. by @jpivarski in Iterate over objects in TDirectory in linear time. #638
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #641
• Awkward v2 update by @kkothari2001 in Awkward v2 update #620
• docs: add aryan26roy as a contributor for code by @allcontributors in docs: add aryan26roy as a contributor for code #645
• docs: add kkothari2001 as a contributor for code by @allcontributors in docs: add kkothari2001 as a contributor for code #646
• docs: add Moelf as a contributor for code by @allcontributors in docs: add Moelf as a contributor for code #647
• chore: cleanup flake8 by @henryiii in chore: cleanup flake8 #527
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #650
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #654
• Bump pypa/gh-action-pypi-publish from 1.5.0 to 1.5.1 by @dependabot in Bump pypa/gh-action-pypi-publish from 1.5.0 to 1.5.1 #656
• Primitive Support for RNTuple by @Moelf in Primitive Support for RNTuple #630
• Set ReadOnlyDirectory attributes when fSeekKeys == 0 by @kakwok in Set ReadOnlyDirectory attributes when fSeekKeys == 0 #660
• docs: add kakwok as a contributor for code by @allcontributors in docs: add kakwok as a contributor for code #663
• Dask awkward support for uproot.dask by @kkothari2001 in Dask awkward support for uproot.dask #652
• pathlib.Path drops '//' (naturally), but it's sometimes used for URLs by @jpivarski in pathlib.Path drops '//' (naturally), but it's sometimes used for URLs #670
• Gets the number of overflow bins for hist.axis.IntCategory, at least. by @jpivarski in Gets the number of overflow bins for hist.axis.IntCategory, at least. #671
• Prevent std::pair from being AsStridedObjects. by @jpivarski in Prevent std::pair from being AsStridedObjects. #673
• Implement transformed axis from boost-histogram/hist. by @jpivarski in Implement transformed axis from boost-histogram/hist. #675
• AsDynamic has no self._header. by @jpivarski in AsDynamic has no self._header. #674
• Fixed TTree write_anew in a subdirectory (consistent caches). by @jpivarski in Fixed TTree write_anew in a subdirectory (consistent caches). #677
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in [pre-commit.ci] pre-commit autoupdate #657
• Implement stl containers for RNTuple by @Moelf in Implement stl containers for RNTuple #662
• Multiple clusters support for RNTuple by @Moelf in Multiple clusters support for RNTuple #682
• ci: Get test dependencies from one source by @jpivarski in ci: Get test dependencies from one source #686
• ci: autocancel repeated runs by @henryiii in ci: autocancel repeated runs #685
• ci: use mamba by @henryiii in ci: use mamba #683
• ci: Lint PR titles according to conventional commits by @jpivarski in ci: Lint PR titles according to conventional commits #689
• docs: Installation requirements and error text for dask/dask-awkward in extras.py by @jpivarski in docs: Installation requirements and error text for dask/dask-awkward in extras.py #690
• ci: use concurrency group for semantic-pr-title by @agoose77 in ci: use concurrency group for semantic-pr-title #691
• docs: Add image for dask docs by @kkothari2001 in docs: Add image for dask docs #708
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in ci: [pre-commit.ci] pre-commit autoupdate #706
• chore: dask_awkward.test_utils moved in ContinuumIO/dask-awkward#76 by @jpivarski in chore: dask_awkward.test_utils moved in ContinuumIO/dask-awkward#76 #714
• test: adjust for boost-histogram 1.3.2's _storage_type deprecation. by @jpivarski in test: adjust for boost-histogram 1.3.2's _storage_type deprecation. #719
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in ci: [pre-commit.ci] pre-commit autoupdate #716
• add veprbl as a contributor by @allcontributors in add veprbl as a contributor #728
• add nikoladze as a contributor by @allcontributors in add nikoladze as a contributor #729
• add klieret as a contributor by @allcontributors in add klieret as a contributor #730
• add dcervenkov as a contributor by @allcontributors in add dcervenkov as a contributor #731
• add beojan as a contributor by @allcontributors in add beojan as a contributor #732
• add agoose77 as a contributor by @allcontributors in add agoose77 as a contributor #733
• docs: add a CITATION.cff for Uproot. by @jpivarski in docs: add a CITATION.cff for Uproot. #726
• chore: drop Python 3.6. by @jpivarski in chore: drop Python 3.6. #742
• ci: remove GHA workaround for macOS Python 3.11 by @henryiii in ci: remove GHA workaround for macOS Python 3.11 #743
• chore: some cleanup inpsired by refurb by @henryiii in chore: some cleanup inpsired by refurb #745
• chore(deps): update pre-commit hooks by @pre-commit-ci in chore(deps): update pre-commit hooks #737
• chore(deps): bump amannn/action-semantic-pull-request from 4 to 5 by @dependabot in chore(deps): bump amannn/action-semantic-pull-request from 4 to 5 #757
• chore: update Uproot to require Awkward 2.0.0rc1. by @jpivarski in chore: update Uproot to require Awkward 2.0.0rc1. #765
• ci: update to Python 3.11 final by @henryiii in ci: update to Python 3.11 final #766
• docs: add lobis as a contributor for code by @allcontributors in docs: add lobis as a contributor for code #771
• refactor: refactor Forth generation by @aryan26roy in refactor: refactor Forth generation #710
• chore: remove Identifier and "uproot" parameter. by @jpivarski in chore: remove Identifier and "uproot" parameter. #770
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in ci: [pre-commit.ci] pre-commit autoupdate #773
• refactor: final refactoring for Forth generation by @aryan26roy in refactor: final refactoring for Forth generation #749
• docs: fix TRef.py doc urls by @veprbl in docs: fix TRef.py doc urls #782
• ci: pre-commit autoupdate by @pre-commit-ci in ci: pre-commit autoupdate #783
• docs: uproot.dask docs by @kkothari2001 in docs: uproot.dask docs #702
• test: make tests parallelizable (custom_classes in uproot.open). by @jpivarski in test: make tests parallelizable (custom_classes in uproot.open). #786
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in ci: [pre-commit.ci] pre-commit autoupdate #789
• chore(deps): bump pypa/gh-action-pypi-publish from 1.5.1 to 1.6.1 by @dependabot in chore(deps): bump pypa/gh-action-pypi-publish from 1.5.1 to 1.6.1 #792
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in ci: [pre-commit.ci] pre-commit autoupdate #793

New Contributors

• @aryan26roy made their first contribution in changed arguments for awkward_form #617
• @dependabot made their first contribution in Bump actions/checkout from 2 to 3 #632
• @Moelf made their first contribution in Primitive Support for RNTuple #630
• @agoose77 made their first contribution in ci: use concurrency group for semantic-pr-title #691
• @lobis made their first contribution in fix: working TList serialization #763

Full Changelog: 4.2.4...v5.0.0

---

This discussion was created from the release Version 5.0.0.