Skip to content

Commit

Permalink
Julearn sk pandas (#183)
Browse files Browse the repository at this point in the history
* MODIFY Internals initial

* new api examples

* MODIFY Internals initial

* Make JuTransformer work

* Hyper Tuning + new API working

Co-authored-by: Sami Hamdan <[email protected]>

* fix for deslib import

* Estimators test OK + no more binary/multiclass

* model selection tests ok

* scoring tests OK

* merge conflict

* transformers + utils test ok

* Fix CI

* codespell

* small clean up

* fix do_scoring tests

* Stacked classifiers

* For Sami to fix

* Fix rebase

* Fix pipeline tests

* Improve SetColumnTypes

* Improve some Tests on X_types

* Make Stacking work with filtering

* example ready

* Make stares shine as regexes

* Adjust warnings and errors +testing for pipeline.py

* Make models use apply_to and needed_types

* fixed print

* remove old returned features from julearn

* rename and clean julearn.estimators to julearn.models

* remove default apply_to from available transformers

* Modify prepare clean up a bit

* Adjust confound removal, get rid of pick_columns, modify JuModel, JuTransformer, JuBaseEstiamtor

* refactor juestimators DataFrameConfoundRemover add ColumnTypes

* Modify examples

* Modify/Add important Docstrings

* Correct: check consitency imports

* Example/stratifiedkfoldcv (#181)

* Adjust example stratifiedkfoldcv

* Integrade ColumnTypes in pipeline

* Oops left print

* Refactor Structure to have a base module for ColumnTypes and JuBaseClasses

* Move JuColumnTransformer and make setting default of apply to in PipelienCreator possible

* Fix tests for old versions

* Add preprocess inspection tool

* Modify inspect

* Modify inspect

* get the right dependency for CI

* Fix lint

* Fix codespell

* Fix test api, but Sami broke it :(

* Trigger CI

* Fix coverage

* Trigger CI

* Adjust dataclass for python=3.11

* Bring back return train scores

* updated example to inspect preprocessing steps (#182)

* Add docstrings to column_types

* Fix for X_types checks in pipeline

* add/update docstrings to DataFrameConfoundRemover (#184)

* add/update docstrings to DataFrameConfoundRemover

* Make some test green again

* Adjust preprocess to remove column_types per default and improve import

* Fix tests

* Bringing back the prepare testing

* Sk pandas examples (#185)

* updated multiclass classification example

* updated inspect SVM example

Co-authored-by: More, Shammi <[email protected]>
Co-authored-by: Sami Hamdan <[email protected]>
Co-authored-by: Fede Raimondo <[email protected]>

* Adjust target transforming

* Fix building doc

* disable examples

* fix stacked classifier example

* Fix linter

* update: add default apply_to to pipeline creator

* LINTEEER

* update: combine pandas examples running

* Fix codespell

* Basic regress example (#188)

* fixed regression example

* formatted with black

* One more example

* One more example

* Random forest example

* Stratified kfold for regression example

* Make Scoring with classification work (#191)

* Make Scoring with classificication work

* flake

* Refactor column types

* Flake8

* Fix PipelineCreator and problem type issues

* Linter

* chore: update `pyproject.toml` (#189)

* fix: use text instead of file for license to make PyPI happy

* fix: correct format for authors and maintainers in pyproject.toml

* chore: use lowercase for keywords in pyproject.toml

* chore: make package classifier development status to 5 in pyproject.toml

Co-authored-by: Fede Raimondo <[email protected]>

* Add example transforming the target variable with zscore (#197)

* Add example transforming the target variable with zscore

* Correcting script with flake8

* Correct white space in title

* Adding = to title

* fix syntax

* Removing white lines to new syntax

* Removing white line for title

* Remove useless plot

Co-authored-by: Paas Oliveros, Lya Katarina <[email protected]>
Co-authored-by: Fede Raimondo <[email protected]>

* update confound removal example (#196)

* update confound removal example

* change name of file

* change name of file

* Fix confound removal

* Spelling

Co-authored-by: Fede <[email protected]>

* Sk pandas examples (#190)

* added pipeline creator

* updated multiclass classification example

* updated inspect SVM example

* updated multiclass classification example

* formatted dis_plot_groupcv_inspect_svm.py

* formatted plot_cm_acc_multiclass.py

Co-authored-by: More, Shammi <[email protected]>
Co-authored-by: Fede Raimondo <[email protected]>

* Fix hyperparameter tuning and remove "wrapping" from model and transformer name (#200)

* Fix Hyperparameter tuning
* Remove Wrapper naming convention

Co-authored-by: Sami Hamdan <[email protected]>

* Example pca featsets (#198)

* [not working] example for two PCAs

* Fix double PCA issue

* [working] multiple PCA steps

* Fix remainder in name

* Add a name in PipelineCreator.add

* [working] multiple PCA steps with preprocess showcase

* rename

* rename

* Codespell

* Fix doc

* Fix scoring

Co-authored-by: Fede <[email protected]>

* Adjust scoring to work with registered once (#203)

* Adjust scoring to work with registered once

* Revert back the problem type to not be overwritable

* Revert back the problem type to not be overwritable

* Grouped cv example (#205)

* Added new file for grouped CV.

* Created example of Grouped CV.

* deleted the old file of Grouped CV and created the new file for that.

* Target transformer (#215)

* WIP: Target transfomers + pipeline

* WIP

* damn git ignore

* The example works

* A todo for @samihamdan

* Type hints + docstrings in progress

* Base module ready

* Typing + docstrings of conftest

* Tests for inspect preprocessing + more types

* Fix linting

* model selection with types and docstrings

* Fix linter

* More tests with docstrings and typing hints

* linter

* Models sub-package done

* Some more tests + typing + docstrings for the pipeline module

* tests + docstrings + types for TargetPipelineCreator

* Almost done with tranformers.target, but got tired...

* Test for target confound remover done

* Some WIP... go sick and I don't remember where I was

* Finished testing confounds

* JuColumnTransformer tested

* More fixes

* Updated prepare and tests

* Updates to the docstings

* Basic API test

* I think the API is tested for the moment

* All test pass, no warnings

* Updated examples

* fix linting

* Codespell

* codespell doesn't like the new verb we invented

* ENH: allow to set X_types using regexp

* some symmetry in the log

* Fix tests

* Fix for leo

* WIP: model comparison

* add corrected t_test

* Add dependencies + fix other tets

* fix test for user-specified splits

* fix test

* chore: improve ci-docs.yml

* chore: improve ci.yml

* chore: improve docs-preview.yml

* chore: improve docs.yml

* chore: improve pypi.yml

* chore: isort

* chore: black

* chore: flake8 fixes

* Retry

* fix coverage

* Add deslib for testing in tox

* increase coverage

* test api with apply_to

* Cbpm sum (#221)

* Minimal version of CPM without weighting but sum as default, only old test adjusted

* Flake8

* lint

* Resolve set_output bug in cbpm

* Fix lint

* Julearn sk pandas row select (#220)

* Add row_select_col

* Add test row_select simple

* Adjust some logic

* Lint

* Adjust row selection to be accessible from PipelineCreator

* Adjust row selection via column type not type.

* Test and fix change_column_types transformer

* fix #213 and #141

* add return confound example

* fix groups

* fix for [ENH] Validate run_cross_validate parameters #178

* fix for #214

* Inspection (#223)

* ADD inspection tools

* Add get_fitted_params to PipelineInspector

* Add bugfix #219

* Fix preprocess import

* adjust pipe

* adjust pipe

* CV Inpsection WIP

* [WIP] Inspection tool (needs testing)

* sort the index

* Fix tests and ling

* Add some test and func for _cv

* Add inspection and test, fix predict_log_proba remove unnecessary todo notes

* Add tests inspector

* Add tests

* New docs (#228)

* Update docs theme to furo

* Add content to index

* Add new chapter

* modify requirements

* Add copybutton extension

* Add installation related info to get started

* Installation was replaced by getting started

* setup structure for chapter 4

* Add messy chapter 3 to push uptodate

* Number files based on chapter order

* Add basic index for chapter 4 and reference

* Change numbering to display in real time

* Add chapter 4 to toctree

* Comment data.rst to make build possible

* Work in progress push for Leo

* initial cbpm

* some formatting

* add introductory paragraph for confound removal and some references

* add docs on confound removal

* fix typo

* capitalise titles

* Work in progress push Leo

* Work in progress push for Leo

* Clean up made changes

* data section preliminary version done

* available pipeline steps

* clean the docs

* correct layout error

* structure and add references

* modify indices

* Update docs theme to furo

* Add content to index

* Add new chapter

* modify requirements

* Add copybutton extension

* Add installation related info to get started

* Installation was replaced by getting started

* setup structure for chapter 4

* Add messy chapter 3 to push uptodate

* Number files based on chapter order

* Add basic index for chapter 4 and reference

* Change numbering to display in real time

* Add chapter 4 to toctree

* Comment data.rst to make build possible

* Work in progress push for Leo

* initial cbpm

* some formatting

* add introductory paragraph for confound removal and some references

* add docs on confound removal

* fix typo

* capitalise titles

* Work in progress push Leo

* Work in progress push for Leo

* Clean up made changes

* data section preliminary version done

* available pipeline steps

* clean the docs

* correct layout error

* structure and add references

* modify indices

* WIP: Docs fixing, rearanging API

* restructure

* Add description of pipeline

* Update examples structure + API reference

* work in progress

* Some refactoring of the what really need know section

* fix CI

* remove comments

---------

Co-authored-by: Vera Komeyer <[email protected]>
Co-authored-by: LeSasse <[email protected]>

* refactor index

* Add general CV information

* Add target preprocessing and refactor

* add reference

* Add model evaluation illustration pictures

* Add further needed scikitlearn links

* Correct typo

* Add cross-reference link

* Add cv-splitter as potential further selcted topic

* Add model evaluation content

* Fix doc sectioning and order

* Fix TOCTREE

* One less numbered element in the TOC

* Add/viz (#227)

* VIZ WIP

* add viz to the dependencies for docs

* add viz dependencies to tox

* fix linter

* delete old files

* add example back

* delete this file

* More doc on VIZ

* Add stats to viz

* Finish with VIZ docs

* Fix linter

* Fix tests

* Adaptations for proper toctree

* Rearrange content to solve toctree

* Minor clean-ups

* Add visualization information

* Fix building

* added initial version of stacking documentation for julearn

* fix markup in cbpm

* some slight improvements to stacked model documentation

* Adjust tests and fix bugs for dataframe transformers

* fix bug no hyperparameters

* fix bug hyperparams with target

* fix bug hyperparams with target

* ADD tests not fitted error target transformer

* ADD test returning inspection tool with differen return_estimator settings

* ADD test rais errors pipeline

* MODIFY insect to allow for gridsearch

* initial start with model inspection

* initial version of model_inspect.rst

* a few small typo fixes

* fix bug in which estimators were not picklable due to local function definition inside a function in base/column_types.py

* whitespaces

* apparently codespell does not like .sav as a file extension and thinks it should be save. Calling it .joblib instead

* [CI]: Use towncrier for news (#222)

* update: add towncrier configuration in pyproject.toml

* docs: refactor whats_new.rst for towncrier integration

* docs: improve contributing.rst and add towncrier instructions

* docs: improve maintaining.rst and add towncrier instructions

* update: add docs/changes/newsfragments

* chore: add 151.bugfix changelog

* chore: add 170.enh changelog

* update: improve docs/Makefile and add towncrier integration

* chore: delete Makefile

* update: add towncrier to pyproject.toml

* [ENH] Allow list of hyperparameter options for tunning (#47)

* [WIP] Multiple hyperparam grids

* fix linter

* increase coverage

* add more tests

* More tests for merger

* add news fragemtn

* Add specific gallery section for doc pages that need to be run.

* Docs on hyperparameter tuning

* some more updates

* FIX predict proba

* Update dependencies

* Fix CV + update first chapter of docs

* Fix tests

* codespell

* fix test for 3.8 maybe

* Fix inspector

* fix inpsecto

* Fix issues with doc building

* Improving some titles

* Update intesphinx links

* Fix inspection + docs

* Docs (almost) ready

* Fix codespell

* Update example of stratification for regression

* Update dependencies

* Fix the run_cross_validation docstring

* Fix test for stats

* MODIFY Target Transformation

* MODIFY K-fold for regression

* Fix for scikit-learn 1.3.0

* Fix whats'new

* Maybe ready for release?

* Lint

* codespell

* Add pearson's R scorer

* ADD: folds inspector prediction returns target value + avoid regexp when no patterns in X

* Set column types even if input is an array

* WIP: Config module

* Add tests for new config flags

* Update X Warns to 5k

* linter

* update doc

* Typo in DOC

---------

Co-authored-by: Sami Hamdan <[email protected]>
Co-authored-by: Leonard Sasse <[email protected]>
Co-authored-by: Shammi270787 <[email protected]>
Co-authored-by: More, Shammi <[email protected]>
Co-authored-by: antogeo <[email protected]>
Co-authored-by: Synchon Mandal <[email protected]>
Co-authored-by: Lya K. Paas <[email protected]>
Co-authored-by: Paas Oliveros, Lya Katarina <[email protected]>
Co-authored-by: Vera Komeyer <[email protected]>
Co-authored-by: kaurao <[email protected]>
Co-authored-by: Kimia Nazarzadeh <[email protected]>
Co-authored-by: Vera Komeyer <[email protected]>
  • Loading branch information
13 people authored Jul 19, 2023
1 parent c3ec513 commit 8a58646
Show file tree
Hide file tree
Showing 244 changed files with 20,663 additions and 8,221 deletions.
7 changes: 0 additions & 7 deletions .codespellrc

This file was deleted.

12 changes: 0 additions & 12 deletions .coveragerc

This file was deleted.

5 changes: 0 additions & 5 deletions .flake8

This file was deleted.

50 changes: 50 additions & 0 deletions .github/workflows/ci-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Run Tests

on:
push:
paths-ignore:
- 'julearn/**'
- '.github/workflows/**'
pull_request:
paths-ignore:
- 'julearn/**'
- '.github/workflows/**'
jobs:
run-tests:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ['3.8', '3.9', '3.10', '3.11']
steps:
- run: 'echo "No build required"'
- name: Create mock coverage.xml file
run: |
mkdir docs
touch docs/conf.py
echo '<?xml version="1.0" ?>' > coverage.xml
echo '<coverage version="6.5.0" timestamp="$(date +%s%3N)" lines-valid="1" lines-covered="1" line-rate="1" branches-valid="1" branches-covered="1" branch-rate="1" complexity="0">' >> coverage.xml
echo '<sources>' >> coverage.xml
echo '<source>/home/runner/work/docs/docs</source>' >> coverage.xml
echo '</sources>' >> coverage.xml
echo '<packages>' >> coverage.xml
echo '<package name="docs" line-rate="1" branch-rate="1" complexity="0">' >> coverage.xml
echo '<classes>' >> coverage.xml
echo '<class name="conf.py" filename="docs/conf.py" complexity="0" line-rate="1" branch-rate="1">' >> coverage.xml
echo '<methods/>' >> coverage.xml
echo '<lines>' >> coverage.xml
echo '<line number="7" hits="1"/>' >> coverage.xml
echo '</lines>' >> coverage.xml
echo '</class>' >> coverage.xml
echo '</classes>' >> coverage.xml
echo '</package>' >> coverage.xml
echo '</packages>' >> coverage.xml
echo '</coverage>' >> coverage.xml
if: matrix.python-version == 3.10
- name: Upload mock coverage to Codecov
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: true
flags: docs
if: success() && matrix.python-version == 3.10
50 changes: 25 additions & 25 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,41 +1,41 @@
name: Run Tests

on: [push, pull_request]
on:
push:
paths:
- 'julearn/**'
- '.github/workflows/**'
pull_request:
paths:
- 'julearn/**'
- '.github/workflows/**'

jobs:
build:

runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ['3.7', '3.8', '3.9', '3.10']
python-version: ['3.8', '3.9', '3.10', '3.11']

steps:
- uses: actions/checkout@v2
- name: Checkout repository
uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r test-requirements.txt
pip install -r requirements.txt
- name: Install julearn
shell: bash -el {0}
run: |
python setup.py build
python setup.py install
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --show-source --statistics
- name: Spell check
python -m pip install --upgrade pip setuptools wheel
python -m pip install tox tox-gh-actions
- name: Test with tox
run: |
codespell julearn/ docs/ examples/
- name: Test with pytest
run: |
PYTHONPATH="." pytest --cov=julearn --cov-report xml -vv julearn/
- name: 'Upload coverage to CodeCov'
uses: codecov/codecov-action@master
if: success() && matrix.python-version == 3.8
tox
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: true
flags: julearn
if: success() && matrix.python-version == 3.10
41 changes: 41 additions & 0 deletions .github/workflows/docs-preview.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Build and preview docs

on:
push:
branches-ignore:
- 'main'
tags-ignore:
- v*
pull_request:
types:
- opened
- reopened
- synchronize
- closed

concurrency: preview-${{ github.ref }}

jobs:
build-docs:
runs-on: ubuntu-latest
steps:
- name: Checkout source
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e .[docs,viz]
- name: Test build docs
run: |
BUILDDIR=_build/main make -C docs/ local
- name: Deploy preview
if: github.event_name == 'pull_request'
uses: rossjrw/pr-preview-action@v1
with:
source-dir: docs/_build/main
92 changes: 40 additions & 52 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -1,60 +1,48 @@
name: Build and publish docs

on: [push, pull_request]
on:
push:
# Sequence of patterns matched against refs/heads
branches:
- main
# Sequence of patterns matched against refs/tags
tags:
- v*

jobs:
build:
build-docs:
runs-on: ubuntu-latest
strategy:
fail-fast: false
steps:
- name: Checkout Source
uses: actions/checkout@v2
with:
# require all of history to see all tagged versions' docs
fetch-depth: 0

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: 3.8

- name: Install Dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -r docs-requirements.txt
python setup.py build
python setup.py install
- name: Checkout gh-pages
# As we already did a deploy of gh-pages above, it is guaranteed to be there
# so check it out so we can selectively build docs below
uses: actions/checkout@v2
with:
- name: Checkout source
uses: actions/checkout@v3
with:
# require all of history to see all tagged versions' docs
fetch-depth: 0
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e .[docs,viz]
python -m pip install git+https://github.com/dls-controls/sphinx-multiversion@only-arg
- name: Checkout gh-pages
# As we already did a deploy of gh-pages above, it is guaranteed to be there
# so check it out so we can selectively build docs below
uses: actions/checkout@v3
with:
ref: gh-pages
path: docs/_build

- name: Test Build Docs
if: github.ref != 'refs/heads/main' && ! startsWith(github.ref, 'refs/tags')
run: |
BUILDDIR=_build/main make -C docs/ local
- name: Build Docs
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags')
# Use the args we normally pass to sphinx-build, but run sphinx-multiversion
run: |
make -C docs/ html
touch docs/_build/.nojekyll
cp docs/redirect.html docs/_build/index.html
- name: Publish Docs to gh-pages
# Only once from main or a tag
if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags')
# We pin to the SHA, not the tag, for security reasons.
# https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions
uses: peaceiris/actions-gh-pages@bbdfb200618d235585ad98e965f4aafc39b4c501 # v3.7.3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: docs/_build
keep_files: true
- name: Build docs
# Use the args we normally pass to sphinx-build, but run sphinx-multiversion
run: |
make -C docs/ html
touch docs/_build/.nojekyll
cp docs/redirect.html docs/_build/index.html
- name: Publish docs to gh-pages
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: docs/_build
keep_files: true
31 changes: 31 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Lint

on:
- push
- pull_request

jobs:
lint:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
python-version: ['3.11']

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
python -m pip install tox tox-gh-actions
- name: Check with flake8
run: |
tox -e flake8
- name: Check with codespell
run: |
tox -e codespell
32 changes: 16 additions & 16 deletions .github/workflows/pypi.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Build and publish to PyPI and TestPyPI
name: Build and publish to PyPI
on:
push:
branches:
Expand All @@ -7,29 +7,29 @@ on:
- 'v*'
jobs:
build-n-publish:
name: Build and publish to PyPI and TestPyPI
name: Build and publish to PyPI
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master
- name: Checkout source
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up Python 3.8
uses: actions/setup-python@v1
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: 3.8
python-version: '3.10'
- name: Install build
run:
run:
pip install build
- name: Build a binary wheel and a source tarball
run: >-
run:
python -m build --sdist --wheel --outdir dist/ .
- name: Publish distribution 📦 to Test PyPI
uses: pypa/gh-action-pypi-publish@master
with:
password: ${{ secrets.testpypi_token }}
repository_url: https://test.pypi.org/legacy/
# - name: Publish distribution 📦 to Test PyPI
# uses: pypa/gh-action-pypi-publish@master
# with:
# password: ${{ secrets.testpypi_token }}
# repository_url: https://test.pypi.org/legacy/
- name: Publish distribution 📦 to PyPI
if: startsWith(github.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@master
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.pypi_token }}
password: ${{ secrets.pypi_token }}
7 changes: 4 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,11 @@ instance/
# Sphinx documentation
docs/_build/
docs/auto_examples
docs/generated
docs/**/generated
docs/make.bat

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints
Expand Down Expand Up @@ -133,4 +132,6 @@ cython_debug/
.DS_store

# Versioning
julearn/_version.py
julearn/_version.py
scratch/
docs/whats_new.rst
Loading

0 comments on commit 8a58646

Please sign in to comment.