Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor code. #177

Draft
wants to merge 78 commits into
base: dev-refactored
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
dc1529c
Renamed orthophyl.py to phyml.py
Jul 31, 2019
68be622
Renamed orthophylip.py to phylip.py
Jul 31, 2019
6ff9dcd
Updated import of phylip.py
Jul 31, 2019
3c9edcc
Updated import of phyml.py
Jul 31, 2019
68608e5
Added docstrings for the init of the PhyML class.
Aug 1, 2019
7bf7740
Refactored init in ETE3PAMl to create better API.
Aug 1, 2019
0d6b792
Fixed logging issue.
Aug 1, 2019
4b267b3
Added check_exe function.
Aug 8, 2019
dfaf2f3
Added new examples to README.
Aug 8, 2019
c7d08cc
Added ApplicationError to try/except in run method.
Aug 8, 2019
7c420b9
Added a test for phyml
Aug 8, 2019
01b458c
Added test data for phyml test.
Aug 8, 2019
f4edaf7
Added phyml installation to travis script.
Aug 8, 2019
7ee4e7b
Added ability for user to choose number of processors.
Aug 8, 2019
a2214ea
Updated README for new api.
Aug 8, 2019
73261f2
Removed deprecated csvtolist
Aug 8, 2019
215a24d
Added logging to ETE3PAML
Aug 8, 2019
2734fc7
Added _import_alignment method
Aug 8, 2019
2feab29
Added a README for the PHYLIP class.
Aug 12, 2019
5e75f38
Added try/except/else/finally for phylip methods
Aug 12, 2019
1d47901
Removed phylip test from Phylip folder.
Aug 12, 2019
a584164
Added docstrings to ETE3PAML class.
Aug 19, 2019
eb5914d
Fixed errors in README.
Aug 19, 2019
32676ae
Added TODO in codeml.py
Aug 19, 2019
f86847a
Changed data in test.
Aug 19, 2019
2a718e3
Added missing doctrings.
Aug 19, 2019
8a27d3b
Updated example in README.
Aug 19, 2019
a764257
Refactored TreeViz api.
Aug 19, 2019
9cbfc28
Refactored ncbi-download script
Aug 19, 2019
073f272
Added validation function. Extended run api.
Aug 21, 2019
c0084ac
Updated PhyML test data.
Aug 21, 2019
a61a05d
Added ability to capture output for Phylip
Aug 22, 2019
0e697f1
Fixed validate format issue.
Aug 22, 2019
03ed667
Fixed PhyML arg.
Aug 22, 2019
d1589a9
Renamed PhyloTree module to TreeViz
Aug 23, 2019
732e120
Added a treeviz test.
Aug 27, 2019
ccc8230
Removed sciluigi. Added matplotlib.
Aug 27, 2019
1d6341a
Removed matplotlib from travis CI pip install line
Aug 27, 2019
eec0b1f
Added updated example to readme.
Aug 27, 2019
e3a1464
Added docstrings.
Aug 27, 2019
10c9261
Removed version of matplotlib
Aug 27, 2019
35dd071
Corrected path to tree file in test.
Aug 27, 2019
c96b91c
Updated Flask to latest version and removed other flask libraries.
Aug 27, 2019
b40adb6
Corrected paths in tests.
Aug 27, 2019
1b7fe33
Additional fix to current directory for tests.
Aug 27, 2019
e8d8638
Changed paths of test output.
Aug 27, 2019
007efd5
Fixed Phyml tests
Aug 27, 2019
e4d860f
Merge branch 'dev-refactored' into refactor-phylo
sdhutchins Dec 16, 2022
fbbc7f8
Merge branch 'dev-refactored' into refactor-phylo
sdhutchins Dec 18, 2022
a5805f2
Fix extra lines in code.
sdhutchins Dec 25, 2022
ff5662c
Fix psutil requirement.
sdhutchins Jan 2, 2023
e167cab
Remove cookiecutter version.
sdhutchins Jan 2, 2023
813215b
Remove Flask version requirements.
sdhutchins Jan 2, 2023
124fdf5
Drop support for Python < 3.7.
sdhutchins Jan 2, 2023
75dee6c
Fix name for log level to format color.
sdhutchins Jan 2, 2023
50d5621
Remove version for setuptools.
sdhutchins Jan 2, 2023
a4dfd41
Merge branch 'dev-refactored' into refactor-phylo
sdhutchins Jan 2, 2023
1219a2e
Remove incorrect character from travis script.
sdhutchins Jan 2, 2023
b41dd60
Add commands to PyBasher.
sdhutchins Jan 2, 2023
59f33f2
Refactor blastpipeline.
sdhutchins Jan 2, 2023
6cd21c5
Remove Flask-User.
sdhutchins Jan 2, 2023
82b2827
Set _COLORS for logging.
sdhutchins Jan 2, 2023
9a59d64
Deprecate airflow.
sdhutchins Jan 2, 2023
33aec9c
Comment out phyml test.
sdhutchins Jan 3, 2023
963ab83
Add more test infrastructure.
sdhutchins Jan 3, 2023
7ba045b
Remove luigi version.
sdhutchins Jan 3, 2023
7b32f0a
Fix cookies test.
sdhutchins Jan 3, 2023
5caa2b8
Test.
sdhutchins Jan 6, 2024
5bcaffd
Test.
sdhutchins Jan 6, 2024
1389ee9
Add github action for ci.
sdhutchins Jan 6, 2024
ed7783d
Fix test.
sdhutchins Jan 6, 2024
e2ae3ca
Fix texts for Oven and CookBook.
sdhutchins Jan 6, 2024
7b7a8c5
Updated utils tests.
sdhutchins Jan 6, 2024
54a72f8
Remove PackageVersion test.
sdhutchins Jan 6, 2024
38f9b58
Fix Cookies tests.
sdhutchins Jan 6, 2024
39788c0
Remove github action.
sdhutchins Jan 6, 2024
3ca103a
Add ci back. Remove blast tests.
sdhutchins Jan 8, 2024
ab2ef59
Change timeout.
sdhutchins Jan 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Python CI

on: [push, pull_request]

jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
matrix:
python-version: [3.7]

steps:
- uses: actions/checkout@v2

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}

- name: Cache pip
uses: actions/cache@v2
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-

- name: Install dependencies
run: |
sudo apt-get install -qq phyml
pip install --upgrade pip setuptools wheel
pip install --only-binary=numpy,scipy numpy scipy
pip install matplotlib ipython jupyter sympy pytest codecov pytest-cov
pip install -r requirements.txt
pip install .

- name: Run tests
run: pytest --cov-report=xml --cov=OrthoEvol tests/

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./coverage.xml
fail_ci_if_error: true

- name: Notify failure
if: failure()
run: echo "Build failed"
20 changes: 9 additions & 11 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,32 +5,30 @@ notifications:
email: [email protected]
jobs:
include:
- python: 3.5
dist: trusty
install:
- pip install --upgrade pip setuptools wheel
- pip install --only-binary=numpy,scipy numpy scipy
- pip install matplotlib ipython jupyter sympy pytest codecov "pytest-cov<=2.6.0"
- pip install -r requirements.txt
- pip install .
- python: 3.6
- python: 3.7
dist: xenial
before_install:
- sudo apt-get install -qq phyml
install:
- pip install --upgrade pip setuptools wheel
- pip install --only-binary=numpy,scipy numpy scipy
- pip install matplotlib ipython jupyter sympy pytest codecov pytest-cov
- pip install -r requirements.txt
- pip install .
- python: 3.7
- python: 3.8
dist: xenial
before_install:
- sudo apt-get install -qq phyml
install:
- pip install --upgrade pip setuptools wheel
- pip install --only-binary=numpy,scipy numpy scipy
- pip install matplotlib ipython jupyter sympy pytest codecov pytest-cov
- pip install -r requirements.txt
- pip install .
- python: 3.8
- python: 3.9
dist: xenial
before_install:
- sudo apt-get install -qq phyml
install:
- pip install --upgrade pip setuptools wheel
- pip install --only-binary=numpy,scipy numpy scipy
Expand Down
150 changes: 73 additions & 77 deletions OrthoEvol/Cookies/new_website/tasks.py
Original file line number Diff line number Diff line change
@@ -1,77 +1,73 @@
#!/usr/bin/env python
"""Invoke tasks."""
import os
import json
import shutil
import webbrowser

from invoke import task

HERE = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(HERE, 'cookiecutter.json'), 'r') as fp:
COOKIECUTTER_SETTINGS = json.load(fp)
# Match default value of website_name from cookiecutter.json
COOKIE = os.path.join(HERE, COOKIECUTTER_SETTINGS['website_name'])
AUTOAPP = os.path.join(COOKIE, 'autoapp.py')
REQUIREMENTS = os.path.join(COOKIE, 'requirements', 'dev.txt')


@task
def build(ctx):
"""Build the cookiecutter.

:param ctx:

"""
ctx.run('cookiecutter {0} --no-input'.format(HERE))


@task
def clean(ctx):
"""Clean out generated cookiecutter.

:param ctx:

"""
if os.path.exists(COOKIE):
shutil.rmtree(COOKIE)
print('Removed {0}'.format(COOKIE))
else:
print('App directory does not exist. Skipping.')


def _run_flask_command(ctx, command):
"""

:param ctx:
:param command:

"""
ctx.run('FLASK_APP={0} flask {1}'.format(AUTOAPP, command), echo=True)


@task(pre=[clean, build])
def test(ctx):
"""Run lint commands and tests.

:param ctx:

"""
ctx.run('pip install -r {0} --ignore-installed'.format(REQUIREMENTS),
echo=True)
os.chdir(COOKIE)
_run_flask_command(ctx, 'lint')
_run_flask_command(ctx, 'test')

@task
def readme(ctx, browse=False):
"""

:param ctx:
:param browse: (Default value = False)

"""
ctx.run("rst2html.py README.rst > README.html")
if browse:
webbrowser.open_new_tab('README.html')

#!/usr/bin/env python
"""Invoke tasks."""

import os
import json
import shutil
import webbrowser

from invoke import task

HERE = os.path.abspath(os.path.dirname(__file__))

# Load the settings from cookiecutter.json
with open(os.path.join(HERE, 'cookiecutter.json'), 'r') as fp:
COOKIECUTTER_SETTINGS = json.load(fp)

# Match default value of website_name from cookiecutter.json
COOKIE = os.path.join(HERE, COOKIECUTTER_SETTINGS['website_name'])

# Path to autoapp.py file
AUTOAPP = os.path.join(COOKIE, 'autoapp.py')

# Path to dev requirements file
REQUIREMENTS = os.path.join(COOKIE, 'requirements', 'dev.txt')

@task
def build(ctx):
"""Build the cookiecutter."""

# Run cookiecutter with no input
ctx.run(f'cookiecutter {HERE} --no-input')

@task
def clean(ctx):
"""Clean out generated cookiecutter."""

# Remove the cookiecutter directory if it exists
if os.path.exists(COOKIE):
shutil.rmtree(COOKIE)
print(f'Removed {COOKIE}')
else:
print('App directory does not exist. Skipping.')

def _run_flask_command(ctx, command):
"""Run a Flask command."""

# Run the specified Flask command
ctx.run(f'FLASK_APP={AUTOAPP} flask {command}', echo=True)

@task(pre=[clean, build])
def test(ctx):
"""Run lint commands and tests."""

# Install dev requirements
ctx.run(f'pip install -r {REQUIREMENTS} --ignore-installed', echo=True)

# Change to the cookiecutter directory
os.chdir(COOKIE)

# Run lint and test commands
_run_flask_command(ctx, 'lint')
_run_flask_command(ctx, 'test')

@task
def readme(ctx, browse=False):
"""Convert the README to HTML."""

# Convert the README to HTML
ctx.run("rst2html.py README.rst > README.html")

# Open the HTML file in a web browser if specified
if browse:
webbrowser.open_new_tab('README.html')
Empty file.
78 changes: 42 additions & 36 deletions OrthoEvol/Manager/db_mana_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,43 +7,49 @@
import getpass
from datetime import datetime as d
import os
_jobname = "jobname"

# Set up project management
pm_config_file = resource_filename(yml.__name__, "initialize_new.yml")
with open(pm_config_file, 'r') as f:
pm_config = yaml.load(f, Loader=yaml.FullLoader)
pm = ProjectManagement(**pm_config["Management_config"])
# Define job name
job_name = "jobname"

# Set up database management
db_config_file = resource_filename(yml.__name__, "databases.yml")
with open(db_config_file, 'r') as f:
db_config = yaml.load(f, Loader=yaml.FullLoader)
# Function to load configuration from YAML file
def load_config(file_name):
file_path = resource_filename(yml.__name__, file_name)
with open(file_path, 'r') as file:
return yaml.load(file, Loader=yaml.FullLoader)

# Load project management configuration
pm_config = load_config("initialize_new.yml")
project_manager = ProjectManagement(**pm_config["Management_config"])

# Load and update database management configuration
db_config = load_config("databases.yml")
db_config.update(pm_config)
db_config['Database_config']['Full']['NCBI']['NCBI_refseq_release']['upload_number'] = 12
db_config['Database_config']['Full']['NCBI']['NCBI_refseq_release']['pbs_dict'] = {
'author': getpass.getuser(),
'description': 'This is a default pbs job.',
'date': d.now().strftime('%a %b %d %I:%M:%S %p %Y'),
'proj_name': 'OrthoEvol',
'select': '1',
'memgb': '6gb',
'cput': '72:00:00',
'wt': '2000:00:00',
'job_name': _jobname,
'outfile': _jobname + '.o',
'errfile': _jobname + '.e',
'script': _jobname,
'log_name': _jobname,
'pbsworkdir': os.getcwd(),
'cmd': 'python3.6 ' + os.path.join(os.getcwd(), _jobname + '.py'),
'email': 'n/a'
}
# Generate main config file for this job
config_file = pm.user_log / Path("upload_config.yml")
with open(str(config_file), 'w') as cf:
yaml.dump(db_config, cf, default_flow_style=False)
ncbi_config = db_config['Database_config']['Full']['NCBI']['NCBI_refseq_release']
ncbi_config['upload_number'] = 12
ncbi_config['pbs_dict'] = {
'author': getpass.getuser(),
'description': 'This is a default pbs job.',
'date': d.now().strftime('%a %b %d %I:%M:%S %p %Y'),
'proj_name': 'OrthoEvol',
'select': '1',
'memgb': '6gb',
'cput': '72:00:00',
'wt': '2000:00:00',
'job_name': job_name,
'outfile': job_name + '.o',
'errfile': job_name + '.e',
'script': job_name,
'log_name': job_name,
'pbsworkdir': os.getcwd(),
'cmd': f'python3.6 {os.path.join(os.getcwd(), job_name + ".py")}',
'email': 'n/a'
}

# Save the updated configuration to a YAML file
config_file_path = project_manager.user_log / Path("upload_config.yml")
with open(str(config_file_path), 'w') as config_file:
yaml.dump(db_config, config_file, default_flow_style=False)

# Set up database dispatcher and dispatch the functions
dd = DatabaseDispatcher(config_file, pm)
dd.dispatch(dd.strategies, dd.dispatcher, dd.configuration)
# Initialize database dispatcher and execute dispatch functions
db_dispatcher = DatabaseDispatcher(config_file_path, project_manager)
db_dispatcher.dispatch(db_dispatcher.strategies, db_dispatcher.dispatcher, db_dispatcher.configuration)
20 changes: 12 additions & 8 deletions OrthoEvol/Orthologs/Phylogenetics/PAML/README.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,40 @@
# PAML Documentation

PAML (Phylogenetic Analysis by Maximum Likelihood) is a package of
programs for phylogenetic analyses of DNA or protein
sequences using maximum likelihood and is maintained by Ziheng Yang.


## Why ETE?
ETE is python package for building, comparing, annotating, manipulating and visualising
trees. It provides a comprehensive API and a collection of command line tools,
including utilities to work with the NCBI taxonomy tree.

ETE is a python package for building, comparing, annotating, manipulating and visualising
trees. It provides a comprehensive API and a collection of command line tools including
utilities to work with the NCBI taxonomy tree.

### Model Selection and Default Parameters

It's important to note the default parameters for `ETE3PAML` are as follows:
`model='M1'`, `workdir=```.
`model='M1'`, `workdir=''`.

## Usage & Examples

### A simple implementation of ETE3PAML

```python
from OrthoEvol.Orthologs.Phylogenetics.PAML import ETE3PAML

paml = ETE3PAML(alignmentfile='.ffn', speciestree='.nw', workdir='')

paml.run(pamlsrc='path/to/codeml/binary', output_folder=None)
paml = ETE3PAML(alignmentfile='.ffn', speciestree='tree.nw', workdir='',
pamlsrc='path/to/codeml/binary')

paml.run(output_folder=None)
```

### Pruning a tree for use with ETE3PAML

```python
from OrthoEvol.Orthologs.Phylogenetics.PAML import ETE3PAML

paml = ETE3PAML(alignmentfile='HTR1A.ffn', speciestree='speciestree.nw', workdir='')
paml = ETE3PAML(infile='HTR1A.ffn', species_tree='speciestree.nw', workdir='')

# Input a list of orgnanisms or an organisms csv file with header as 'Organisms'
paml.prune_tree(organisms='organisms.csv')
Expand Down
Loading
Loading