-
Notifications
You must be signed in to change notification settings - Fork 9
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
github-action
committed
Aug 27, 2024
0 parents
commit 64df10c
Showing
92 changed files
with
11,099 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: 10a9503c046d44e2cf9617368b4a445c | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
Empty file.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
.. _api-reference: | ||
|
||
API Reference | ||
============= | ||
|
||
This is a reference covering **the Python modules and objects across | ||
the CATS codebase**. For the interface for the *command-line*, which is | ||
likely to be more relevant to *users* (rather than developers) of CATS, | ||
please see :ref:`cli-reference`. | ||
|
||
Modules | ||
------- | ||
|
||
``cats`` | ||
^^^^^^^^^^^^^^^^^ | ||
|
||
.. automodule:: cats | ||
:members: | ||
|
||
``cats.configure`` | ||
^^^^^^^^^^^^^^^^^^ | ||
|
||
.. automodule:: cats.configure | ||
:members: | ||
|
||
``cats.CI_api_interface`` | ||
^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
.. automodule:: cats.CI_api_interface | ||
:members: | ||
|
||
``cats.CI_api_query`` | ||
^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
.. automodule:: cats.CI_api_query | ||
:members: | ||
|
||
``cats.carbonFootprint`` | ||
^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
.. automodule:: cats.carbonFootprint | ||
:members: | ||
|
||
``cats.forecast`` | ||
^^^^^^^^^^^^^^^^^ | ||
|
||
.. automodule:: cats.forecast | ||
:members: | ||
|
||
|
||
``cats.optimise_starttime`` | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
.. automodule:: cats.optimise_starttime | ||
:members: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
.. _cli-reference: | ||
|
||
Command-line (CLI) reference | ||
============================ | ||
|
||
|
||
.. argparse:: | ||
:module: cats | ||
:func: parse_arguments | ||
:prog: cats |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
.. _contributing: | ||
|
||
.. _GitHubrepo: https://github.com/GreenScheduler/cats | ||
|
||
Contributing | ||
============ | ||
|
||
We welcome contributions from the community! | ||
|
||
If you find a bug or have an idea for a new feature, please open an | ||
issue on `our GitHub repository <GitHubrepo_>`_ or submit a pull request. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
.. _index: | ||
|
||
Documentation for CATS: the Climate-Aware Task Scheduler | ||
======================================================== | ||
|
||
CATS is a **C**\limate-**A**\ware **T**\ask **S**\cheduler. It schedules | ||
cluster jobs to minimize predicted carbon intensity of running the process. | ||
|
||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
:caption: Contents: | ||
|
||
introduction.rst | ||
installation.rst | ||
quickstart.rst | ||
use-with-schedulers.rst | ||
cli-reference.rst | ||
api-reference.rst | ||
contributing.rst | ||
|
||
|
||
Indices and tables | ||
================== | ||
|
||
* :ref:`genindex` | ||
* :ref:`modindex` | ||
* :ref:`search` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
.. _installation: | ||
|
||
Installation | ||
============ | ||
|
||
Install CATS via pip as follows: | ||
|
||
.. code-block:: console | ||
:caption: *Command to install CATS with pip.* | ||
$ pip install climate-aware-task-scheduler | ||
To install the development version | ||
|
||
.. code-block:: console | ||
$ pip install git+https://github.com/GreenScheduler/cats |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
.. _introduction: | ||
|
||
.. The list below references links that may be used in more than one place | ||
.. _SSIsite: https://www.software.ac.uk/ | ||
.. _CW23page: https://software.ac.uk/cw23 | ||
.. _CW23HackDaypage: https://www.software.ac.uk/cw23/hack-day | ||
.. _NationalGridESO: https://www.nationalgrideso.com/ | ||
.. _CarbonIntensityAPI: https://carbonintensity.org.uk/ | ||
.. _GitHubrepoissues: https://github.com/GreenScheduler/cats/issues | ||
|
||
|
||
Introduction | ||
============ | ||
|
||
CATS is a **C**\limate-**A**\ware **T**\ask **S**\cheduler. It schedules | ||
cluster jobs to minimize predicted carbon intensity of running the process. | ||
|
||
|
||
Summary | ||
------- | ||
|
||
The Climate-Aware Task Scheduler is a lightweight Python package designed | ||
to schedule tasks based on the estimated carbon intensity of the | ||
electricity grid at any given moment. | ||
|
||
This tool uses real-time | ||
carbon intensity data from the `National Grid ESO <NationalGridESO_>`_ | ||
via `their API <CarbonIntensityAPI_>`_ to | ||
estimate the carbon intensity of the electricity grid, and schedules | ||
tasks at times when the estimated carbon intensity is lowest. This | ||
helps to reduce the carbon emissions associated with running | ||
computationally intensive tasks, making it an ideal solution for | ||
environmentally-conscious developers. | ||
|
||
|
||
Scope | ||
----- | ||
|
||
Currently CATS only works in the UK. If you are aware of APIs for | ||
realtime grid carbon intensity data in other countries, please | ||
`open an issue <GitHubrepoissues_>`_ to let us know. | ||
|
||
|
||
Background | ||
---------- | ||
|
||
It was created as part of the | ||
`Software Sustainability Institute’s <SSIsite_>`_ | ||
`Collaborations Workshop 2023 <CW23page_>`_ | ||
`Hack Day <CW23HackDaypage_>`_. | ||
|
||
|
||
Features | ||
-------- | ||
|
||
* Estimates the carbon intensity of the electricity grid in real-time | ||
* Schedules tasks based on the estimated carbon intensity, minimizing | ||
carbon emissions | ||
* Provides a simple and intuitive API for developers | ||
* Lightweight and easy to integrate into existing workflows | ||
* Supports Python 3.9+ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,122 @@ | ||
.. _quickstart: | ||
|
||
Quickstart | ||
========== | ||
|
||
Basic usage | ||
----------- | ||
|
||
You can run CATS with: | ||
|
||
.. code-block:: console | ||
:caption: *A basic command to run CATS when a job duration and postcode | ||
are provided.* | ||
$ cats --duration 480 --location "EH8" | ||
The ``--location`` option is optional, and can be pulled from a | ||
configuration file (see :ref:`configuration-file`), or inferred using | ||
the server's IP address. | ||
|
||
The ``--duration`` option indicates the expected job duration in | ||
minutes. | ||
|
||
The scheduler then calls a function that estimates the best time to start | ||
the job given predicted carbon intensity over the next 48 hours. The | ||
workflow is the same as for other popular schedulers. Switching to CATS | ||
should be transparent to cluster users. | ||
|
||
It will display the time to start the job on standard out and optionally | ||
some information about the carbon intensity on standard error. | ||
|
||
.. _configuration-file: | ||
|
||
Using a configuration file | ||
-------------------------- | ||
|
||
Information about location can be provided by a configuration file | ||
instead of a command line arguments to the ``cats`` command. | ||
|
||
.. code-block:: yaml | ||
location: "EH8" | ||
Use the ``--config`` option to specify a path to the configuration | ||
file, relative to the current directory. | ||
|
||
In case of a missing location command line argument, ``cats`` looks | ||
for a file named ``config.yaml`` or ``config.yml`` in the current directory. | ||
|
||
.. code-block:: shell | ||
# Override duration value at the command line | ||
cats --config /path/to/config.y(a)ml --location "OX1" | ||
When ``--duration`` information is not provided via the option, and | ||
location information is not provided in the YAML configuration file | ||
specified or detected, CATS will try to estimate location from the | ||
machine IP address: | ||
|
||
.. code-block:: console | ||
$ cats --duration 480 | ||
WARNING:root:config file not found | ||
WARNING:root:Unspecified carbon intensity forecast service, using carbonintensity.org.uk | ||
WARNING:root:location not provided. Estimating location from IP address: RG2. | ||
Best job start time 2024-08-22 07:30:49.800951+01:00 | ||
Carbon intensity if job started now = 117.95 gCO2eq/kWh | ||
Carbon intensity at optimal time = 60.93 gCO2eq/kWh | ||
Use --format=json to get this in machine readable format | ||
Displaying carbon footprint estimates | ||
------------------------------------- | ||
|
||
CATS is able to provide an estimate for the carbon footprint reduction | ||
resulting from delaying your job. To enable the footprint estimation, | ||
you must provide the ``--footprint`` option, the memory consumption in GB | ||
and a hardware profile: | ||
|
||
.. code-block:: shell | ||
cats --duration 480 --location "EH8" --footprint --memory 8 --profile <my_profile> | ||
The ``--profile`` option specifies information power consumption and | ||
quantity of hardware the job using. This information is provided by | ||
adding a section ``profiles`` to the :ref:`cats YAML configuration | ||
file <configuration-file>`. | ||
|
||
You can define an arbitrary number of profiles as subsection of the | ||
top-level ``profiles`` section: | ||
|
||
.. literalinclude :: ../../cats/config.yml | ||
:language: yaml | ||
:caption: *An example provision of machine information by YAML file | ||
to enable estimation of the carbon footprint reduction.* | ||
The name of the profile section is arbitrary, but each profile section | ||
*must* contain one ``cpu`` section, or one ``gpu`` section, or both. | ||
Each hardware type (``cpu`` or ``gpu``) section *must* contain the | ||
``power`` (in Watts, for one unit) and ``nunits`` sections. The ``model`` section is optional, | ||
meant for documentation. | ||
|
||
When running ``cats``, you can specify which profile to use for carbon | ||
footprint estimation with the ``--profile`` option: | ||
|
||
.. code-block:: shell | ||
cats --duration 480 --location "EH8" --footprint --memory 6.7 --profile my_gpu_profile | ||
The default number of units specified for a profile can be overidden | ||
at the command line: | ||
|
||
.. code-block:: shell | ||
cats --duration 480 --location "EH8" --footprint --memory 16 \ | ||
--profile my_gpu_profile --gpu 4 --cpu 1 | ||
.. warning:: | ||
The ``--profile`` option is optional. If not provided, ``cats`` uses the | ||
first profile defined in the configuration file as the default | ||
profile. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
.. _use-with-schedulers: | ||
|
||
Use with Schedulers | ||
=================== | ||
|
||
Using CATS with the ``at`` scheduler | ||
------------------------------------ | ||
|
||
You can use CATS with the ``at`` job scheduler by running: | ||
|
||
.. code-block:: console | ||
$ cats -d <job_duration> --loc <postcode> --scheduler at --command '<command>' | ||
As an example, if you want to schedule a run of ``ls`` with a 5 minute | ||
duration, in the 'OX1' postcode that would look like: | ||
|
||
.. code-block:: console | ||
$ cats -d 5 --loc OX1 --scheduler at --command 'ls' | ||
Demonstration | ||
^^^^^^^^^^^^^ | ||
|
||
.. image:: ../../cats.gif | ||
:width: 400 | ||
:alt: A video demonstration of CATS being used standalone and with ``at``. | ||
:align: center |
Oops, something went wrong.