Skip to content

Commit

Permalink
Merge pull request #506 from cmoussa1/update.readme
Browse files Browse the repository at this point in the history 
doc: add "Database Administration" section, update README to point to docs site
  • Loading branch information
@mergify
mergify[bot] authored Oct 15, 2024
2 parents 859ee8f + f9861d4 commit 039f066
Show file tree
Hide file tree
Showing 5 changed files with 198 additions and 170 deletions.
162 changes: 7 additions & 155 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
[![Build Status](https://travis-ci.org/flux-framework/flux-accounting.svg?branch=master)](https://travis-ci.org/flux-framework/flux-accounting)
[![ci](https://github.com/flux-framework/flux-accounting/workflows/ci/badge.svg)](https://github.com/flux-framework/flux-accounting/actions?query=workflow%3A.github%2Fworkflows%2Fmain.yml)
[![codecov](https://codecov.io/gh/flux-framework/flux-accounting/branch/master/graph/badge.svg)](https://codecov.io/gh/flux-framework/flux-accounting)
![docs](https://readthedocs.org/projects/flux-accounting/badge/?version=latest)

_NOTE: The interfaces of flux-accounting are being actively developed and are
not yet stable. The Github issue tracker is the primary way to communicate with
Expand Down Expand Up @@ -37,89 +38,7 @@ flux-core that it is configured against; this is the default behavior of
PYTHON_VERSION=3.7 ./configure --localstatedir=/var/
```

### Configuring flux-accounting background scripts

There are a number of scripts that run in the background to update both job
usage and fairshare values. These require configuration upon setup of
flux-accounting. The first thing to configure when first setting up the
flux-accounting database is to set the PriorityUsageResetPeriod and
PriorityDecayHalfLife parameters. Both of these parameters represent a number of
weeks by which to hold usage factors up to the time period where jobs no longer
play a factor in calculating a usage factor. If these parameters are not passed
when creating the DB, PriorityDecayHalfLife is set to 1 week and
PriorityUsageResetPeriod is set to 4 weeks, i.e the flux-accounting database
will store up to a month's worth of jobs broken up into one week chunks:

```
flux account create-db --priority-decay-half-life=2 --priority-usage-reset-period=8
```

The other component to load is the multi-factor priority plugin, which can be
loaded with `flux jobtap load`:

```
flux jobtap load mf_priority.so
```

After the DB and job priority plugin are set up, the `update-usage` subcommand
should be set up to run as a cron job. This subcommand fetches the most recent
job records for every user in the flux-accounting DB and calculates a new job
usage value. This subcommand takes one optional argument,
`--priority-decay-half-life`, which, like the parameter set in the database
creation step above, represents the number of weeks to hold one job usage
"chunk." If not specified, this optional argument also defaults to 1 week.

```
flux account update-usage --priority-decay-half-life=2 path/to/job-archive-DB
```

After the job usage values are re-calculated and updated, the fairshare values
for each user also need to be updated. This can be accomplished by configuring
the `flux-update-fshare` script to also run as a cron job. This fetches user
account data from the flux-accounting DB and recalculates and writes the updated
fairshare values back to the DB.

```
flux account-update-fshare
```

Once the fairshare values for all of the users in the flux-accounting DB get
updated, this updated information will be sent to the priority plugin. This
script can be also be configured to run as a cron job:

```
flux account-priority-update
```

### Run flux-accounting's commands:

```
usage: flux-account.py [-h] [-p PATH] [-o OUTPUT_FILE]
{view-user,add-user,delete-user,edit-user,view-job-records,create-db,add-bank,view-bank,delete-bank,edit-bank,print-hierarchy} ...
Description: Translate command line arguments into SQLite instructions for the Flux Accounting Database.
positional arguments:
{view-user,add-user,delete-user,edit-user,view-job-records,create-db,add-bank,view-bank,delete-bank,edit-bank,print-hierarchy}
sub-command help
view-user view a user's information in the accounting database
add-user add a user to the accounting database
delete-user remove a user from the accounting database
edit-user edit a user's value
view-job-records view job records
create-db create the flux-accounting database
add-bank add a new bank
view-bank view bank information
delete-bank remove a bank
edit-bank edit a bank's allocation
print-hierarchy print accounting database
optional arguments:
-h, --help show this help message and exit
-p PATH, --path PATH specify location of database file
-o OUTPUT_FILE, --output-file OUTPUT_FILE
specify location of output file
```
### Testing

To run the unit tests in a Docker container, or launch into an interactive container on your local machine, you can run `docker-run-checks.sh`:

Expand All @@ -136,78 +55,11 @@ mounting /Users/moussa1/src/flux-framework/flux-accounting as /usr/src
[moussa1@docker-desktop src]$
```

### User and Bank Information

The accounting tables in this database stores information like username and
ID, banks to submit jobs against, allocated shares to the user, as well as
static limits, including a max number of running jobs at a given time and
a max number of submitted jobs per user/bank combo.

### Interacting With the Accounting DB

In order to add, edit, or remove information from the flux-accounting database,
you must also have read/write access to the directory that the DB file resides
in. The [SQLite documentation](https://sqlite.org/omitted.html) states:

> Since SQLite reads and writes an ordinary disk file, the only access
permissions that can be applied are the normal file access permissions of the
underlying operating system.

There are two ways you can interact with the tables contained in the Accounting
DB. The first way is to launch into an interactive SQLite shell. From there, you
can open the database file and interface with any of the tables using SQLite
commands:

```
$ sqlite3 path/to/FluxAccounting.db
SQLite version 3.24.0 2018-06-04 14:10:15
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .tables
association_table bank_table
```

To get nicely formatted output from queries (like headers for the tables and
proper spacing), you can also set the following options in your shell:

```
sqlite> .mode columns
sqlite> .headers on
```

This will output queries like the following:

```
sqlite> SELECT * FROM association_table;
creation_time mod_time deleted username bank shares max_jobs max_wall_pj
------------- ---------- ---------- ---------- ---------- ---------- ---------- -----------
1605309320 1605309320 0 fluxuser foo 1 1 60
```

The second way is to use flux-accounting's command line arguments. You can pass in
a path to the database file, or it will default to the "compiled-in" path of
`${prefix}/var/FluxAccounting.db`.

With flux-accounting's command line tools, you can view a user's account
information, add and remove users to the accounting database, and edit an existing
user's account information:

```
$ flux account view-user fluxuser
creation_time mod_time deleted username bank shares max_jobs max_wall_pj
1595438356 1595438356 0 fluxuser foo 1 100 60
```

Multiple rows of data can be loaded to the database at once using `.csv` files
and the `flux account pop-db` command. Run `flux account pop-db --help` for
`.csv` formatting instructions.
### Documentation

User and bank information can also be exported from the database using the
`flux account export-db` command, which will extract information from both the
user and bank tables and place them into `.csv` files.
To see detailed documentation about installing, administering, and configuring
flux-accounting, please see our official
[documentation pages](https://flux-accounting.readthedocs.io/en/latest/index.html).

#### Release

Expand Down
173 changes: 173 additions & 0 deletions doc/components/database-administration.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
.. _database-administration:

#######################################
Flux Accounting Database Administration
#######################################

This document outlines the various methods of administration on the
flux-accounting database.

********
Overview
********

The tables in the flux-accounting database stores information for associations,
banks, job records, queues, and projects.

For associations, shares, limits, queue and project permissions, and bank
membership can be managed.

**********************************
Interacting With the Accounting DB
**********************************

In order to add, edit, or remove information from the flux-accounting database,
you must also have read/write access to the directory that the DB file resides
in. The `SQLite documentation <https://sqlite.org/omitted.html>`_ states:

Since SQLite reads and writes an ordinary disk file, the only access
permissions that can be applied are the normal file access permissions of
the underlying operating system.

flux-accounting provides an interface to the SQLite database containing all of
the aforementioned data for associations, banks, queues, and projects. Commands
can be run to add/remove all of these, edit certain attributes, and view this
information in different formats. See ``flux account --help`` to list all of
the available flux-accounting commands.

The bank/association database hierarchy assumes a tree structure, meaning that
there is one root bank which all other banks and associations descend from:

.. code-block:: console
$ flux account view-bank root -t
Account Username RawShares RawUsage Fairshare
root 1 0.0
bank_A 1 0.0
bank_A user_1 1 0.0 0.5
bank_B 1 0.0
bank_B user_2 1 0.0 0.5
bank_B user_3 1 0.0 0.5
bank_C 1 0.0
bank_C_a 1 0.0
bank_C_a user_4 1 0.0 0.5
bank_C_b 1 0.0
bank_C_b user_5 1 0.0 0.5
bank_C_b user_6 1 0.0 0.5
Multiple levels of banks can be defined under this root bank. Users can belong
to more than one bank and will have at most one default bank.

To add a bank to the database, you can use the ``flux account add-bank``
command. Each ``add-bank`` call requires a bank name, their allocated shares,
and a parent bank name (if it is not the root bank):

.. code-block:: console
$ flux account add-bank root 1
$ flux account add-bank --parent-bank=root bank_A 1
From here, users can be added to these banks to create *associations*, a
2-tuple combination of a username and bank name:

.. code-block:: console
$ flux account add-user --username=user_1 --bank=bank_A
If you wish to delete an association or bank from the database, you can run the
``flux account delete-user`` or ``flux account delete-bank`` commands. Note
that this will not actually remove the association's or bank's row from the
table where their data is stored, but will instead set their ``active`` column
to 0. To re-enable an association's ``active`` status, simply re-add them with
``flux account add-user``.

Information for associations can be viewed in more than one format:

.. code-block:: console
$ flux account view-user user_1
creation_time: 1728922316
mod_time: 1728922316
active: 1
username: user_1
userid: 5001
bank: bank_A
default_bank: bank_A
shares: 1
job_usage: 0.0
fairshare: 0.5
max_running_jobs: 5
max_active_jobs: 7
max_nodes: 2147483647
queues:
projects: *
default_project: *
$ flux account view-user user_1 --json
{
"username": "user_1",
"userid": 5001,
"default_bank": "bank_A",
"banks": [
{
"bank": "bank_A",
"active": 1,
"shares": 1,
"job_usage": 0.0,
"fairshare": 0.5,
"max_running_jobs": 5,
"max_active_jobs": 7,
"max_nodes": 2147483647,
"queues": "",
"projects": "*",
"default_project": "*"
}
]
}
If you are familiar with SQLite syntax, you can also launch into an interactive
SQLite shell. From there, you can open the database file and interface with
any of the tables using SQLite commands:

.. code-block:: console
$ sqlite3 path/to/FluxAccounting.db
SQLite version 3.24.0 2018-06-04 14:10:15
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite>
To get nicely formatted output from queries (like headers for the tables and
proper spacing), you can also set the following options in your shell:

.. code-block:: console
sqlite> .mode columns
sqlite> .headers on
This will output queries like the following:

.. code-block:: console
sqlite> SELECT * FROM association_table;
creation_time mod_time deleted username bank shares max_jobs max_wall_pj
------------- ---------- ---------- ---------- ---------- ---------- ---------- -----------
1605309320 1605309320 0 fluxuser foo 1 1 60
**********************************
Flux-Accounting Data Import/Export
**********************************

Multiple rows of data can be loaded to the database at once using ``.csv`` files
and the ``flux account pop-db`` command. Run ``flux account pop-db --help`` for
``.csv`` formatting instructions.

User and bank information can also be exported from the database using the
``flux account export-db`` command, which will extract information from both the
user and bank tables and place them into ``.csv`` files.
16 changes: 5 additions & 11 deletions doc/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,12 @@ Documentation for flux-accounting
This site contains documentation for flux-accounting configuration and
management.

.. toctree::
:maxdepth: 2

guide/accounting-guide


.. _man-pages:

flux-accounting Manual Pages
============================
Table of Contents
=================

.. toctree::
:maxdepth: 2

man5/index
guide/accounting-guide
components/database-administration
index_man
9 changes: 9 additions & 0 deletions doc/index_man.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
.. _man-pages:

Manual Pages
============

.. toctree::
:maxdepth: 1

man5/index
Loading

0 comments on commit 039f066

Please sign in to comment.