diff --git a/README.md b/README.md index 0f6b8bf95..d2162047b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,6 @@ -[![Build Status](https://travis-ci.org/flux-framework/flux-accounting.svg?branch=master)](https://travis-ci.org/flux-framework/flux-accounting) +[![ci](https://github.com/flux-framework/flux-accounting/workflows/ci/badge.svg)](https://github.com/flux-framework/flux-accounting/actions?query=workflow%3A.github%2Fworkflows%2Fmain.yml) [![codecov](https://codecov.io/gh/flux-framework/flux-accounting/branch/master/graph/badge.svg)](https://codecov.io/gh/flux-framework/flux-accounting) +![docs](https://readthedocs.org/projects/flux-accounting/badge/?version=latest) _NOTE: The interfaces of flux-accounting are being actively developed and are not yet stable. The Github issue tracker is the primary way to communicate with @@ -37,89 +38,7 @@ flux-core that it is configured against; this is the default behavior of PYTHON_VERSION=3.7 ./configure --localstatedir=/var/ ``` -### Configuring flux-accounting background scripts - -There are a number of scripts that run in the background to update both job -usage and fairshare values. These require configuration upon setup of -flux-accounting. The first thing to configure when first setting up the -flux-accounting database is to set the PriorityUsageResetPeriod and -PriorityDecayHalfLife parameters. Both of these parameters represent a number of -weeks by which to hold usage factors up to the time period where jobs no longer -play a factor in calculating a usage factor. If these parameters are not passed -when creating the DB, PriorityDecayHalfLife is set to 1 week and -PriorityUsageResetPeriod is set to 4 weeks, i.e the flux-accounting database -will store up to a month's worth of jobs broken up into one week chunks: - -``` -flux account create-db --priority-decay-half-life=2 --priority-usage-reset-period=8 -``` - -The other component to load is the multi-factor priority plugin, which can be -loaded with `flux jobtap load`: - -``` -flux jobtap load mf_priority.so -``` - -After the DB and job priority plugin are set up, the `update-usage` subcommand -should be set up to run as a cron job. This subcommand fetches the most recent -job records for every user in the flux-accounting DB and calculates a new job -usage value. This subcommand takes one optional argument, -`--priority-decay-half-life`, which, like the parameter set in the database -creation step above, represents the number of weeks to hold one job usage -"chunk." If not specified, this optional argument also defaults to 1 week. - -``` -flux account update-usage --priority-decay-half-life=2 path/to/job-archive-DB -``` - -After the job usage values are re-calculated and updated, the fairshare values -for each user also need to be updated. This can be accomplished by configuring -the `flux-update-fshare` script to also run as a cron job. This fetches user -account data from the flux-accounting DB and recalculates and writes the updated -fairshare values back to the DB. - -``` -flux account-update-fshare -``` - -Once the fairshare values for all of the users in the flux-accounting DB get -updated, this updated information will be sent to the priority plugin. This -script can be also be configured to run as a cron job: - -``` -flux account-priority-update -``` - -### Run flux-accounting's commands: - -``` -usage: flux-account.py [-h] [-p PATH] [-o OUTPUT_FILE] - {view-user,add-user,delete-user,edit-user,view-job-records,create-db,add-bank,view-bank,delete-bank,edit-bank,print-hierarchy} ... - -Description: Translate command line arguments into SQLite instructions for the Flux Accounting Database. - -positional arguments: - {view-user,add-user,delete-user,edit-user,view-job-records,create-db,add-bank,view-bank,delete-bank,edit-bank,print-hierarchy} - sub-command help - view-user view a user's information in the accounting database - add-user add a user to the accounting database - delete-user remove a user from the accounting database - edit-user edit a user's value - view-job-records view job records - create-db create the flux-accounting database - add-bank add a new bank - view-bank view bank information - delete-bank remove a bank - edit-bank edit a bank's allocation - print-hierarchy print accounting database - -optional arguments: - -h, --help show this help message and exit - -p PATH, --path PATH specify location of database file - -o OUTPUT_FILE, --output-file OUTPUT_FILE - specify location of output file -``` +### Testing To run the unit tests in a Docker container, or launch into an interactive container on your local machine, you can run `docker-run-checks.sh`: @@ -136,78 +55,11 @@ mounting /Users/moussa1/src/flux-framework/flux-accounting as /usr/src [moussa1@docker-desktop src]$ ``` -### User and Bank Information - -The accounting tables in this database stores information like username and -ID, banks to submit jobs against, allocated shares to the user, as well as -static limits, including a max number of running jobs at a given time and -a max number of submitted jobs per user/bank combo. - -### Interacting With the Accounting DB - -In order to add, edit, or remove information from the flux-accounting database, -you must also have read/write access to the directory that the DB file resides -in. The [SQLite documentation](https://sqlite.org/omitted.html) states: - -> Since SQLite reads and writes an ordinary disk file, the only access -permissions that can be applied are the normal file access permissions of the -underlying operating system. - -There are two ways you can interact with the tables contained in the Accounting -DB. The first way is to launch into an interactive SQLite shell. From there, you -can open the database file and interface with any of the tables using SQLite -commands: - -``` -$ sqlite3 path/to/FluxAccounting.db -SQLite version 3.24.0 2018-06-04 14:10:15 -Enter ".help" for usage hints. -Connected to a transient in-memory database. -Use ".open FILENAME" to reopen on a persistent database. - -sqlite> .tables -association_table bank_table -``` - -To get nicely formatted output from queries (like headers for the tables and -proper spacing), you can also set the following options in your shell: - -``` -sqlite> .mode columns -sqlite> .headers on -``` - -This will output queries like the following: - -``` -sqlite> SELECT * FROM association_table; -creation_time mod_time deleted username bank shares max_jobs max_wall_pj -------------- ---------- ---------- ---------- ---------- ---------- ---------- ----------- -1605309320 1605309320 0 fluxuser foo 1 1 60 -``` - -The second way is to use flux-accounting's command line arguments. You can pass in -a path to the database file, or it will default to the "compiled-in" path of -`${prefix}/var/FluxAccounting.db`. - -With flux-accounting's command line tools, you can view a user's account -information, add and remove users to the accounting database, and edit an existing -user's account information: - -``` -$ flux account view-user fluxuser - -creation_time mod_time deleted username bank shares max_jobs max_wall_pj - 1595438356 1595438356 0 fluxuser foo 1 100 60 -``` - -Multiple rows of data can be loaded to the database at once using `.csv` files -and the `flux account pop-db` command. Run `flux account pop-db --help` for -`.csv` formatting instructions. +### Documentation -User and bank information can also be exported from the database using the -`flux account export-db` command, which will extract information from both the -user and bank tables and place them into `.csv` files. +To see detailed documentation about installing, administering, and configuring +flux-accounting, please see our official +[documentation pages](https://flux-accounting.readthedocs.io/en/latest/index.html). #### Release diff --git a/doc/components/database-administration.rst b/doc/components/database-administration.rst new file mode 100644 index 000000000..2a5fda53a --- /dev/null +++ b/doc/components/database-administration.rst @@ -0,0 +1,173 @@ +.. _database-administration: + +####################################### +Flux Accounting Database Administration +####################################### + +This document outlines the various methods of administration on the +flux-accounting database. + +******** +Overview +******** + +The tables in the flux-accounting database stores information for associations, +banks, job records, queues, and projects. + +For associations, shares, limits, queue and project permissions, and bank +membership can be managed. + +********************************** +Interacting With the Accounting DB +********************************** + +In order to add, edit, or remove information from the flux-accounting database, +you must also have read/write access to the directory that the DB file resides +in. The `SQLite documentation `_ states: + + Since SQLite reads and writes an ordinary disk file, the only access + permissions that can be applied are the normal file access permissions of + the underlying operating system. + +flux-accounting provides an interface to the SQLite database containing all of +the aforementioned data for associations, banks, queues, and projects. Commands +can be run to add/remove all of these, edit certain attributes, and view this +information in different formats. See ``flux account --help`` to list all of +the available flux-accounting commands. + +The bank/association database hierarchy assumes a tree structure, meaning that +there is one root bank which all other banks and associations descend from: + +.. code-block:: console + + $ flux account view-bank root -t + + Account Username RawShares RawUsage Fairshare + root 1 0.0 + bank_A 1 0.0 + bank_A user_1 1 0.0 0.5 + bank_B 1 0.0 + bank_B user_2 1 0.0 0.5 + bank_B user_3 1 0.0 0.5 + bank_C 1 0.0 + bank_C_a 1 0.0 + bank_C_a user_4 1 0.0 0.5 + bank_C_b 1 0.0 + bank_C_b user_5 1 0.0 0.5 + bank_C_b user_6 1 0.0 0.5 + +Multiple levels of banks can be defined under this root bank. Users can belong +to more than one bank and will have at most one default bank. + +To add a bank to the database, you can use the ``flux account add-bank`` +command. Each ``add-bank`` call requires a bank name, their allocated shares, +and a parent bank name (if it is not the root bank): + +.. code-block:: console + + $ flux account add-bank root 1 + $ flux account add-bank --parent-bank=root bank_A 1 + +From here, users can be added to these banks to create *associations*, a +2-tuple combination of a username and bank name: + +.. code-block:: console + + $ flux account add-user --username=user_1 --bank=bank_A + +If you wish to delete an association or bank from the database, you can run the +``flux account delete-user`` or ``flux account delete-bank`` commands. Note +that this will not actually remove the association's or bank's row from the +table where their data is stored, but will instead set their ``active`` column +to 0. To re-enable an association's ``active`` status, simply re-add them with +``flux account add-user``. + +Information for associations can be viewed in more than one format: + +.. code-block:: console + + $ flux account view-user user_1 + + creation_time: 1728922316 + mod_time: 1728922316 + active: 1 + username: user_1 + userid: 5001 + bank: bank_A + default_bank: bank_A + shares: 1 + job_usage: 0.0 + fairshare: 0.5 + max_running_jobs: 5 + max_active_jobs: 7 + max_nodes: 2147483647 + queues: + projects: * + default_project: * + + $ flux account view-user user_1 --json + + { + "username": "user_1", + "userid": 5001, + "default_bank": "bank_A", + "banks": [ + { + "bank": "bank_A", + "active": 1, + "shares": 1, + "job_usage": 0.0, + "fairshare": 0.5, + "max_running_jobs": 5, + "max_active_jobs": 7, + "max_nodes": 2147483647, + "queues": "", + "projects": "*", + "default_project": "*" + } + ] + } + +If you are familiar with SQLite syntax, you can also launch into an interactive +SQLite shell. From there, you can open the database file and interface with +any of the tables using SQLite commands: + +.. code-block:: console + + $ sqlite3 path/to/FluxAccounting.db + SQLite version 3.24.0 2018-06-04 14:10:15 + Enter ".help" for usage hints. + Connected to a transient in-memory database. + Use ".open FILENAME" to reopen on a persistent database. + + sqlite> + +To get nicely formatted output from queries (like headers for the tables and +proper spacing), you can also set the following options in your shell: + +.. code-block:: console + + sqlite> .mode columns + sqlite> .headers on + +This will output queries like the following: + +.. code-block:: console + + sqlite> SELECT * FROM association_table; + creation_time mod_time deleted username bank shares max_jobs max_wall_pj + ------------- ---------- ---------- ---------- ---------- ---------- ---------- ----------- + 1605309320 1605309320 0 fluxuser foo 1 1 60 + + +********************************** +Flux-Accounting Data Import/Export +********************************** + +Multiple rows of data can be loaded to the database at once using ``.csv`` files +and the ``flux account pop-db`` command. Run ``flux account pop-db --help`` for +``.csv`` formatting instructions. + +User and bank information can also be exported from the database using the +``flux account export-db`` command, which will extract information from both the +user and bank tables and place them into ``.csv`` files. diff --git a/doc/index.rst b/doc/index.rst index 5f21743fe..b9f5153f9 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -4,18 +4,12 @@ Documentation for flux-accounting This site contains documentation for flux-accounting configuration and management. -.. toctree:: - :maxdepth: 2 - - guide/accounting-guide - - -.. _man-pages: - -flux-accounting Manual Pages -============================ +Table of Contents +================= .. toctree:: :maxdepth: 2 - man5/index + guide/accounting-guide + components/database-administration + index_man diff --git a/doc/index_man.rst b/doc/index_man.rst new file mode 100644 index 000000000..21e47d272 --- /dev/null +++ b/doc/index_man.rst @@ -0,0 +1,9 @@ +.. _man-pages: + +Manual Pages +============ + +.. toctree:: + :maxdepth: 1 + + man5/index \ No newline at end of file diff --git a/doc/man5/index.rst b/doc/man5/index.rst index 6bc66fa61..713d71752 100644 --- a/doc/man5/index.rst +++ b/doc/man5/index.rst @@ -1,8 +1,8 @@ -man5 -==== +Section 5 - Flux Configuration File +=================================== .. toctree:: - :caption: File formats and conventions :maxdepth: 1 + :glob: - flux-config-accounting + *