Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

arc-ce: Export arc-ce feature from aquilon at RAL #232

Draft
wants to merge 47 commits into
base: umd-4
Choose a base branch
from
Draft

arc-ce: Export arc-ce feature from aquilon at RAL #232

wants to merge 47 commits into from
+3,629 −0

Commits on Mar 5, 2020

  1. Add stub for base of arc-ce feature

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    7c7f6e8 View commit details
    Browse the repository at this point in the history

  2. Add stub for condor part of arc-ce config

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    05747ce View commit details
    Browse the repository at this point in the history

  3. Add support for blacklisting/banning named ARC-CE users 

    This prevents the named user accounts performing actions at the HTCondor level.
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    a1217e4 View commit details
    Browse the repository at this point in the history

  4. Add ARC-CE HTCondor security config 

    Specify lists of central service nodes, submitters, users and worker nodes.
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    e6e98ef View commit details
    Browse the repository at this point in the history

  5. Add ARC-CE HTCondor rules for specifying whether jobs require local x… 

    …rootd gateways
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    1f92321 View commit details
    Browse the repository at this point in the history

  6. ARC-CE: HTCondor: Map user account prefixes to accounting groups 

    These groups are then used for priorities, fairshares and limits.
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    c6efa55 View commit details
    Browse the repository at this point in the history

  7. ARC-CE: HTCondor: Use shared port for increased scalability 

    "Without the condor_shared_port daemon, approximately 2.1 ephemeral ports per running job are required, and possibly more, depending on the rate of job completion."
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    bceafc3 View commit details
    Browse the repository at this point in the history

  8. ARC-CE: HTCondor: Overwrite main HTCondor config file

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    3554ad0 View commit details
    Browse the repository at this point in the history

  9. ARC-CE: HTCondor: Add main condor config file

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    5442f32 View commit details
    Browse the repository at this point in the history

  10. ARC-CE: HTCondor: Configure schedd for job submission to farm

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    cb5efc2 View commit details
    Browse the repository at this point in the history

  11. ARC-CE: Start ARC services on boot

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    b35e3c4 View commit details
    Browse the repository at this point in the history

  12. ARC-CE: HTCondor: Configure condor_q to use preserve old behaviour 

    For compatibility with monitoring scripts and tooling that parses output directly
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    2a0c960 View commit details
    Browse the repository at this point in the history

  13. ARC-CE: HTCondor: Write history files for each job to reduce CPU usage 

    It is very improtant to clean these up automatically, otherwise disk space will disappear very quickly.
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    eaaef98 View commit details
    Browse the repository at this point in the history

  14. ARC-CE: Run fetch-crl more frequently than the default

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    64e5ee1 View commit details
    Browse the repository at this point in the history

  15. ARC-CE: Use filebeat to ship ARC job and HTCondor schedd logs

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    e642b80 View commit details
    Browse the repository at this point in the history

  16. ARC-CE: Find and overwrite large job output files with dummy content 

    Very large output files are known to cause problems with grid frameworks, so we knobble them.
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    7e5cf61 View commit details
    Browse the repository at this point in the history

  17. ARC-CE: Rotate ARC logs

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    b27bfc8 View commit details
    Browse the repository at this point in the history

  18. ARC-CE: Periodically count running and idle jobs per VO for accountin… 

    …g use

Add base of arc-ce feature - number of running & idle jobs by VO

Add arc-ce jobs config

Clean up jobs.pan

Add condorjobs.sh
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    8d2d01c View commit details
    Browse the repository at this point in the history

  19. ARC-CE: Remove archived accounting record after a number of days

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    6368194 View commit details
    Browse the repository at this point in the history

  20. ARC-CE: Local changes, GLUE generator and memory tweakiness

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    393ee23 View commit details
    Browse the repository at this point in the history

  21. ARC-CE: Local plugins 

    * scaling_factors_plugin.py
* job_logs_to_s3.py
* default_rte_plugin.py
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    44b4941 View commit details
    Browse the repository at this point in the history

  22. ARC-CE: Add stub for access control configuration

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    2e74e89 View commit details
    Browse the repository at this point in the history

  23. ARC-CE: Access Control: Configure LCAS

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    5e9e185 View commit details
    Browse the repository at this point in the history

  24. ARC-CE: Access Control: User banning

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    9a850a2 View commit details
    Browse the repository at this point in the history

  25. ARC-CE: Access Control: Configure LCMAPS for local ARGUS service

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    4fae781 View commit details
    Browse the repository at this point in the history

  26. ARC-CE: Build vomsdir from list of supported VOs

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    2e416dd View commit details
    Browse the repository at this point in the history

  27. ARC-CE: Access Control: Provide an empty local-grid-mapfile

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    569fc80 View commit details
    Browse the repository at this point in the history

  28. ARC-CE: Access Control: Provide an empty grid-mapfile

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    48dcbe2 View commit details
    Browse the repository at this point in the history

  29. ARC-CE: Access Control: Provide a default VO mapfile 

    This contains an empty mapping for all DNs to nobody as we only want ARGUS providing these mappings.
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    958f63f View commit details
    Browse the repository at this point in the history

  30. ARC-CE: Create required directories

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    36c4432 View commit details
    Browse the repository at this point in the history

  31. Include arc-ce/service

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    24ea1bc View commit details
    Browse the repository at this point in the history

  32. Repositories, GIIS and RPMs

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    439f6bf View commit details
    Browse the repository at this point in the history

  33. giis.conf

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    79b940e View commit details
    Browse the repository at this point in the history

  34. ARC-CE: Specify HTCondor and ARC versions

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    b654898 View commit details
    Browse the repository at this point in the history

  35. ARC-CE: Specify list of supported VOs

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    8b6e3a5 View commit details
    Browse the repository at this point in the history

  36. VO_SW_AREAS_USE_SWMGR?

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    41d4e80 View commit details
    Browse the repository at this point in the history

  37. ARC-CE: HTCondor: Transform jobs to use the Docker universe 

    * Map queue name to the operating system version, we provide two images the jobs will run in - EL6 and EL7.
* Specify the tags (versions) of the containers to allow us to control testing and rollout of updated images.
* Make use of external data scraped for DockerHub by and hourly cron to validate that specified tags do actually exist.
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    df3b26d View commit details
    Browse the repository at this point in the history

  38. Add arc-ce feature service configuration 

    Clean up service.pan

Add queues.conf

Add arc.conf
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    1e71882 View commit details
    Browse the repository at this point in the history

  39. Add arc-ce runtime environments 

    Clean up atlas-site-lcg.pan
Clean up biomed.pan
Clean up enmr.pan
Clean up glite.pan
Clean up proxy.pan
Clean up t2k.pan
Cleanup runtime-environments config
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    cf3a3ac View commit details
    Browse the repository at this point in the history

  40. Add CONDOR_LIMITS_CONTENTS 

    Move condor related file to subdirectory and update references - condor/limits
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    2ae1f8d View commit details
    Browse the repository at this point in the history

  41. Add CONDOR_RESOURCE_LIMITS_CONTENTS 

    Move condor related file to subdirectory and update references - resourcelimits
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    75e9279 View commit details
    Browse the repository at this point in the history

  42. Add missing accounting script to process condor history files

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    81ce824 View commit details
    Browse the repository at this point in the history

  43. Use correct lcas.db file

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    e2a0d91 View commit details
    Browse the repository at this point in the history

  44. Include arc-ce runtime environments

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    248dcb0 View commit details
    Browse the repository at this point in the history

  45. Launch jobs using July 2019 images 

    See https://github.com/stfc/grid-workernode/releases/tag/2019-07-02.1 for changes.
    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    ec7de19 View commit details
    Browse the repository at this point in the history

  46. Added dune to set of authorizedvo statements. Only affects publishing… 

    …, not ACLs, per http://www.nordugrid.org/documents/arc-ce-sysadm-guide.pdf
    @jrha
    David Crooks authored and jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    ddf198f View commit details
    Browse the repository at this point in the history

  47. ARC-CE: Place in features directory

    @jrha
    jrha committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    2b2e3e0 View commit details
    Browse the repository at this point in the history