[EPIC] Internal or manual conformance tests and audits at CSP side

[mbuechse]

edit text (mainly) by @markus-hentsch:

More recent SCS standards which go beyond simple user-facing aspects of an SCS infrastructure create an increasing demand for a standardized approach for executing internal and/or manual conformance tests and audits in the infrastructure directly.
This is necessary for test and audit scenarios where the information is not accessible from the outside (e.g. via API) and/or requires admin privileges.

This issue should serve as a starting point to establish such a process as well as keeping track of applicable standards.

The exact implementation is up for debate. So far there have been some ideas about test scripts that can be executed by a CSP admin and generate a report file which can be submitted to some SCS service. Challenges arise when the tests need to go deep into the infrastructure (e.g. checking the config file entries of every Neutron agent) and infrastructures may differ greatly in their architecture between CSPs making it hard to create a one-size-fits-all script for verification.

Applicable Standards

The standards which currently have limited conformance test capabilities due to the lack of CSP-side audit possibilites will be tracked here:

• volume backup functionality
  • check that Cinder Backup is configured to a separate storage backend different from the volume storage backend
• secure connections
  • check that RPC and DB channels are encrypted
  • check that internal API channels are encrypted
  • assess libvirt interface and Nova live migration security
  • assess Neutron network security
• tenant VM DNS
  • check mandated Neutron server & agent config values related to DNS
• Domain Manager
  • test preparation requires cloud admin rights (creating domain-manager users)
• Key Manager
  • test whether the Master-KEK is protected (MAster KEK should not be written in plain text on the Key Manager Host)
• Availability Zones
  • there are physical requirements, that need tests with a physical component (someone going through the deployment)
    • fire zones
    • redundancy of power supplies (line or generator // redundant PDUs)
    • redundant cooling system
    • etc....