This document describes a set of runtime flags available in cAdvisor.
--store_container_labels=false- do not convert container labels and environment variables into labels on prometheus metrics for each container.--whitelisted_container_labels- comma separated list of container labels to be converted to labels on prometheus metrics for each container.store_container_labelsmust be set to false for this to take effect.
--env_metadata_whitelist: a comma-separated list of environment variable keys that needs to be collected for containers, only support containerd and docker runtime for now.
--docker_only=false- do not report raw cgroup metrics, except the root cgroup.--raw_cgroup_prefix_whitelist- a comma-separated list of cgroup path prefix that needs to be collected even when--docker_onlyis specified--disable_root_cgroup_stats=false- disable collecting root Cgroup stats.
Container hints are a way to pass extra information about a container to cAdvisor. In this way cAdvisor can augment the stats it gathers. For more information on the container hints format see its definition. Note that container hints are only used by the raw container driver today.
--container_hints="/etc/cadvisor/container_hints.json": location of the container hints file
--enable_load_reader=false: Whether to enable cpu load reader
--max_procs=0: max number of CPUs that can be used simultaneously. Less than 1 for default (number of cores).
cAdvisor-native flags that help in debugging:
--log_backtrace_at="": when logging hits line file:N, emit a stack trace
--log_cadvisor_usage=false: Whether to log the usage of the cAdvisor container
--version=false: print cAdvisor version and exit
--profiling=false: Enable profiling via web interface host:port/debug/pprof/
From glog here are some flags we find useful:
--log_dir="": If non-empty, write log files in this directory
--logtostderr=false: log to standard error instead of files
--alsologtostderr=false: log to standard error as well as files
--stderrthreshold=0: logs at or above this threshold go to stderr
--v=0: log level for V logs
--vmodule=: comma-separated list of pattern=N settings for file-filtered logging
--docker="unix:///var/run/docker.sock": docker endpoint (default "unix:///var/run/docker.sock")
--docker_root="/var/lib/docker": DEPRECATED: docker root is read from docker info (this is a fallback, default: /var/lib/docker) (default "/var/lib/docker")
--docker-tls: use TLS to connect to docker
--docker-tls-cert="cert.pem": client certificate for TLS-connection with docker
--docker-tls-key="key.pem": private key for TLS-connection with docker
--docker-tls-ca="ca.pem": trusted CA for TLS-connection with docker
--podman="unix:///var/run/podman/podman.sock": podman endpoint (default "unix:///var/run/podman/podman.sock")Housekeeping is the periodic actions cAdvisor takes. During these actions, cAdvisor will gather container stats. These flags control how and when cAdvisor performs housekeeping.
Dynamic housekeeping intervals let cAdvisor vary how often it gathers stats. It does this depending on how active the container is. Turning this off provides predictable housekeeping intervals, but increases the resource usage of cAdvisor.
--allow_dynamic_housekeeping=true: Whether to allow the housekeeping interval to be dynamic
Intervals for housekeeping. cAdvisor has two housekeepings: global and per-container.
Global housekeeping is a singular housekeeping done once in cAdvisor. This typically does detection of new containers. Today, cAdvisor discovers new containers with kernel events so this global housekeeping is mostly used as backup in the case that there are any missed events.
Per-container housekeeping is run once on each container cAdvisor tracks. This typically gets container stats.
--global_housekeeping_interval=1m0s: Interval between global housekeepings
--housekeeping_interval=1s: Interval between container housekeepings
--max_housekeeping_interval=1m0s: Largest interval to allow between container housekeepings (default 1m0s)
Specify where cAdvisor listens.
--http_auth_file="": HTTP auth file for the web UI
--http_auth_realm="localhost": HTTP auth realm for the web UI (default "localhost")
--http_digest_file="": HTTP digest file for the web UI
--http_digest_realm="localhost": HTTP digest file for the web UI (default "localhost")
--listen_ip="": IP to listen on, defaults to all IPs
--port=8080: port to listen (default 8080)
--url_base_prefix=/: optional path prefix aded to all resource URLs; useful when running cAdvisor behind a proxy. (default /)
cAdvisor stores the latest historical data in memory. How long of a history it stores can be configured with the --storage_duration flag.
--storage_duration=2m0s: How long to store data.
--boot_id_file="/proc/sys/kernel/random/boot_id": Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
--machine_id_file="/etc/machine-id,/var/lib/dbus/machine-id": Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
--update_machine_info_interval=5m: Interval between machine info updates. (default 5m)
--application_metrics_count_limit=100: Max number of application metrics to store (per container) (default 100)
--collector_cert="": Collector's certificate, exposed to endpoints for certificate based authentication.
--collector_key="": Key for the collector's certificate
--disable_metrics=<metrics>: comma-separated list of metrics to be disabled. Options are accelerator,advtcp,app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,referenced_memory,resctrl,sched,tcp,udp. (default advtcp,cpu_topology,cpuset,hugetlb,memory_numa,process,referenced_memory,resctrl,sched,tcp,udp)
--enable_metrics=<metrics>: comma-separated list of metrics to be enabled. If set, overrides 'disable_metrics'. Options are accelerator,advtcp,app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,referenced_memory,resctrl,sched,tcp,udp.
--prometheus_endpoint="/metrics": Endpoint to expose Prometheus metrics on (default "/metrics")
--disable_root_cgroup_stats=false: Disable collecting root Cgroup stats
--storage_driver="": Storage driver to use. Data is always cached shortly in memory, this controls where data is pushed besides the local cache. Empty means none. Options are: <empty>, bigquery, elasticsearch, influxdb, kafka, redis, statsd, stdout
--storage_driver_buffer_duration="1m0s": Writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction (default 1m0s)
--storage_driver_db="cadvisor": database name (default "cadvisor")
--storage_driver_host="localhost:8086": database host:port (default "localhost:8086")
--storage_driver_password="root": database password (default "root")
--storage_driver_secure=false: use secure connection with database
--storage_driver_table="stats": table name (default "stats")
--storage_driver_user="root": database username (default "root")
--perf_events_config="" Path to a JSON file containing configuration of perf events to measure. Empty value disables perf events measuring.
Core perf events can be exposed on Prometheus endpoint per CPU or aggregated by event. It is controlled through --disable_metrics and --enable_metrics parameters with option percpu, e.g.:
--disable_metrics="percpu"- core perf events are aggregated--disable_metrics=""- core perf events are exposed per CPU.
It's possible to get "too many opened files" error when a lot of perf events are exposed per CPU. This happens because of passing system limits.
Try to increase max number of file desctriptors with ulimit -n <value>.
Aggregated form of core perf events significantly decrease volume of data. For aggregated form of core perf events scaling ratio (container_perf_metric_scaling ratio) indicates the lowest value of scaling ratio for specific event to show the worst precision.
One of the goals of kernel perf subsystem is to instrument CPU performance counters that allow to profile applications. Profiling is performed by setting up performance counters that count hardware events (e.g. number of retired instructions, number of cache misses). The counters are CPU hardware registers and amount of them is limited.
Other goals of perf subsystem (such as tracing) are beyond the scope of this documentation and you can follow Further Reading section below to learn more about them.
Familiarize yourself with following perf-event-related terms:
multiplexing- 2nd Generation Intel® Xeon® Scalable Processors provides 4 counters per each hyper thread. If number of configured events is greater than number of available counters then Linux will multiplex counting and some (or even all) of the events will not be accounted for all the time. In such situation information about amount of time that event was accounted for and amount of time when event was enabled is provided. Counter value that cAdvisor exposes is scaled automatically.grouping- in scenario when accounted for events are used to calculate derivative metrics, it is reasonable to measure them in transactional manner: all the events in a group must be accounted for in the same period of time. Keep in mind that it is impossible to group more events that there are counters available.uncore events- events which can be counted by PMUs outside core.PMU- Performance Monitoring Unit
Using perf tools:
- Identify the event in
perf listoutput. - Execute command:
perf stat -I 5000 -vvv -e EVENT_NAME - Find
perf_event_attrsection onperf statoutput, copy config and type field to configuration file.
------------------------------------------------------------
perf_event_attr:
type 18
size 112
config 0x304
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
exclude_guest 1
------------------------------------------------------------
- Configuration file should look like:
{
"core": {
"events": [
"event_name"
],
"custom_events": [
{
"type": 18,
"config": [
"0x304"
],
"name": "event_name"
}
]
},
"uncore": {
"events": [
"event_name"
],
"custom_events": [
{
"type": 18,
"config": [
"0x304"
],
"name": "event_name"
}
]
}
}Config values can be also obtain from:
Uncore Event name should be in form PMU_PREFIX/event_name where PMU_PREFIX mean
that statistics would be counted on all PMUs with that prefix in name.
Let's explain this by example:
{
"uncore": {
"events": [
"uncore_imc/cas_count_read",
"uncore_imc_0/cas_count_write",
"cas_count_all"
],
"custom_events": [
{
"config": [
"0x304"
],
"name": "uncore_imc_0/cas_count_write"
},
{
"type": 19,
"config": [
"0x304"
],
"name": "cas_count_all"
}
]
}
}-
uncore_imc/cas_count_read- because ofuncore_imctype and no entry in custom events, it would be counted by all Integrated Memory Controller PMUs with config provided from libpfm package. (using this function: https://man7.org/linux/man-pages/man3/pfm_get_os_event_encoding.3.html) -
uncore_imc_0/cas_count_write- because ofuncore_imc_0type and entry in custom events it would be counted byuncore_imc_0PMU with provided config. -
uncore_imc_1/cas_count_all- because of entry in custom events with type field, event would be counted by PMU with 19 type and provided config.
It is possible to configure perf events by names using events supported in libpfm4, for detailed information please see libpfm4 documentation.
Discovery of perf events supported on platform can be made using python script - pmu.py provided with libpfm4, please see script reqirements.
Example output of pmu.py:
$ python pmu.py
INSTRUCTIONS 1
u 0
k 1
period 3
freq 4
precise 5
excl 6
mg 7
mh 8
cpu 9
pinned 10
INSTRUCTION_RETIRED 192
e 2
i 3
c 4
t 5
intx 7
intxcp 8
u 0
k 1
period 3
freq 4
excl 6
mg 7
mh 8
cpu 9
pinned 10
UNC_M_CAS_COUNT 4
RD 3
WR 12
e 0
i 1
t 2
period 3
freq 4
excl 6
cpu 9
pinned 10
and perf events configuration for listed events:
{
"core": {
"events": [
"instructions",
"instruction_retired"
]
},
"uncore": {
"events": [
"uncore_imc/unc_m_cas_count:rd",
"uncore_imc/unc_m_cas_count:wr"
]
}
}Notice: PMU_PREFIX is provided in the same way as for configuration with config values.
{
"core": {
"events": [
["instructions", "instruction_retired"]
]
},
"uncore": {
"events": [
["uncore_imc_0/unc_m_cas_count:rd", "uncore_imc_0/unc_m_cas_count:wr"],
["uncore_imc_1/unc_m_cas_count:rd", "uncore_imc_1/unc_m_cas_count:wr"]
]
}
}- perf Examples on Brendan Gregg's blog
- Kernel Perf Wiki
man perf_event_open- perf subsystem in Linux kernel
- Uncore Performance Monitoring Reference Manuals
See example configuration below:
{
"core": {
"events": [
"instructions",
"instructions_retired"
],
"custom_events": [
{
"type": 4,
"config": [
"0x5300c0"
],
"name": "instructions_retired"
}
]
},
"uncore": {
"events": [
"uncore_imc/cas_count_read"
],
"custom_events": [
{
"config": [
"0xc04"
],
"name": "uncore_imc/cas_count_read"
}
]
}
}In the example above:
instructionswill be measured as a non-grouped event and is specified using human friendly interface that can be obtained by callingperf list. You can use any name that appears in the output ofperf listcommand. This is interface that majority of users will rely on.instructions_retiredwill be measured as non-grouped event and is specified using an advanced API that allows to specify any perf event available (some of them are not named and can't be specified with plain string). Event name should be a human readable string that will become a metric name.cas_count_readwill be measured as uncore non-grouped event on all Integrated Memory Controllers Performance Monitoring Units because of unsettypefield anduncore_imcprefix.
To gain metrics, cAdvisor creates own monitoring groups with cadvisor prefix.
Resctrl file system is not hierarchical like cgroups, so users should set --docker_only flag to avoid race conditions and unexpected behaviours.
--resctrl_interval=0: Resctrl mon groups updating interval. Zero value disables updating mon groups.