-
Notifications
You must be signed in to change notification settings - Fork 149
Loggregator Component Properties
Warren Fernandes edited this page Mar 27, 2018
·
4 revisions
Here is some more detailed info on some of the Doppler properties and what they mean and do within the code. Of course, there may be a tendency for this list to diverge from actual properties.
Code Config/Manifest Property Name | Description |
---|---|
BlackListIps /doppler.blacklisted_syslog_ranges
|
Blacklist for IPs that should not be used as syslog drains, e.g. internal ip addresses. |
ContainerMetricTTLSeconds /doppler.container_metric_ttl_seconds
|
TTL (in seconds) for container usage metrics. It is used to obtain the latest container metrics for the app. |
LegacyIncomingMessagesPort /doppler.incoming_port
|
Defaults to 3456. This is legacy. It is no longer used in code and will be removed from the template spec. |
IncomingUDPPort /doppler.dropsonde_incoming_port
|
Defaults to 3457. This is the port Doppler listens on for messages from Metron via UDP. Metrics like `dropsondeListener.{receivedByteCount |
IncomingTCPPort /doppler.incoming_tcp_port
|
Defaults to 3458. Metrics like `tcpListener.{receivedByteCount |
EnableTLSTransport /doppler.tls.enable
|
Defaults to false . This makes Doppler listen over TLS. If this property is true then the TLSListenerConfig property must be configured as well. |
TLSListenerConfig |
Requires the TLS port, Doppler cert and key, and Loggregator CA Cert. |
EtcdRequireTLS /loggregator.etcd.require_ssl
|
Require TLS for communicating with etcd. |
EtcdTLSClientConfig |
Cert, key and CA files for communicating with etcd over TLS. |
MaxRetainedLogMessages /doppler.maxRetainedLogMessages
|
Defaults to 100. This is the buffer size for the Dump Sinks. Dump Sinks are created for every app because it contains the most n recentLogs of the app. If you want more logs for cf logs app_name --recent then configure this property. |
MessageDrainBufferSize /doppler.message_drain_buffer_size
|
Defaults to 10000. Size of the internal buffer used by doppler to store messages. If the buffer gets full doppler will drop the messages. This is used by the websocket and syslog sinks. |
MetricBatchIntervalMilliseconds |
Defaults to 5000ms or 5s. This doesn't have a corresponding spec property. This is the interval for how often Doppler will emit batched metrics. |
MetronAddress |
Defaults to 127.0.0.1:3457 . Doppler also needs to send metrics/logs about itself to Metron. |
MonitorIntervalSeconds |
Defaults to 60 seconds. This property is not in the template spec. This is used for the uptime monitor to send Doppler's uptime via dropsonde as a metric. |
OutgoingPort /doppler.outgoing_port
|
Defaults to 8081. This is the port Doppler listens on for requests from Traffic Controller. |
SharedSecret /doppler_endpoint.shared_secret
|
This secret is shared with metron. It is used to verify cryptographically signed dropsonde messages. |
SinkDialTimeoutSeconds / doppler.sink_dial_timeout_seconds
|
Defaults to 1s. Dial timeout for syslog sinks. |
SinkIOTimeoutSeconds / doppler.sink_io_timeout_seconds
|
Defaults to 0s. Write timeout for syslog sinks. This will only be set if greater than zero. See here |
SinkInactivityTimeoutSeconds / doppler.sink_inactivity_timeout_seconds
|
Defaults to 3600s. Interval before removing a sink due to inactivity. This is used by Dump Sinks and ContainerMetrics Sinks. |
SinkSkipCertVerify / doppler.syslog_skip_cert_verify
|
Defaults to true. When connecting over TLS, don't verify certificates for syslog sink. |
Syslog |
Defaults to vcap.doppler . This is used to setup a gosteno logger syslog sink when starting up Doppler. Has nothing to do with Doppler's Syslog Sinks for apps. |
UnmarshallerCount / doppler.unmarshaller_count
|
Defaults to 5. Number of parallel dropsonde unmarshallers to run within Doppler |
WebsocketWriteTimeoutSeconds / doppler.websocket_write_timeout_seconds
|
Defaults to 60s. This sets the write deadline for websocket sinks when streaming logs or firehose. See here |
PPROFPort / doppler.pprof_port
|
Defaults to 0 (Random port will be picked). This is the port for pprof runtime profiling data. |
Here is some more detailed info on some of the Metron properties and what they mean and do within the code. Of course, there may be a tendency for this list to diverge from actual properties.
Code Config/Manifest Property Name | Description |
---|---|
Syslog |
Defaults to vcap.metron_agent . This is used to setup a gosteno logger syslog sink when starting up Metron. |
IncomingUDPPort /metron_agent.dropsonde_incoming_port
|
Defaults to 3457. This is the port on which metron listens to locally for incoming dropsonde messages via UDP. |
LoggregatorDropsondePort /loggregator.dropsonde_incoming_port
|
Defaults to 3457. This is a duplicate of IncomingUDPPort . This seems to be a legacy property. Its usage should be switched to IncomingUDPPort . |
EtcdQueryIntervalMilliseconds |
This is not used. This needs to be removed soon. |
EtcdRequireTLS /loggregator.etcd.require_ssl
|
Require TLS for communicating with etcd. |
EtcdTLSClientConfig |
Cert, key and CA files for communicating with etcd over TLS. |
SharedSecret /metron_endpoint.shared_secret
|
This secret is shared with Doppler. It is used to prefix/sign each UDP message bytes. So when Doppler gets a message on its UDP listener it uses this shared secret to verify that the message bytes haven't been tampered with. |
MetricBatchIntervalMilliseconds |
Defaults to 5000ms or 5s. This doesn't have a corresponding spec property. This is the interval on which aggregated metrics are sent. |
RuntimeStatsIntervalMilliseconds |
Defaults to 15000ms or 15s. This doesn't have a corresponding spec property. This is the interval on which the memoryStats metrics are emitted. |
TCPBatchSizeBytes /metron_agent.tcp.batching_buffer_bytes
|
Defaults to 10240 bytes. The number of bytes which can be buffered prior to TCP writes (applies to TLS over TCP). Metron will error out if this is less than 1024 bytes. |
TCPBatchIntervalMilliseconds /...batching_buffer_flush_interval_milliseconds
|
Defaults to 100. The maximum time that a message can stay in the batching buffer before being sent to Doppler. |
PreferredProtocol /metron_agent.preferred_protocol
|
As of the latest implementation, this is more of a required protocol. That is if this set to tls and there aren't any Dopplers with TLS listeners, Metron will panic with a message of No available Dopplers . |
PPROFPort / metron_agent.pprof_port
|
Defaults to 0 (Random port will be picked). This is the port for pprof runtime profiling data. |
Here is some more detailed info on some of the traffic controller properties and what they mean and do within the code. Of course, there may be a tendency for this list to diverge from actual properties.
Code Config/Manifest Property Name | Description |
---|---|
EtcdUrls /loggregator.etcd.machines
|
The ETCD urls that Doppler advertises itself to. |
EtcdMaxConcurrentRequests /loggregator.etcd.maxconcurrentrequests
|
Sets the max workers for the work pool when setting up the etcdStoreAdapter
|
EtcdRequireTLS /loggregator.etcd.require_ssl
|
Require TLS for communicating with etcd. |
EtcdTLSClientConfig |
Cert, key and CA files for communicating with etcd over TLS. |
Syslog |
Set to vcap.trafficcontroller here. This is used to setup a gosteno logger syslog sink when starting up TC. This has nothing to do with Doppler's syslog sink. |
ApiHost /cc.srv_api_uri
|
TC uses this to verify if an app has log access permissions |
DopplerPort /loggregator.doppler_port
|
Defaults to 8081. TC uses this as part of the dopplerservice.LegacyFinder to build the doppler url by appending to the doppler IP retrieved from the legacy etcd key healthstatus/doppler... . These urls are used to create a websocket connection to dopplers. This is a legacy property.
|
OutgoingPort /traffic_controller.outgoing_port
|
Defaults to 8080. This is legacy. TC listens on this port for requests in legacy format. That is /tail/ , /dump , /recent which is converted to /apps/<appId>/stream and /apps/<appId>/recentlogs
|
OutgoingDropsondePort /loggregator.outgoing_dropsonde_port
|
Defaults to 8081. This is related to OutgoingPort . TC listens on this port for requests in the new format of /apps/<appId>/{stream/recentlogs/containermetrics}
|
MetronPort /metron_endpoint.dropsonde_port
|
Defaults to 3457. TC initializes dropsonde with this port. It sends its logs and metrics to Metron on this port |
SystemDomain /system_domain
|
TC uses this to set a cookie domain with the structure of doppler.<SystemDomain> when a request is made to TC with path /set-cookie. MORE DETAILS HERE. WHY IS THIS ENDPOINT USED?
|
SkipCertVerify /ssl.skip_cert_verify
|
Skips cert verification to the CC and the UAA. |
MonitorIntervalSeconds |
Defaults to 60 seconds. This property is not in the template spec. This is used for the uptime monitor to send TC's uptime via dropsonde as a metric. |
Zone /traffic_contoller.zone
|
DEPRECATED. We removed this from our spec because it isn't being used in the traffic controller job. |
PPROFPort / traffic_controller.pprof_port
|
Defaults to 0 (Random port will be picked). This is the port for pprof runtime profiling data. |