Enable global file logging (#2371)

This ensures we can specify file logging globally and uniformly. Greatly simplifying the debug-ability of the agent.

The preferred way is to set any or all of the following:


* `ELASTIC_OTEL_LOG_TARGETS` (to anything but `none`) 
  * if only set to `stdout` no file will be created but global logging will kicking )
* `OTEL_DOTNET_AUTO_LOG_DIRECTORY`
* `OTEL_LOG_LEVEL`  
  * `trace` `debug` will enable global file logging if the other two variables are not set  explicitly.

This ensure we have one way to debug both our proprietary agent as well as the [Elastic OpenTelemetry Distribution](https://github.com/elastic/elastic-otel-dotnet).

See: elastic/elastic-otel-dotnet#129

For backwards compatible reasons the profiler variables are also supported:

* `ELASTIC_APM_PROFILER_LOG`
* `ELASTIC_APM_PROFILER_LOG_DIR`
* `ELASTIC_APM_PROFILER_LOG_TARGETS` 

Globally setting 

* `ELASTIC_APM_LOG_LEVEL` and `ELASTIC_APM_LOG_DIRECTORY` is also supported but not preferred.


Setting these in any of our supported deploy scenarios:

* Manual instrumentation (nuget)
  * ASP.NET (classic)
  * ASP.NET core
    * NOTE: we now log to both the configured ILogger and our global logging.
* Auto Instrumentation
  * Profiler
  * Startup hooks
    To keep support existing deploys this now always globally logs to file if only `ELASTIC_APM_STARTUP_HOOKS_LOGGING` is set as well.
 


This further updates our docs for the profiler and troubleshooting to prefer `ELASTIC_OTEL_*` variables. 

The profiler is updated to read the same environment variables as managed code.

docs/setup-auto-instrumentation.asciidoc

@@ -398,7 +398,7 @@ A semi-colon separated list of APM service names to exclude from auto-instrument
 Values defined are checked against the value of <<config-service-name,`ELASTIC_APM_SERVICE_NAME`>>
 environment variable.
 
-`ELASTIC_APM_PROFILER_LOG` _(optional)_::
+`ELASTIC_OTEL_LOG_LEVEL` _(optional)_::
 
 The log level at which the profiler should log. Valid values are
 
@@ -409,11 +409,15 @@ The log level at which the profiler should log. Valid values are
 * error
 * none
 
+
 The default value is `warn`. More verbose log levels like `trace` and `debug` can
 affect the runtime performance of profiler auto instrumentation, so are recommended
 _only_ for diagnostics purposes.
 
-`ELASTIC_APM_PROFILER_LOG_DIR` _(optional)_::
+This takes precedence over the now deprecated `ELASTIC_APM_PROFILER_LOG`
+
+
+`ELASTIC_OTEL_LOG_DIRECTORY` _(optional)_::
 
 The directory in which to write profiler log files. If unset, defaults to
 
@@ -424,6 +428,8 @@ If the default directory cannot be written to for some reason, the profiler
 will try to write log files to a `logs` directory in the home directory specified 
 by `ELASTIC_APM_PROFILER_HOME` environment variable.
 
+This takes precedence over the now deprecated `ELASTIC_APM_PROFILER_LOG_DIR`
+
 [IMPORTANT]
 --
 The user account under which the profiler process runs must have permission to 
@@ -432,7 +438,8 @@ on IIS, the https://learn.microsoft.com/en-us/iis/manage/configuring-security/ap
 has write permissions in the target directory.
 --
 
-`ELASTIC_APM_PROFILER_LOG_TARGETS` _(optional)_::
+
+`ELASTIC_OTEL_LOG_TARGETS` _(optional)_::
 
 A semi-colon separated list of targets for profiler logs. Valid values are
 
@@ -441,3 +448,5 @@ A semi-colon separated list of targets for profiler logs. Valid values are
 
 The default value is `file`, which logs to the directory specified by
 `ELASTIC_APM_PROFILER_LOG_DIR` environment variable.
+
+This takes precedence over the now deprecated `ELASTIC_APM_PROFILER_LOG_TARGETS`

docs/troubleshooting.asciidoc

@@ -32,7 +32,58 @@ If you don't see anything suspicious in the agent logs (no warning or error), it
 [[collect-agent-logs]]
 === Collecting agent logs
 
-The way to collect logs depends on the setup of your application.
+[float]
+[[collect-logs-globally]]
+==== Enable global file logging.
+
+The easiest way to get debug information from the Agent, regardless of the way it's run, is to enable global file logging.
+
+Specifying at least one of the following environment variables will ensure the agent logs to a file
+
+`OTEL_LOG_LEVEL` _(optional)_::
+
+The log level at which the profiler should log. Valid values are
+
+* trace
+* debug
+* info
+* warn
+* error
+* none
+
+
+The default value is `warn`. More verbose log levels like `trace` and `debug` can
+affect the runtime performance of profiler auto instrumentation, so are recommended
+_only_ for diagnostics purposes.
+
+NOTE: if `ELASTIC_OTEL_LOG_TARGETS` is not explicitly set to include `file` global file logging will only 
+be enabled when configured with `trace` or `debug`.
+
+`OTEL_DOTNET_AUTO_LOG_DIRECTORY` _(optional)_::
+
+The directory in which to write log files. If unset, defaults to
+
+* `%PROGRAMDATA%\elastic\apm-agent-dotnet\logs` on Windows
+* `/var/log/elastic/apm-agent-dotnet` on Linux
+
+
+[IMPORTANT]
+--
+The user account under which the profiler process runs must have permission to
+write to the destination log directory. Specifically, ensure that when running
+on IIS, the https://learn.microsoft.com/en-us/iis/manage/configuring-security/application-pool-identities[AppPool identity]
+has write permissions in the target directory.
+--
+
+`ELASTIC_OTEL_LOG_TARGETS` _(optional)_::
+
+A semi-colon separated list of targets for profiler logs. Valid values are
+
+* file
+* stdout
+
+The default value is `file` if `OTEL_DOTNET_AUTO_LOG_DIRECTORY` is set or `OTEL_LOG_LEVEL` is set to `trace` or `debug`.
+
 
 [float]
 [[collect-logs-core]]
@@ -62,49 +113,9 @@ For example, the following configuration in `appsettings.json` limits APM agents
 ----
 <1> Control the verbosity of the agent logs by setting the log level for the `Elastic.Apm` category
 
-[float]
-[[collect-logs-classic]]
-==== ASP.NET Classic
-
-ASP.NET (classic) does not have a predefined logging system. By default, the agent is configured to
-emit log messages to a 
-https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.tracesource[`System.Diagnostics.TraceSource`] 
-with the source name `"Elastic.Apm"`. The TraceSource adheres to the log levels defined in the
-APM agent configuration. Typically, you will configure a preferred log level using an application setting in `web.config`.
-
-[IMPORTANT]
---
-System.Diagnostics.TraceSource requires the https://docs.microsoft.com/en-us/dotnet/framework/debug-trace-profile/how-to-compile-conditionally-with-trace-and-debug[`TRACE` compiler directive to be specified], which is specified
-by default for both Debug and Release build configurations.
---
-
-https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.tracelistener[TraceListeners]
-can be configured to monitor log messages for the trace source, using the https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/trace-debug/system-diagnostics-element[`<system.diagnostics>`] section of
-web.config. For example, the following web.config section writes Elastic.Apm log messages to a file
-named my_log_file.log:
-
-[source,xml]
-----
-<configuration>
-  <!-- other sections .... -->
-  <system.diagnostics>
-  <sources>
-    <source name="Elastic.Apm"> <1>
-      <listeners>
-        <add name="file" 
-          type="System.Diagnostics.TextWriterTraceListener" 
-          initializeData="my_log_file.log" />
-      </listeners>
-    </source>
-  </sources>
-</system.diagnostics>
-</configuration>  
-----
-<1> Define listeners under a source with name `"Elastic.Apm"` to capture agent logs
-
 [float]
 [[collect-logs-class-other-logging-systems]]
-===== Other logging systems
+==== Other logging systems
 
 If you have a logging system in place such as https://nlog-project.org/[NLog], https://serilog.net/[Serilog], 
 or similar, you can direct the agent logs into your logging system by creating an adapter between 
@@ -242,17 +253,7 @@ The agent is only supported on IIS7 and higher where the `Integrated Pipeline Mo
 [[startup-hook-failure]]
 === Startup hooks failure
 
-If the <<zero-code-change-setup, startup hook>> integration throws an exception, additional detail can be obtained by 
-setting the `ELASTIC_APM_STARTUP_HOOKS_LOGGING` environment variable before starting the application
-
-[source,sh]
-----
-set ELASTIC_APM_STARTUP_HOOKS_LOGGING=1
-----
-
-and then running the application in a context where the environment variable will be visible. In setting this value,
-an `ElasticApmAgentStartupHook.log` file is written to the directory containing the startup hook assembly, in addition to
-writing to standard output.
+If the <<zero-code-change-setup, startup hook>> integration throws an exception, additional detail can be obtained through <<collect-logs-globally, enabling the global log collection>>.
 
 [float]
 [[agent-overhead]]

src/Elastic.Apm/AgentComponents.cs

@@ -52,7 +52,7 @@ internal AgentComponents(
 				var config = CreateConfiguration(logger, configurationReader);
 				hostNameDetector ??= new HostNameDetector();
 
-				Logger = logger ?? CheckForProfilerLogger(DefaultLogger(null, configurationReader), config.LogLevel);
+				Logger = logger ?? GetGlobalLogger(DefaultLogger(null, configurationReader), config.LogLevel);
 				ConfigurationStore = new ConfigurationStore(new RuntimeConfigurationSnapshot(config), Logger);
 
 				Service = Service.GetDefaultService(config, Logger);
@@ -199,38 +199,39 @@ private static IConfigurationReader CreateConfiguration(IApmLogger logger, IConf
 #endif
 		}
 
-
-		//
-		// This is the hooking point that checks for the existence of profiler-related
-		// logging settings.
-		// If no agent logging is configured but we detect profiler logging settings, those
-		// will be honoured.
-		// The finer-grained log-level (agent vs profiler) will be used.
-		// This has the benefit that users will also get agent logs in addition to profiler-only
-		// logs.
-		//
-		internal static IApmLogger CheckForProfilerLogger(IApmLogger fallbackLogger, LogLevel agentLogLevel, IDictionary environmentVariables = null)
+		/// <summary>
+		/// This ensures agents will respect externally provided loggers.
+		/// <para>If the agent is started as part of profiling it should adhere to profiling configuration</para>
+		/// <para>If file logging environment variables are set we should always log to that location</para>
+		/// </summary>
+		/// <param name="fallbackLogger"></param>
+		/// <param name="agentLogLevel"></param>
+		/// <param name="environmentVariables"></param>
+		/// <returns></returns>
+		internal static IApmLogger GetGlobalLogger(IApmLogger fallbackLogger, LogLevel agentLogLevel, IDictionary environmentVariables = null)
 		{
 			try
 			{
-				var profilerLogConfig = ProfilerLogConfig.Check(environmentVariables);
-				if (profilerLogConfig.IsActive)
+				var fileLogConfig = GlobalLogConfiguration.FromEnvironment(environmentVariables);
+				if (!fileLogConfig.IsActive)
 				{
-					var effectiveLogLevel = LogLevelUtils.GetFinest(agentLogLevel, profilerLogConfig.LogLevel);
+					fallbackLogger.Info()?.Log("No system wide logging configured, defaulting to fallback logger");
+					return fallbackLogger;
+				}
 
-					if ((profilerLogConfig.LogTargets & ProfilerLogTarget.File) == ProfilerLogTarget.File)
-						TraceLogger.TraceSource.Listeners.Add(new TextWriterTraceListener(profilerLogConfig.LogFilePath));
-					if ((profilerLogConfig.LogTargets & ProfilerLogTarget.StdOut) == ProfilerLogTarget.StdOut)
-						TraceLogger.TraceSource.Listeners.Add(new TextWriterTraceListener(Console.Out));
+				var effectiveLogLevel = LogLevelUtils.GetFinest(agentLogLevel, fileLogConfig.LogLevel);
+				if ((fileLogConfig.LogTargets & GlobalLogTarget.File) == GlobalLogTarget.File)
+					TraceLogger.TraceSource.Listeners.Add(new TextWriterTraceListener(fileLogConfig.AgentLogFilePath));
+				if ((fileLogConfig.LogTargets & GlobalLogTarget.StdOut) == GlobalLogTarget.StdOut)
+					TraceLogger.TraceSource.Listeners.Add(new TextWriterTraceListener(Console.Out));
 
-					var logger = new TraceLogger(effectiveLogLevel);
-					logger.Info()?.Log($"{nameof(ProfilerLogConfig)} - {profilerLogConfig}");
-					return logger;
-				}
+				var logger = new TraceLogger(effectiveLogLevel);
+				logger.Info()?.Log($"{nameof(fileLogConfig)} - {fileLogConfig}");
+				return logger;
 			}
 			catch (Exception e)
 			{
-				fallbackLogger.Warning()?.LogException(e, "Error in CheckForProfilerLogger");
+				fallbackLogger.Warning()?.LogException(e, "Error in GetGlobalLogger");
 			}
 			return fallbackLogger;
 		}

src/Elastic.Apm/Config/Net4FullFramework/FullFrameworkDefaultImplementations.cs

@@ -22,7 +22,7 @@ internal static IApmLogger CreateDefaultLogger(LogLevel? configuredDefault)
 		if (!string.IsNullOrEmpty(logLevel))
 			Enum.TryParse(logLevel, true, out level);
 
-		return AgentComponents.CheckForProfilerLogger(new TraceLogger(level), level);
+		return AgentComponents.GetGlobalLogger(new TraceLogger(level), level);
 	}
 
 	/// <summary>