Skip to content

Latest commit

 

History

History
235 lines (155 loc) · 11.4 KB

14-validation-logs.md

File metadata and controls

235 lines (155 loc) · 11.4 KB
Raw

Access resource logs & Microsoft Defender for Cloud data

Your infrastructure and workload is emitting logs, here are a few key logs you may wish to consider being familiar with or building log-based queries around. Below are some a few queries to get you started exploring the captured data.

You can access these logs all directly from the attached Log Analytics workspaces, but when you do you'll need to filter to specific resources. For simplicity the following steps direct you to the pre-filtered view offered by the Azure Portal when viewing within the context of each service.

Remember, since this implementation builds on the AKS Baseline, validations performed there such as viewing Prometheus metrics and Kured logs are also applicable to this cluster.

Azure Firewall

Azure Firewall is the last network control point in your architecture. It captures logs for traffic that is allowed and denied.

  1. In the Azure Portal, navigate to your Azure Firewall resource.
  2. Click on Logs.

View traffic blocked by Azure Firewall

Your Azure Firewall has already blocked traffic originating in the cluster headed to https://example.org as part of the workload. To see those denied requests, execute the following query:

AzureDiagnostics
| where Category == "AzureFirewallApplicationRule"
| where msg_s contains "example.org:443"
| order by TimeGenerated desc

View Azure Firewall DNS proxy logs

Your Azure Firewall is acting as a DNS proxy for your spokes. To see DNS requests that the Azure Firewall has serviced, you can execute the following query.

AzureDiagnostics
| where Category == "AzureFirewallDnsProxy"
| parse kind=regex msg_s with "DNS Request: " SourceIp ":.+ IN " RequestedName " ..p.+s"
| project TimeGenerated, SourceIp, RequestedName
| order by TimeGenerated desc

Azure Container Registry

  1. In the Azure Portal, navigate to your Azure Container Registry resource.
  2. Click on Logs.

View image imports

To view all the image imports that have happened against your container registry, you can execute the following query.

ContainerRegistryRepositoryEvents | where OperationName == "importImage" | order by TimeGenerated desc

View image pulls

To view all the pulls that have happened against your container registry, you can execute the following query.

ContainerRegistryRepositoryEvents
| where OperationName == "Pull"
| order by TimeGenerated desc

Unless you performed some additional steps beyond this walkthrough's instructions, you'll notice that the value for all of the records share the same Identity -- that is the Object ID of the AKS cluster's managed identity, specifically kubelet (<your-cluster-name>-agentpool).

Microsoft Defender for Containers

If you execute the following query, you'll see all denied requests for access to your container registry. Your container registry is private, and most (if not all) of this traffic is originating from Microsoft Defender for containers -- which does not support private container registries.

ContainerRegistryLoginEvents
| where ResultDescription == "403"
| order by TimeGenerated desc

Image quarantine violation alert

If you executed the optional extra import command on the image quarantine step page, which bypasses the quarantine, you'll be able to see that alert.

  1. Open the Azure Monitor Alerts Summary page in the Azure Portal.
  2. Find the alert and click on View query results which will show you the offending images' details.

AKS Cluster

Monitoring your cluster is critical, especially when you're running a production cluster. Azure Monitor is configured to surface cluster logs, here you can see those logs as they are generated. Azure Monitor for containers is configured on this cluster for this purpose.

  1. In the Azure Portal, navigate to your AKS cluster resource.
  2. Select Logs.
  3. You can click Queries to see and execute some common log queries.

Kubernetes API Server access logs

This reference implementation logs all AKS control plane interactions in the associated Log Analytics workspace. Specifically this is enabled through the use of kube-audit-admin Diagnostics setting that was enabled on the cluster.

AzureDiagnostics
| where Category == 'kube-audit-admin'
| order by TimeGenerated desc

This returns all Kubernetes API Server interaction happening in your cluster, other than most GET requests. Basically any interaction that might potentially have the capability to modifying the system. Even an "idle" cluster can fill this log very quickly (don't be surprised to see over 200 messages in a 30 minute window). Most regulations do not require it, but if you disable kube-audit-admin and instead simply enable kube-audit the system will also log all of the GET (read) requests as well. This will dramatically increase the number of logs, but you will then see 100% of the requests to the Kubernetes API Server. Never enable both at the same time.

For example, if you wanted to see all interactions you had while going through this walkthrough, you can execute the following, replacing with the user you used while performing the bootstrapping.

AzureDiagnostics
| where Category == 'kube-audit-admin'
| where log_s contains '"username":"[email protected]"'

Workload logs

The example workload uses the standard dotnet logger interface, which are captured in ContainerLogs in Azure Monitor. You could also include additional logging and telemetry frameworks in your workload, such as Application Insights. Execute the following query to view application logs.

ContainerLogV2
| where ContainerName contains "microservice"
| where PodNamespace in ("a00095-i", "a0005-o")
| order by TimeGenerated desc

Azure Policy logs

Azure Policy definitions sync with your cluster about once every 15 minutes. To see when they sync you can execute the following query.

ContainerLogV2
| where ContainerName has "azure-policy"
| where PodNamespace has "kube-system"
| where LogMessage contains "Syncing policies with cluster"
| order by TimeGenerated desc

And audit results will be sent to Azure Policy about once every 30 minutes. To see when they sync you can execute the following query.

ContainerLogV2
| where ContainerName has "azure-policy"
| where PodNamespace has "kube-system"
| where LogMessage contains "Sending audit result"
| order by TimeGenerated desc

Syslog

Once Syslog events are collected from the AKS baseline cluster, it is possible to execute a Kusto query and see them.

// Last 100 Syslog. 
Syslog
| top 100 by TimeGenerated desc

Azure Application Gateway

Azure Application Gateway will log key information such as requests, routing, backend health, and even your WAF rule blocks.

  1. In the Azure Portal, navigate to your Azure Application Gateway resource.
  2. Select Logs.

Access logs

All traffic that the gateway services can be viewed via the following query. This includes source and destination information.

AzureDiagnostics
| where Category == "ApplicationGatewayAccessLog"
| order by TimeGenerated desc

View Web Application Firewall (WAF) logs

Blocked requests (along with other gateway data) will be visible in the attached Log Analytics workspace. Execute the following query to show WAF logs, for example. If you executed the intentionally malicious request on the previous page, you should already see logs in here.

AzureDiagnostics
| where Category == "ApplicationGatewayFirewallLog"
| order by TimeGenerated desc

Azure Key Vault

Azure Key Vault will log all operations with every secret and certificate in the vault.

  1. In the Azure Portal, navigate to your Azure key vault resource.
  2. Select Logs.

View all requests for secrets

Both your cluster and Application Gateway will be pulling secrets from your key vault. To see that traffic, you can execute the following query.

AzureDiagnostics
| where OperationName == "SecretGet"
| order by TimeGenerated desc

You should see requests for sslcert & appgw-ingress-internal-aks-ingress-contoso-com-tls from Application Gateway and ingress-internal-aks-ingress-contoso-com-tls from your AKS cluster.

Other common operations you might be interested in are SecretResourcePut, Authentication, and CertificateImport.

Microsoft Defender for Cloud

Regulatory compliance

If your subscription has the Azure Security Benchmark Azure Policy initiative applied, and Industry & regulatory standards enabled (such as PCI DSS 3.2.1), the Regulatory compliance dashboard will allow you to see compliance status for controls that have been mapped by Azure to the specific standard. The view is updated about once every 24 hours, this includes its initial scan. So if you enabled this as part of the walkthrough (steps found on the Subscription page), you may not yet see any content in here.

  1. Open the Regulatory compliance dashboard in Microsoft Defender for Cloud.
  2. Review the Industry summary and any recommendations.

Review Security Alerts

Microsoft Defender for Containers reviews your cluster's logs and detects runtime workload behavior that might be undesired. Those alerts surface in the Azure Resource Graph (securityresources -> microsoft.security/locations/alerts) and also in Microsoft Defender for Cloud in the Azure Portal. Within about 24 hours of your cluster being up and running, you may start to see some Low and Medium alerts show up.

  1. Open the Security Alerts view in Microsoft Defender for Cloud.
  2. View the alerts, and optionally add a Filter for Affected resource being your newly created cluster.

Microsoft Sentinel

Data from Log Analytic on the networking and cluster infrastructure are being delivered to Microsoft Sentinel.

  1. Open the Microsoft Sentinel hub on the Azure Portal.
  2. Select a Log Analytics workspace (networking or cluster)

From here you can view Alerts, Incidents, start a Hunting session, view/add related Workbooks (such as the Azure Kubernetes Service (AKS) Security workbook), and so on.

Traffic Analytics

NSG Flow Logs were configured as part of the default install of this reference implementation. Along with the raw logs being stored in the storage account in the hub resource group, you can access Azure Traffic Analytics for a dashboard of network flows.

  1. Open the Azure Traffic Analytics hub on the Azure Portal.
  2. Adjust filters as needed at the top.
  3. You can then see Application Ports, NSG Hits, App Gateway, and other network flows.

Next step

▶️ Clean Up Azure Resources