Support adding additional attr[0-5] for endpoint level metrics. (#12753)

docs/en/api/metrics-query-expression.md

@@ -240,7 +240,7 @@ top_n(<metric_name>, <top_number>, <order>, <attrs>)
    The attrs filter also supports not-equal filter `!=`, the format is `attr0 != 'value'`.
 
 **Notice**: 
-- The `attrs` only support Service metrics for now and should be added in the metrics first, see [Metrics Additional Attributes](../concepts-and-designs/metrics-additional-attributes.md).
+- The `attrs` should be added in the metrics first, see [Metrics Additional Attributes](../concepts-and-designs/metrics-additional-attributes.md).
 - When use not-equal filter, for example `attr1 != 'value'`, if the storage is using `MySQL` or other JDBC storage and `attr1 value is NULL` in the metrics, 
 the result of `attr1 != 'value'` will always `false` and would NOT include this metric in the result due to SQL can't compare `NULL` with the `value`.
 

docs/en/changes/changes.md

@@ -11,7 +11,7 @@
 * Change the endpoint_traffic to updatable for the additional column `last_ping`.
 * Add Component ID(5023) for the GoZero framework.
 * Support Kong monitoring.
-* Support adding additional attr[0-4] for service level metrics.
+* Support adding additional attr[0-5] for service/endpoint level metrics.
 * Support async-profiler feature for performance analysis.
 * Add metrics value owner for metrics topN query result.
 * Add naming control for `EndpointDependencyBuilder`.
@@ -28,6 +28,9 @@
 * Add service global topN widget on `General-Root`, `Mesh-Root`, `K8S-Root` dashboard.
 * Fix initialization dashboards.
 * Update the Kubernetes metrics for reduce multiple metrics calculate in MQE.
+* Support view data value related dashboards in TopList widgets.
+* Add endpoint global topN widget on `General-Root`, `Mesh-Root`.
+
 
 #### Documentation
 * Update release document to adopt newly added revision-based process.

docs/en/concepts-and-designs/metrics-additional-attributes.md

@@ -18,7 +18,10 @@ and the Java Class must follow the following rules:
 - The Class must implement the [ISourceDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/ISourceDecorator.java) interface.
 - The Class package must be under the `org.apache.skywalking.*`.
 
-SkyWalking provides a default implementation [ServiceDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ServiceDecorator.java) which set the `attr0` to the service `Layer`.
+### Default Decorator
+SkyWalking provides some default implementation of decorator:
+
+- [ServiceDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ServiceDecorator.java) which set the `attr0` to the service `Layer`.
 The following OAL metrics had been decorated by the `ServiceDecorator` by default:
 ```text
 // Service scope metrics
@@ -28,6 +31,23 @@ service_cpm = from(Service.*).cpm().decorator("ServiceDecorator");
 service_apdex = from(Service.latency).apdex(name, status).decorator("ServiceDecorator");
 ```
 
+- [EndpointDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/EndpointDecorator.java) which set the `attr0` to the endpoint `Layer`.
+The following OAL metrics had been decorated by the `EndpointDecorator` by default:
+```text
+endpoint_cpm = from(Endpoint.*).cpm().decorator("EndpointDecorator");
+endpoint_resp_time = from(Endpoint.latency).longAvg().decorator("EndpointDecorator");
+endpoint_sla = from(Endpoint.*).percent(status == true).decorator("EndpointDecorator");
+```
+
+- [K8SServiceDecorator](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/K8SServiceDecorator.java) which set the `attr0` to the k8s service `Layer`.
+The following OAL metrics had been decorated by the `K8SServiceDecorator` by default:
+```text
+kubernetes_service_http_call_cpm = from(K8SService.*).filter(detectPoint == DetectPoint.SERVER).filter(type == "protocol").filter(protocol.type == "http").cpm().decorator("K8SServiceDecorator");
+kubernetes_service_http_call_time = from(K8SService.protocol.http.latency).filter(detectPoint == DetectPoint.SERVER).filter(type == "protocol").filter(protocol.type == "http").longAvg().decorator("K8SServiceDecorator");
+kubernetes_service_http_call_successful_rate = from(K8SService.*).filter(detectPoint == DetectPoint.SERVER).filter(type == "protocol").filter(protocol.type == "http").percent(protocol.success == true).decorator("K8SServiceDecorator");
+kubernetes_service_apdex = from(K8SService.protocol.http.latency).filter(detectPoint == DetectPoint.SERVER).filter(type == "protocol").filter(protocol.type == "http").apdex(name, protocol.success).decorator("K8SServiceDecorator");
+```    
+
 ## MAL Source Decorate
 In the MAL script, you can use the [decorate](mal.md#decorate-function) function to decorate the source, and must follow the following rules:
 - The decorate function must after service() function.

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java

@@ -125,8 +125,10 @@ public void enterDecorateSource(OALParser.DecorateSourceContext ctx) {
         current.setSourceDecorator(decoratorNameTrim);
         Map<String, ISourceDecorator<ISource>> map = SourceDecoratorManager.DECORATOR_MAP;
         int currentScopeId = current.getFrom().getSourceScopeId();
-        if (currentScopeId != DefaultScopeDefine.SERVICE && currentScopeId != DefaultScopeDefine.K8S_SERVICE) {
-            throw new IllegalArgumentException("OAL metric: " + current.getMetricsName() + ", decorate source only support service scope.");
+        if (currentScopeId != DefaultScopeDefine.SERVICE
+            && currentScopeId != DefaultScopeDefine.K8S_SERVICE
+            && currentScopeId != DefaultScopeDefine.ENDPOINT) {
+            throw new IllegalArgumentException("OAL metric: " + current.getMetricsName() + ", decorate source not support scope: " + DefaultScopeDefine.nameOf(currentScopeId));
         }
         ISourceDecorator<ISource> decorator = map.get(decoratorNameTrim);
         if (decorator == null) {

oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/DeepAnalysisTest.java

@@ -89,7 +89,7 @@ public void testEndpointAnalysis() {
         Assertions.assertEquals("(long)(1)", method.getArgsExpressions().get(1));
 
         List<SourceColumn> source = result.getFieldsFromSource();
-        Assertions.assertEquals(2, source.size());
+        Assertions.assertEquals(8, source.size());
 
         List<DataColumn> persistentFields = result.getPersistentFields();
         Assertions.assertEquals(4, persistentFields.size());
@@ -117,7 +117,7 @@ public void testFilterAnalysis() {
         Assertions.assertEquals("(long)(1)", method.getArgsExpressions().get(1));
 
         List<SourceColumn> source = result.getFieldsFromSource();
-        Assertions.assertEquals(2, source.size());
+        Assertions.assertEquals(8, source.size());
 
         List<DataColumn> persistentFields = result.getPersistentFields();
         Assertions.assertEquals(4, persistentFields.size());

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/Endpoint.java

@@ -23,7 +23,9 @@
 import lombok.Getter;
 import lombok.Setter;
 import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.ISourceDecorator;
 import org.apache.skywalking.oap.server.core.analysis.Layer;
+import org.apache.skywalking.oap.server.core.analysis.SourceDecoratorManager;
 import org.apache.skywalking.oap.server.library.util.StringUtil;
 import org.apache.skywalking.oap.server.core.analysis.IDManager;
 
@@ -89,6 +91,39 @@ public String getEntityId() {
     @Getter
     @Setter
     private Layer serviceLayer;
+    @Getter
+    @Setter
+    @ScopeDefaultColumn.DefinedByField(columnName = "attr0", isAttribute = true)
+    private String attr0;
+    @Getter
+    @Setter
+    @ScopeDefaultColumn.DefinedByField(columnName = "attr1", isAttribute = true)
+    private String attr1;
+    @Getter
+    @Setter
+    @ScopeDefaultColumn.DefinedByField(columnName = "attr2", isAttribute = true)
+    private String attr2;
+    @Getter
+    @Setter
+    @ScopeDefaultColumn.DefinedByField(columnName = "attr3", isAttribute = true)
+    private String attr3;
+    @Getter
+    @Setter
+    @ScopeDefaultColumn.DefinedByField(columnName = "attr4", isAttribute = true)
+    private String attr4;
+    @Getter
+    @Setter
+    @ScopeDefaultColumn.DefinedByField(columnName = "attr5", isAttribute = true)
+    private String attr5;
+
+    /**
+     * Get the decorator through given name and invoke.
+     * @param decorator The decorator class simpleName.
+     */
+    public void decorate(String decorator) {
+        ISourceDecorator<ISource> sourceDecorator = SourceDecoratorManager.DECORATOR_MAP.get(decorator);
+        sourceDecorator.decorate(this);
+    }
 
     @Override
     public void prepare() {

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/EndpointDecorator.java

@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.source;
+
+import org.apache.skywalking.oap.server.core.analysis.ISourceDecorator;
+
+public class EndpointDecorator implements ISourceDecorator<Endpoint> {
+    @Override
+    public int getSourceScope() {
+        return DefaultScopeDefine.ENDPOINT;
+    }
+
+    /**
+     * Set the Layer name to attr0
+     * @param source The source instance to be decorated
+     */
+    @Override
+    public void decorate(final Endpoint source) {
+        source.setAttr0(source.getServiceLayer().name());
+    }
+}

oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/AsyncQueryUtils.java

@@ -22,13 +22,15 @@
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.Executor;
 import java.util.concurrent.ForkJoinPool;
+import lombok.extern.slf4j.Slf4j;
 
 import static java.util.concurrent.ForkJoinPool.defaultForkJoinWorkerThreadFactory;
 
 /**
  * The utility class for async GraphQL query.
  * All the async GraphQL query should be wrapped by this class and shared the same executor.
  */
+@Slf4j
 public class AsyncQueryUtils {
     private static final Executor EXECUTOR = new ForkJoinPool(
         Runtime.getRuntime().availableProcessors(), defaultForkJoinWorkerThreadFactory, null, true);
@@ -37,7 +39,8 @@ public static <U> CompletableFuture<U> queryAsync(Callable<U> caller) {
         return CompletableFuture.supplyAsync(() -> {
             try {
                 return caller.call();
-            } catch (Exception e) {
+            } catch (Throwable e) {
+                log.error(e.getMessage(), e);
                 throw new RuntimeException(e);
             }
         }, EXECUTOR);

oap-server/server-starter/src/main/resources/oal/core.oal

@@ -51,9 +51,9 @@ service_instance_resp_time = from(ServiceInstance.latency).longAvg();
 service_instance_cpm = from(ServiceInstance.*).cpm();
 
 // Endpoint scope metrics
-endpoint_cpm = from(Endpoint.*).cpm();
-endpoint_resp_time = from(Endpoint.latency).longAvg();
-endpoint_sla = from(Endpoint.*).percent(status == true);
+endpoint_cpm = from(Endpoint.*).cpm().decorator("EndpointDecorator");
+endpoint_resp_time = from(Endpoint.latency).longAvg().decorator("EndpointDecorator");
+endpoint_sla = from(Endpoint.*).percent(status == true).decorator("EndpointDecorator");
 endpoint_percentile = from(Endpoint.latency).percentile2(10); // Multiple values including p50, p75, p90, p95, p99
 endpoint_mq_consume_latency = from((str->long)Endpoint.tag["transmission.latency"]).filter(type == RequestType.MQ).filter(tag["transmission.latency"] != null).longAvg();
 

oap-server/server-starter/src/main/resources/ui-initialized-templates/general/general-root.json

@@ -5,7 +5,7 @@
       "children": [
         {
           "x": 0,
-          "y": 13,
+          "y": 24,
           "w": 24,
           "h": 52,
           "i": "1",
@@ -211,7 +211,8 @@
           },
           "widget": {
             "title": "Service Apdex"
-          }
+          },
+          "valueRelatedDashboard": "General-Service"
         },
         {
           "x": 12,
@@ -229,7 +230,8 @@
           },
           "widget": {
             "title": "Service Avg Response Time (ms)"
-          }
+          },
+          "valueRelatedDashboard": "General-Service"
         },
         {
           "x": 6,
@@ -242,12 +244,13 @@
             "top_n(service_sla,10,asc,attr0='GENERAL')/100"
           ],
           "widget": {
-            "title": "Success Rate"
+            "title": "Service Success Rate"
           },
           "graph": {
             "type": "TopList",
             "color": "purple"
-          }
+          },
+          "valueRelatedDashboard": "General-Service"
         },
         {
           "x": 18,
@@ -266,7 +269,66 @@
           "graph": {
             "type": "TopList",
             "color": "purple"
-          }
+          },
+          "valueRelatedDashboard": "General-Service"
+        },
+        {
+          "x": 0,
+          "y": 13,
+          "w": 8,
+          "h": 11,
+          "i": "105",
+          "type": "Widget",
+          "expressions": [
+            "top_n(endpoint_sla,10,asc,attr0='GENERAL')/100"
+          ],
+          "graph": {
+            "type": "TopList",
+            "color": "purple"
+          },
+          "widget": {
+            "title": "Endpoint Success Rate"
+          },
+          "valueRelatedDashboard": "General-Endpoint"
+        },
+        {
+          "x": 8,
+          "y": 13,
+          "w": 8,
+          "h": 11,
+          "i": "106",
+          "type": "Widget",
+          "expressions": [
+            "top_n(endpoint_resp_time,10,des,attr0='GENERAL')"
+          ],
+          "graph": {
+            "type": "TopList",
+            "color": "purple"
+          },
+          "widget": {
+            "title": "Endpoint Avg Response Time (ms)"
+          },
+          "valueRelatedDashboard": "General-Endpoint"
+        },
+        {
+          "x": 16,
+          "y": 13,
+          "w": 8,
+          "h": 11,
+          "i": "107",
+          "type": "Widget",
+          "expressions": [
+            "top_n(endpoint_cpm,10,des,attr0='GENERAL')"
+          ],
+          "graph": {
+            "type": "TopList",
+            "color": "purple"
+          },
+          "widget": {
+            "title": "Endpoint Load (calls / min)",
+            "tips": "For HTTP 1/2, gRPC, RPC services, this means Calls Per Minute (calls / min), for TCP services"
+          },
+          "valueRelatedDashboard": "General-Endpoint"
         }
       ],
       "layer": "GENERAL",