Merge pull request #58 from SoftwareAG/develop

Develop

README.md

@@ -247,13 +247,13 @@ Mappings are persisted as Managed Objects and can be easily changed, deleted or
 In addition to using plain properties of the source payload, you can apply functions on the payload properties. This covers a scenario where a device name should be a combination of a generic name and an external device Id.
 Complex mapping expressions are supported by using [JSONata](https://jsonata.org). \
 In this case the following function could be used:
-```$join([device_name, _DEVICE_IDENT_])```. 
+```$join([device_name, id])```. 
 
 Further example for JSONata expressions are:
 * to convert a UNIX timestamp to ISO date format use:
       <code>$fromMillis($number(deviceTimestamp))</code>
 * to join substring starting at position 5 of property <code>txt</code> with device
-      identifier use: <code>$join([$substring(txt,5), "-", _DEVICE_IDENT_])</code>
+      identifier use: <code>$join([$substring(txt,5), "-", id])</code>
 
 >**_NOTE:_**
 > * escape properties with special characters with <code>`</code>. The property
@@ -398,52 +398,59 @@ Connected devices send their data using an external device identifier, e.g. IMEI
 #### Define templates and substitutions for source and target payload
 
 In the second wizard step, shown on the screenshot below the mapping is further defined:
-1. Editing the source template directly or use a snooped template by pressing button ```<-```, arrow left
-2. Editing the target template directly or use a sample template by pressing button ```->```, arrow right
-3. Adding substitutions
+1. Editing the source template directly
+2. Editing the target template directly
 
 
 <p align="center">
 <img src="resources/image/Generic_MQTT_MappingTemplate.png"  style="width: 70%;" />
 </p>
 <br/>
 
-In order to define a substitution ( substitute values in the target payload with values extracted at runtime from the source payload), the UI offers the following features:
-1. Add mapping (button with "+" sign)
-2. Show & Select already defined substitutions (button with skip symbol). A selected substitution is colored and can be deleted by pressing the button with "-" sign
-3. Delete mapping (button with one "-" sign), the selected substitution is deleted
-4. Delete all mappings (button with two "--" signs). In this case the substitution to define the deviceIdentifier is automatically added again. This is the case when a template topic contains a wildcard, either "+"- single level or "#" - multi level
+In order to define a substitution (a substitution substitutes values in the target payload with values extracted at runtime from the source payload), the UI offers the following feaoptionstures:
+1. Add new substitution by pressing button "Add substitution". Further details for the substitution can be defined in the next modal dialog. See as well the next paragraph.
+2. Update an existing substitution, by selecting the substitution in the table of substitutions in the lower section of the wizard. Then press button  "Update substitution" 
+3. Delete an existing substitution, by pressing the button with the red minus
 
 <p align="center">
 <img src="resources/image/Generic_MQTT_MappingTemplate_annnotated.png"  style="width: 70%;" />
 </p>
 <br/>
 
 To define a new substitution the following steps have to be performed:
-1. Select a property in the source JSON payload by click on the respective property. Then the JSONpath is appears in the field with the label ```Evaluate expression on source```
-2. Select a property in the target JSON payload by click on the respective property. Then the JSONpath is appears in the field with the label ```Substitute in target```
-3. Select  ```Expand Array``` if the result of the source expression is an array and you want to generate any of the following substitutions:
-  * ```multi-device-single-value```
-  * ```multi-device-multi-value```
-  * ```single-device-multi-value```\
+1. Select a property in the source JSON payload by click on the respective property. Then the JSONpath is appears in the field with the label ```Evaluate Expression on Source```
+1. Select a property in the target JSON payload by click on the respective property. Then the JSONpath is appears in the field with the label ```Evaluate Expression on Target```
+>**_NOTE:_** Use the same <a href="https://jsonata.org" target="_blank">JSONata</a>
+              expressions as in the source template. In addition you can use <code>$</code> to merge the 
+              result of the source expression with the existing target template. Special care is 
+              required since this can overwrite mandatory Cumulocity attributes, e.g. <code>source.id</code>.  This can result in API calls that are rejected by the Cumulocity backend!
+
+3. Press the button "Add substitution". In the next modal dialog the following details can be specified:
+   1. Select option ```Expand Array``` if the result of the source expression is an array and you want to generate any of the following substitutions:
+      * ```multi-device-single-value```
+      * ```multi-device-multi-value```
+      * ```single-device-multi-value```\
   Otherwise an extracted array is treated as a single value, see [Different type of substitutions](#different-type-of-substitutions).
-4. Select a repair strategy that determines how the mapping is applied:
-  *  ```DEFAULT```: Map the extracted values to the attribute addressed on right side
-  *  ```USE_FIRST_VALUE_OF_ARRAY```: When the left side of the mapping returns an array, only use the 1. item in the array and map this to the right side
-  *  ```USE_LAST_VALUE_OF_ARRAY```: When the left side of the mapping returns an array, only use the last item in the array and map this to the right side
-  *  ```REMOVE_IF_MISSING```: When the left side of the mapping returns no result (not NULL), then delete the attribute (that is addressed in mapping) in the target on the right side. This avoids empty attribute, e.d. ```airsensor: undefined```
-  *  ```REMOVE_IF_NULL```: When the left side of the mapping returns ```null```, then delete the attribute (that is addressed in mapping) in the target on the right side. This avoids empty attribute, e.d. ```airsensor: undefined```
-5. Press the add button with the ```+``` sign, to add the substitution to the list of substitutions.
+   1. Select option ```Resolve to externalId``` if you want to resolve system Cumulocity Id to externalId using externalIdType. This can onlybe used for OUTBOUND mappings.
+   1. Select a ```Reapir Strategy``` that determines how the mapping is applied:
+      *  ```DEFAULT```: Map the extracted values to the attribute addressed on right side
+      *  ```USE_FIRST_VALUE_OF_ARRAY```: When the left side of the mapping returns an array, only use the 1. item in the array and map this to the right side
+      *  ```USE_LAST_VALUE_OF_ARRAY```: When the left side of the mapping returns an array, only use the last item in the array and map this to the right side
+      *  ```REMOVE_IF_MISSING```: When the left side of the mapping returns no result (not NULL), then delete the attribute (that is addressed in mapping) in the target on the right side. This avoids empty attribute, e.d. ```airsensor: undefined```
+      *  ```REMOVE_IF_NULL```: When the left side of the mapping returns ```null```, then delete the attribute (that is addressed in mapping) in the target on the right side. This avoids empty attribute, e.d. ```airsensor: undefined```
+<p align="center">
+<img src="resources/image/Generic_MQTT_MappingTemplate_EditModal.png"  style="width: 70%;" />
+</p>
 <br/>
 
 >**_NOTE:_** When adding a new substitution the following two consistency rules are checked:
 >1. Does another substitution for the same target property exist? If so, a modal dialog appears and asks the user for confirmation to overwrite the existing substitution.
 >2. If the new substitution defines the device identifier, it is checked if another substitution already withe the same proprty exists. If so, a modal dialog appears and asks for confirmation to overwrite the existing substitution.
 
 
-To avoid inconsistent JSON being sent to the Cumulocity API schemas are defined for all target payloads (Measurement, Event, Alarm, Inventory). The schemas validate if reqiured properties are defined and if the time is in the correct format.
+To avoid inconsistent JSON being sent to the Cumulocity API the defined target tmeplate are validated with schemas. These are defined for all target payloads (Measurement, Event, Alarm, Inventory). The schemas validate if reqiured properties are defined and if the time is in the correct format.
 
-In the sample below, e.g. a warning is shown since the required property ```c8y_IsDevice``` is  missing in the payload.
+In the sample below, e.g. a warning is shown since the required property ```source.id``` is  missing in the payload.
 
 
 <p align="center">

mqtt-mapping-service/src/main/java/mqtt/mapping/core/C8YAgent.java

@@ -463,6 +463,9 @@ public ManagedObjectRepresentation upsertDevice(ID identity, ProcessingContext<?
                     // append external id to name
                     mor.setName(mor.getName());
                     mor.set(new IsDevice());
+                    // remove id
+                    mor.setId(null);
+
                     mor = inventoryApi.create(mor, context);
                     log.info("New device created: {}", mor);
                     identityApi.create(mor, identity, context);

mqtt-mapping-service/src/main/java/mqtt/mapping/core/MappingComponent.java

@@ -109,16 +109,18 @@ public void setObjectMapper(ObjectMapper objectMapper) {
     // cache of inbound mappings stored in a tree used for resolving
     private TreeNode resolverMappingInbound = InnerNode.createRootNode();
 
-    private void initializeMappingStatus() {
-        log.info("Initializing status: {}, {} ", mappingServiceRepresentation.getMappingStatus(),
-                (mappingServiceRepresentation.getMappingStatus() == null
-                        || mappingServiceRepresentation.getMappingStatus().size() == 0 ? 0
-                                : mappingServiceRepresentation.getMappingStatus().size()));
+    public void initializeMappingStatus(boolean reset) {
 
-        if (mappingServiceRepresentation.getMappingStatus() != null) {
+        if (mappingServiceRepresentation.getMappingStatus() != null && !reset) {
+            log.info("Initializing status: {}, {} ", mappingServiceRepresentation.getMappingStatus(),
+                    (mappingServiceRepresentation.getMappingStatus() == null
+                            || mappingServiceRepresentation.getMappingStatus().size() == 0 ? 0
+                                    : mappingServiceRepresentation.getMappingStatus().size()));
             mappingServiceRepresentation.getMappingStatus().forEach(ms -> {
                 statusMapping.put(ms.ident, ms);
             });
+        } else {
+            statusMapping = new HashMap<String, MappingStatus>();
         }
         if (!statusMapping.containsKey(MappingStatus.IDENT_UNSPECIFIED_MAPPING)) {
             statusMapping.put(MappingStatus.IDENT_UNSPECIFIED_MAPPING, MappingStatus.UNSPECIFIED_MAPPING_STATUS);
@@ -128,7 +130,7 @@ private void initializeMappingStatus() {
 
     public void initializeMappingComponent(MappingServiceRepresentation mappingServiceRepresentation) {
         this.mappingServiceRepresentation = mappingServiceRepresentation;
-        initializeMappingStatus();
+        initializeMappingStatus(false);
     }
 
     public void sendStatusMapping() {
@@ -144,7 +146,8 @@ public void sendStatusMapping() {
                 updateMor.setAttrs(service);
                 this.inventoryApi.update(updateMor);
             } else {
-                log.debug("Ignoring mapping monitoring: {}, intialized: {}", statusMapping.values().size(), intialized);
+                log.debug("Ignoring mapping monitoring: {}, initialized: {}", statusMapping.values().size(),
+                        intialized);
             }
         });
     }
@@ -180,12 +183,6 @@ public List<MappingStatus> getMappingStatus() {
         return new ArrayList<MappingStatus>(statusMapping.values());
     }
 
-    public List<MappingStatus> resetMappingStatus() {
-        ArrayList<MappingStatus> msl = new ArrayList<MappingStatus>(statusMapping.values());
-        msl.forEach(ms -> ms.reset());
-        return msl;
-    }
-
     public void saveMappings(List<Mapping> mappings) {
         subscriptionsService.runForTenant(tenant, () -> {
             mappings.forEach(m -> {
@@ -243,7 +240,8 @@ public List<Mapping> getMappings() {
         return result;
     }
 
-    public Mapping updateMapping(Mapping mapping, boolean allowUpdateWhenActive) throws Exception {
+    public Mapping updateMapping(Mapping mapping, boolean allowUpdateWhenActive, boolean ignoreValidation)
+            throws Exception {
         // test id the mapping is active, we don't delete or modify active mappings
         MutableObject<Exception> exception = new MutableObject<Exception>(null);
         Mapping result = subscriptionsService.callForTenant(tenant, () -> {
@@ -255,14 +253,15 @@ public Mapping updateMapping(Mapping mapping, boolean allowUpdateWhenActive) thr
             // mapping is deactivated and we can delete it
             List<Mapping> mappings = getMappings();
             List<ValidationError> errors = MappingRepresentation.isMappingValid(mappings, mapping);
-            if (errors.size() == 0) {
+            if (errors.size() == 0 || ignoreValidation) {
                 MappingRepresentation mr = new MappingRepresentation();
                 mapping.lastUpdate = System.currentTimeMillis();
                 mr.setType(MappingRepresentation.MQTT_MAPPING_TYPE);
                 mr.setC8yMQTTMapping(mapping);
                 mr.setId(mapping.id);
                 ManagedObjectRepresentation mor = toManagedObject(mr);
                 mor.setId(GId.asGId(mapping.id));
+                mor.setName(mapping.name);
                 inventoryApi.update(mor);
                 return mapping;
             } else {
@@ -301,7 +300,7 @@ public Mapping createMapping(Mapping mapping) {
             mr.getC8yMQTTMapping().setId(mapping.id);
             mor = toManagedObject(mr);
             mor.setId(GId.asGId(mapping.id));
-
+            mor.setName(mapping.name);
             inventoryApi.update(mor);
             log.info("Created mapping: {}", mor);
             return mapping;
@@ -422,12 +421,15 @@ public void setActivationMapping(String id, Boolean active) throws Exception {
         log.info("Setting active: {} got mapping: {}", id, active);
         Mapping mapping = getMapping(id);
         mapping.setActive(active);
-        if  (Direction.INBOUND.equals(mapping.direction)) {
+        if (Direction.INBOUND.equals(mapping.direction)) {
             // step 2. retrieve collected snoopedTemplates
             mapping.setSnoopedTemplates(getCacheMappingInbound().get(id).getSnoopedTemplates());
         }
         // step 3. update mapping in inventory
-        updateMapping(mapping, true);
+        // don't validate mapping when setting active = false, this allows to remove
+        // mappings that are not working
+        boolean ignoreValidation = !active;
+        updateMapping(mapping, true, ignoreValidation);
         // step 4. delete mapping from update cache
         removeDirtyMapping(mapping);
         // step 5. update caches
@@ -446,7 +448,7 @@ public void cleanDirtyMappings() throws Exception {
         for (Mapping mapping : dirtyMappings) {
             log.info("Found mapping to be saved: {}, {}", mapping.id, mapping.snoopStatus);
             // no reload required
-            updateMapping(mapping, true);
+            updateMapping(mapping, true, false);
         }
         // reset dirtySet
         dirtyMappings = new HashSet<Mapping>();

mqtt-mapping-service/src/main/java/mqtt/mapping/model/API.java

@@ -30,7 +30,7 @@ public enum API {
     ALARM("ALARM", "source.id", "alarms"),
     EVENT("EVENT", "source.id", "events"),
     MEASUREMENT("MEASUREMENT", "source.id", "measurements"),
-    INVENTORY("INVENTORY", "_DEVICE_IDENT_", "managedObjects"),
+    INVENTORY("INVENTORY", "id", "managedObjects"),
     OPERATION("OPERATION", "deviceId", "operations"),
     EMPTY("NN", "nn", "nn"),
 

mqtt-mapping-service/src/main/java/mqtt/mapping/model/Mapping.java

@@ -41,6 +41,9 @@
 @ToString(exclude = { "source", "target", "snoopedTemplates" })
 public class Mapping implements Serializable {
 
+  public static String TOKEN_TOPIC_LEVEL = "_TOPIC_LEVEL_";
+
+  public static String TIME = "time";
   public static int SNOOP_TEMPLATES_MAX = 5;
   public static String SPLIT_TOPIC_REGEXP = "((?<=/)|(?=/))";
   public static Mapping UNSPECIFIED_MAPPING;

mqtt-mapping-service/src/main/java/mqtt/mapping/model/MappingSubstitution.java

@@ -32,6 +32,8 @@
 
 import javax.validation.constraints.NotNull;
 import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
 
 @Getter
 @ToString()
@@ -57,18 +59,26 @@ public SubstituteValue(JsonNode value, TYPE type, RepairStrategy repair) {
         }
 
         public Object typedValue() {
-            Object result;
             DocumentContext dc;
             switch (type) {
                 case OBJECT:
+                    Map <String,Object> ro = null;
+                    if (value != null && !value.isNull()) {
+                        dc = JsonPath.parse(value.toString());
+                        ro = dc.read("$");
+                    } else {
+                        ro= null;
+                    }
+                    return ro;
                 case ARRAY:
+                    List<Map <String,Object>> ra = null;
                     if (value != null && !value.isNull()) {
                         dc = JsonPath.parse(value.toString());
-                        result = dc.read("$");
+                        ra = dc.read("$");
                     } else {
-                        result= value;
+                        ra= null;
                     }
-                    return result;
+                    return ra;
                 case IGNORE:
                     return null;
                 case NUMBER: