Merge branch 'master' into counter-optimization-all-in-one

README.md

@@ -21,55 +21,97 @@ The SWitch State Service (SWSS) is a collection of software that provides a data
 
 ## Getting Started
 
-### Install
+### Prerequisites
+
+Install the following dependencies:
+```
+sudo apt install redis-server
+sudo apt install libhiredis0.14
+sudo apt install libzmq5 libzmq3-dev
+sudo apt install libboost-serialization1.74.0
+sudo apt install libboost1.71-dev
+sudo apt install libasan6
+```
+**Note:** If your are using Ubuntu 18.04, install `libhiredis0.13` instead.
+
+Visit the [official sonic-buildimage Azure pipeline for the VS platform](https://dev.azure.com/mssonic/build/_build?definitionId=142&view=branches) and choose the branch that matches the sonic-swss branch you are trying to build or install. Then select the latest successful build.
+From the Summary tab, access build artifacts.
+<img width="1048" alt="image" src="https://github.com/user-attachments/assets/faa6f08d-788b-4801-8439-3f31a52efaa1">
+Download the folder `sonic-buildimage.vs/target/debs/{your host machine's Debian code name}`. You can check the Debian code name of your machine by running `cat /etc/debian_version`.
+<img width="1022" alt="image" src="https://github.com/user-attachments/assets/1ad750eb-252c-4913-b14f-91b5533a1295">
+Extract the downloaded zip file using `unzip sonic-buildimage.vs.zip`. Then navigate to `sonic-buildimage.vs/target/debs/{Debian code name}/` and install the following Debian packages:
+```
+sudo dpkg -i libdashapi_1.0.0_amd64.deb libnl-3-200_3.5.0-1_amd64.deb libnl-3-dev_3.5.0-1_amd64.deb libnl-cli-3-200_3.5.0-1_amd64.deb libnl-cli-3-dev_3.5.0-1_amd64.deb libnl-genl-3-200_3.5.0-1_amd64.deb libnl-genl-3-dev_3.5.0-1_amd64.deb libnl-nf-3-200_3.5.0-1_amd64.deb libnl-nf-3-dev_3.5.0-1_amd64.deb libnl-route-3-200_3.5.0-1_amd64.deb libnl-route-3-dev_3.5.0-1_amd64.deb libprotobuf32_3.21.12-3_amd64.deb libsaimetadata_1.0.0_amd64.deb  libsaimetadata-dev_1.0.0_amd64.deb libsairedis_1.0.0_amd64.deb libsairedis-dev_1.0.0_amd64.deb libsaivs_1.0.0_amd64.deb libsaivs-dev_1.0.0_amd64.deb  libswsscommon_1.0.0_amd64.deb libswsscommon-dev_1.0.0_amd64.deb libteam5_1.31-1_amd64.deb libteamdctl0_1.31-1_amd64.deb libyang_1.0.73_amd64.deb libyang-dev_1.0.73_amd64.deb python3-swsscommon_1.0.0_amd64.deb
+```
+**Note:** You can also [build these packages yourself (for the VS platform)](https://github.com/sonic-net/sonic-buildimage/blob/master/README.md).
+
+Now, you can either directly install the SONiC SWSS package or you can build it from source and then install it. To install the SONiC SWSS package that is already in `sonic-buildimage.vs/target/debs/{Debian code name}/`, simply run the following command:
+```
+sudo dpkg -i swss_1.0.0_amd64.deb
+```
 
-Before installing, add key and package sources:
+#### Install from Source
 
-    sudo apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893
-    echo 'deb http://apt-mo.trafficmanager.net/repos/sonic/ trusty main' | sudo tee -a /etc/apt/sources.list.d/sonic.list
-    sudo apt-get update
+Install build dependencies:
+```
+sudo apt install libtool
+sudo apt install autoconf automake
+sudo apt install dh-exec
+sudo apt install nlohmann-json3-dev
+sudo apt install libgmock-dev
+```
 
-Install dependencies:
+Clone the `sonic-swss` repository on your host machine: `git clone https://github.com/sonic-net/sonic-swss.git`.
 
-    sudo apt-get install redis-server -t trusty
-    sudo apt-get install libhiredis0.13 -t trusty
-    sudo apt-get install libzmq5 libzmq3-dev
-
-Install building dependencies:
+Make sure that SAI header files exist in `/usr/include/sai`. Since you have already installed `libsairedis-dev`, `libsaimetadata-dev`, and `libsaivs-dev`, this should already be the case. If you have compiled `libsairedis` yourself, make sure that the SAI header files are copied to `/usr/include/sai`.
 
-    sudo apt-get install libtool
-    sudo apt-get install autoconf automake
-    sudo apt-get install dh-exec
+You can compile and install from source using:
+```
+./autogen.sh
+./configure
+make && sudo make install
+```
+**Note:** This will NOT run the mock tests located under `tests/mock_tests`.
 
-There are a few different ways you can install SONiC-SWSS.
+You can also build a debian package using:
+```
+./autogen.sh
+fakeroot debian/rules binary
+```
+## Common issues
 
-#### Install from Debian Repo
+#### Cannot find `libboost-serialization1.74.0`
 
-For your convenience, you can install prepared packages on Debian Jessie:
+Unfortunately, `libboost-serialization1.74.0` is not officially supported on Ubuntu 20.04 (focal) even though it is supported on Debian 11 (bullseye). Therefore, you must build this package from source. You can use a script similar to [this one](https://github.com/ulikoehler/deb-buildscripts/blob/master/deb-boost.sh), but you only need to create a package for the Boost serialization library. You should also make sure that the generated package is named `libboost-serialization1.74.0`. After the package is created, you can install it by running `sudo dpkg -i libboost-serialization1.74.0_1.74.0_amd64.deb`.
 
-    sudo apt-get install swss
+#### Dependency issue when installing `libzmq3-dev`
 
-#### Install from Source
+If you cannot install `libzmq3-dev` because of dependency issues, please check the version of `libkrb5` packages installed on your host machine:
+```
+    sudo dpkg -l | grep "libkrb5"
+```
+If the version is not `1.17-6ubuntu4.7`, then you need to install the correct version:
 
-Checkout the source: `git clone https://github.com/sonic-net/sonic-swss.git` and install it yourself.
+    sudo apt install libkrb5support0=1.17-6ubuntu4.7
+    sudo apt install libzmq3-dev
 
-Get SAI header files into /usr/include/sai. Put the SAI header files that you use to compile
-libsairedis into /usr/include/sai
+**Warning:** This may remove many packages that are already installed on your system. Please take note of what is being removed.
 
-Install prerequisite packages:
+**Note:** Do NOT install `*krb5*` packages that are located in the `sonic-buildimage.vs` folder that you downloaded. These packages have a higher version and will cause dependency issues.
 
-    sudo apt-get install libswsscommon libswsscommon-dev libsairedis libsairedis-dev
+#### Dependency issues when installing some package
 
-You can compile and install from source using:
+If you run into dependency issues during the installation of a package, you can run `sudo apt -f install` to fix the issue. But note that if `apt` is unable to fix the dependency problem, it will attempt to remove the broken package(s).
 
-    ./autogen.sh
-    ./configure
-    make && sudo make install
+#### Too many open files
 
-You can also build a debian package using:
+If you get a C++ exception with the description "Too many open files" during the mock tests, you should check the maximum number of open files that are permitted on your system:
+```
+ulimit -a | grep "open files"
+```
+You can increase it by executing this command: `ulimit -n 8192`. Feel free to change `8192`. This value worked fine for me.
 
-    ./autogen.sh
-    fakeroot debian/rules binary
+**Note:** This change is only valid for the current terminal session. If you want a persistent change, append `ulimit -n 8192` to `~/.bashrc`.
 
 ## Need Help?
 

cfgmgr/vlanmgr.cpp

@@ -79,7 +79,8 @@ VlanMgr::VlanMgr(DBConnector *cfgDb, DBConnector *appDb, DBConnector *stateDb, c
     //               /sbin/bridge vlan del vid 1 dev Bridge self;
     //               /sbin/ip link del dummy 2>/dev/null;
     //               /sbin/ip link add dummy type dummy &&
-    //               /sbin/ip link set dummy master Bridge"
+    //               /sbin/ip link set dummy master Bridge &&
+    //               /sbin/ip link set dummy up"
 
     const std::string cmds = std::string("")
       + BASH_CMD + " -c \""
@@ -90,7 +91,8 @@ VlanMgr::VlanMgr(DBConnector *cfgDb, DBConnector *appDb, DBConnector *stateDb, c
       + BRIDGE_CMD + " vlan del vid " + DEFAULT_VLAN_ID + " dev " + DOT1Q_BRIDGE_NAME + " self; "
       + IP_CMD + " link del dev dummy 2>/dev/null; "
       + IP_CMD + " link add dummy type dummy && "
-      + IP_CMD + " link set dummy master " + DOT1Q_BRIDGE_NAME + "\"";
+      + IP_CMD + " link set dummy master " + DOT1Q_BRIDGE_NAME + " && "
+      + IP_CMD + " link set dummy up" + "\"";
 
     std::string res;
     EXEC_WITH_ERROR_THROW(cmds, res);

fpmsyncd/fpmsyncd.cpp

@@ -9,13 +9,34 @@
 #include "subscriberstatetable.h"
 #include "warmRestartHelper.h"
 #include "fpmsyncd/fpmlink.h"
+#include "fpmsyncd/fpmsyncd.h"
 #include "fpmsyncd/routesync.h"
 
 #include <netlink/route/route.h>
 
 using namespace std;
 using namespace swss;
 
+// gSelectTimeout specifies the maximum wait time in milliseconds (-1 == infinite)
+static int gSelectTimeout;
+#define INFINITE -1
+#define FLUSH_TIMEOUT 500  // 500 milliseconds
+static int gFlushTimeout = FLUSH_TIMEOUT;
+// consider the traffic is small if pipeline contains < 500 entries
+#define SMALL_TRAFFIC 500
+
+/**
+ * @brief fpmsyncd invokes redispipeline's flush with a timer
+ * 
+ * redispipeline would automatically flush itself when full,
+ * but fpmsyncd can invoke pipeline's flush even if it's not full yet.
+ * 
+ * By setting gSelectTimeout, fpmsyncd controls the flush interval.
+ * 
+ * @param pipeline reference to the pipeline to be flushed
+ */
+void flushPipeline(RedisPipeline& pipeline);
+
 /*
  * Default warm-restart timer interval for routing-stack app. To be used only if
  * no explicit value has been defined in configuration.
@@ -61,7 +82,7 @@ int main(int argc, char **argv)
     DBConnector applStateDb("APPL_STATE_DB", 0);
     std::unique_ptr<NotificationConsumer> routeResponseChannel;
 
-    RedisPipeline pipeline(&db);
+    RedisPipeline pipeline(&db, ROUTE_SYNC_PPL_SIZE);
     RouteSync sync(&pipeline);
 
     DBConnector stateDb("STATE_DB", 0);
@@ -152,12 +173,14 @@ int main(int argc, char **argv)
                 sync.m_warmStartHelper.setState(WarmStart::WSDISABLED);
             }
 
+            gSelectTimeout = INFINITE;
+
             while (true)
             {
                 Selectable *temps;
 
                 /* Reading FPM messages forever (and calling "readMe" to read them) */
-                s.select(&temps);
+                s.select(&temps, gSelectTimeout);
 
                 /*
                  * Upon expiration of the warm-restart timer or eoiu Hold Timer, proceed to run the
@@ -286,8 +309,7 @@ int main(int argc, char **argv)
                 }
                 else if (!warmStartEnabled || sync.m_warmStartHelper.isReconciled())
                 {
-                    pipeline.flush();
-                    SWSS_LOG_DEBUG("Pipeline flushed");
+                    flushPipeline(pipeline);
                 }
             }
         }
@@ -304,3 +326,35 @@ int main(int argc, char **argv)
 
     return 1;
 }
+
+void flushPipeline(RedisPipeline& pipeline) {
+
+    size_t remaining = pipeline.size();
+
+    if (remaining == 0) {
+        gSelectTimeout = INFINITE;
+        return;
+    }
+
+    int idle = pipeline.getIdleTime();
+
+    // flush the pipeline if
+    // 1. traffic is not scaled (only prevent fpmsyncd from flushing ppl too frequently in the scaled case)
+    // 2. the idle time since last flush has exceeded gFlushTimeout
+    // 3. idle <= 0, due to system clock drift, should not happen since we already use steady_clock for timing
+    if (remaining < SMALL_TRAFFIC || idle >= gFlushTimeout || idle <= 0) {
+
+        pipeline.flush();
+
+        gSelectTimeout = INFINITE;
+
+        SWSS_LOG_DEBUG("Pipeline flushed");
+    }
+    else
+    {
+        // skip flushing ppl and set the timeout of fpmsyncd select function to be (gFlushTimeout - idle)
+        // so that fpmsyncd select function would block at most for (gFlushTimeout - idle)
+        // by doing this, we make sure every entry eventually gets flushed
+        gSelectTimeout = gFlushTimeout - idle;
+    }
+}

fpmsyncd/fpmsyncd.h

@@ -0,0 +1,8 @@
+#ifndef __FPMSYNCD__
+#define __FPMSYNCD__
+
+
+// redispipeline has a maximum capacity of 50000 entries
+#define ROUTE_SYNC_PPL_SIZE 50000
+
+#endif

mclagsyncd/mclaglink.cpp

@@ -193,7 +193,7 @@ void MclagLink::setPortIsolate(char *msg)
         BRCM_PLATFORM_SUBSTRING,
         BFN_PLATFORM_SUBSTRING,
         CTC_PLATFORM_SUBSTRING,
-        MRVL_PLATFORM_SUBSTRING
+        MRVL_PRST_PLATFORM_SUBSTRING
     };
 
     const char *platform = getenv("platform");

mclagsyncd/mclaglink.h

@@ -54,7 +54,7 @@
 #define BRCM_PLATFORM_SUBSTRING "broadcom"
 #define BFN_PLATFORM_SUBSTRING  "barefoot"
 #define CTC_PLATFORM_SUBSTRING  "centec"
-#define MRVL_PLATFORM_SUBSTRING "marvell"
+#define MRVL_PRST_PLATFORM_SUBSTRING "marvell-prestera"
 
 using namespace std;
 

orchagent/Makefile.am

@@ -20,7 +20,7 @@ dist_swss_DATA = \
 		 pfc_detect_marvell_teralynx.lua  \
 		 pfc_detect_mellanox.lua  \
 		 pfc_detect_broadcom.lua \
-		 pfc_detect_marvell.lua \
+		 pfc_detect_marvell_prestera.lua \
 		 pfc_detect_barefoot.lua \
 		 pfc_detect_nephos.lua \
 		 pfc_detect_cisco-8000.lua \

orchagent/aclorch.cpp

@@ -3306,7 +3306,7 @@ void AclOrch::init(vector<TableConnector>& connectors, PortsOrch *portOrch, Mirr
             platform == CISCO_8000_PLATFORM_SUBSTRING ||
             platform == MLNX_PLATFORM_SUBSTRING ||
             platform == BFN_PLATFORM_SUBSTRING  ||
-            platform == MRVL_PLATFORM_SUBSTRING ||
+            platform == MRVL_PRST_PLATFORM_SUBSTRING ||
             platform == MRVL_TL_PLATFORM_SUBSTRING ||
             platform == NPS_PLATFORM_SUBSTRING ||
             platform == XS_PLATFORM_SUBSTRING ||
@@ -3327,7 +3327,7 @@ void AclOrch::init(vector<TableConnector>& connectors, PortsOrch *portOrch, Mirr
         };
     }
 
-    if ( platform == MRVL_PLATFORM_SUBSTRING ||
+    if ( platform == MRVL_PRST_PLATFORM_SUBSTRING ||
 	    platform == MRVL_TL_PLATFORM_SUBSTRING ||
             platform == VS_PLATFORM_SUBSTRING)
     {
@@ -3362,7 +3362,7 @@ void AclOrch::init(vector<TableConnector>& connectors, PortsOrch *portOrch, Mirr
     // In Broadcom DNX platform also, V4 and V6 rules are stored in different tables
     if (platform == MLNX_PLATFORM_SUBSTRING ||
         platform == CISCO_8000_PLATFORM_SUBSTRING ||
-        platform == MRVL_PLATFORM_SUBSTRING ||
+        platform == MRVL_PRST_PLATFORM_SUBSTRING ||
         platform == XS_PLATFORM_SUBSTRING ||
         (platform == BRCM_PLATFORM_SUBSTRING && sub_platform == BRCM_DNX_PLATFORM_SUBSTRING))
     {
@@ -5938,7 +5938,7 @@ bool MetaDataMgr::isValidMetaData(uint16_t metadata)
 
 uint16_t MetaDataMgr::getFreeMetaData(uint8_t dscp)
 {
-    uint16_t metadata = metaMax + 1;
+    uint16_t metadata = (uint16_t)(metaMax + 1);
     SWSS_LOG_INFO("Metadata Request for dscp %d", dscp);
 
     if (initComplete)
@@ -5962,7 +5962,7 @@ uint16_t MetaDataMgr::getFreeMetaData(uint8_t dscp)
             m_dscpMetadata[dscp] = metadata;
             SWSS_LOG_INFO("New Metadata %d allocated for dscp %d", metadata, dscp);
         }
-        m_MetadataRef[metadata] += 1;
+        m_MetadataRef[metadata] = (uint16_t)(m_MetadataRef[metadata] + 1);
     }
     else
     {
@@ -5975,7 +5975,7 @@ void MetaDataMgr::recycleMetaData(uint16_t metadata)
 {
     if (initComplete)
     {
-        m_MetadataRef[metadata] -= 1;
+        m_MetadataRef[metadata] = (uint16_t)(m_MetadataRef[metadata] - 1);
         SWSS_LOG_INFO("Freeing Metadata %d refcount %d", metadata, m_MetadataRef[metadata]);
         if (m_MetadataRef[metadata] == 0)
         {