MAcsec SAI_MACSEC_SA_STAT_IN_PKTS for the Ingress SC/SA not present in counters_db

[judyjoseph]

Description

MAcsec SAI_MACSEC_SA_STAT_IN_PKTS for the Ingress SC/SA not present in counters_db. The ingress SA stats is missing in the ASIC_DB/COUNTER_DB when doing a redis-monitor on those db's

The following counters are missing in counters DB

                SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
                SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
                SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            8790329
                SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
                SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0

Steps to reproduce the issue:

1. Do a show macsec after setting a macsec session with master/202405 image

Describe the results you received:

MACsec port(Ethernet240)
---------------------  ---------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                MACSEC_PROFILE
replay_window          0
send_sci               true
---------------------  ---------------
        MACsec Egress SC (xxx)
        -----------  -
        encoding_an  1
        -----------  -
                MACsec Egress SA (1)
                -------------------------------------  ----------------------------------------------------------------
                auth_key                               xxx
                next_pn                                1
                sak                                    xxx
                salt                                   xxx
                ssci                                   2
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         139
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    32776
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  137
                SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
                -------------------------------------  ----------------------------------------------------------------
        MACsec Ingress SC (xxx)
                MACsec Ingress SA (1)
                -----------------------------------  ----------------------------------------------------------------
                active                               true
                auth_key                             xxx
                lowest_acceptable_pn                 1
                sak                                  xxx
                salt                                 xxx
                ssci                                 1
                SAI_MACSEC_SA_ATTR_CURRENT_XPN       2122
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED  521250
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED  0
                -----------------------------------  ----------------------------------------------------------------

Describe the results you expected:

MACsec port(Ethernet120)
---------------------  ------------------
cipher_suite           GCM-AES-XPN-256
enable                 true
enable_encrypt         true
enable_protect         true
enable_replay_protect  false
profile                macsec-profile-two
replay_window          0
send_sci               true
---------------------  ------------------
        MACsec Egress SC (xxxx)
        -----------  -
        encoding_an  0
        -----------  -
                MACsec Egress SA (0)
                -------------------------------------  ----------------------------------------------------------------
                auth_key                               xxx
                next_pn                                1
                sak                                    xxx
                salt                                   xxx
                ssci                                   1
                SAI_MACSEC_SA_ATTR_CURRENT_XPN         70488823
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED    778860948858289
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED    0
                SAI_MACSEC_SA_STAT_OUT_PKTS_ENCRYPTED  70129287
                SAI_MACSEC_SA_STAT_OUT_PKTS_PROTECTED  0
                -------------------------------------  ----------------------------------------------------------------
        MACsec Ingress SC (xxxx)
                MACsec Ingress SA (0)
                ---------------------------------------  ----------------------------------------------------------------
                active                                   true
                auth_key                                 xxx
                lowest_acceptable_pn                     1
                sak                                      xxxx
                salt                                     xxx
                ssci                                     2
                SAI_MACSEC_SA_ATTR_CURRENT_XPN           8892076
                **SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
                SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
                SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
                SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
                SAI_MACSEC_SA_STAT_IN_PKTS_OK            8790329
                SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
                SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0**
                SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      117076375044777
                SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0
                ---------------------------------------  ----------------------------------------------------------------

Output of show version:

SONiC Software Version: SONiC.20240510.18

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[judyjoseph]

This is currently seen with Broadcom DNX platforms, Triaging further if this is a brcm SAI specific issue or sonic flex counter issue

[judyjoseph]

@anamehra @abdosi do you see this issue with packet chassis ?

[anamehra]

Hi @judyjoseph ,
we see the counts on Cisco chassis with 202405 image:

		MACsec Ingress SA (1)
		---------------------------------------  ----------------------------------------------------------------
		active                                   true
		auth_key                                 xxx
		lowest_acceptable_pn                     1
		sak                                      xxx
		salt                                     xxx
		ssci                                     1
		SAI_MACSEC_SA_ATTR_CURRENT_XPN           161771393
		SAI_MACSEC_SA_STAT_IN_PKTS_DELAYED       0
		SAI_MACSEC_SA_STAT_IN_PKTS_INVALID       0
		SAI_MACSEC_SA_STAT_IN_PKTS_LATE          0
		SAI_MACSEC_SA_STAT_IN_PKTS_NOT_USING_SA  0
		SAI_MACSEC_SA_STAT_IN_PKTS_NOT_VALID     0
		SAI_MACSEC_SA_STAT_IN_PKTS_OK            152337668
		SAI_MACSEC_SA_STAT_IN_PKTS_UNCHECKED     0
		SAI_MACSEC_SA_STAT_IN_PKTS_UNUSED_SA     0
		SAI_MACSEC_SA_STAT_OCTETS_ENCRYPTED      53919084748
		SAI_MACSEC_SA_STAT_OCTETS_PROTECTED      0
		---------------------------------------  ----------------------------------------------------------------```

[judyjoseph]

@stepanblyschak @Junchao-Mellanox, in this PR sonic-net/sonic-swss#3076 -- there is a change in macsecorch.cpp as below

 <<<<           MACsecSaAttrStatManager(ctx).setCounterIdList(obj_id, counter_type, counter_stats);
 >>>>           MACsecSaAttrStatManager(ctx).setCounterIdList(obj_id, counter_type, counter_stats, ***ctx.get_switch_id());**

Trying to understand if this is a generic change or is it applicable only to GB based devices? Here we have an issue for a non-GB device -- where from 202405/master onwards a few of the macsec counters are missing.

Share some steps of debugging counters as well, I check the redis-monitor logs for ASIC/COUNTERSDB, I see those counters we are looking for is missing.

[Junchao-Mellanox]

Hi @judyjoseph , here is just my thought:

1. Check flex counter DB (DB 5) to make sure all macsec ingress counters are there (keys MAC). If some are missing there, the issue is on orchagent side. Put a breakpoint at https://github.com/sonic-net/sonic-swss/blob/master/orchagent/macsecorch.cpp#L2150 to see what happened.
2. Put a breakpoint here https://github.com/sonic-net/sonic-sairedis/blob/9fe90f6bf9290138e776d8b19582c01d81849d84/syncd/FlexCounter.cpp#L1003 to make sure all macsec related attributes are added for query
3. Put a breakpoint here https://github.com/sonic-net/sonic-sairedis/blob/9fe90f6bf9290138e776d8b19582c01d81849d84/syncd/FlexCounter.cpp#L1025 to make sure all macsec related attributes are really queried