Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gbsyncd] Add asic db prefix for channel RESTARTQUERY #1302

Merged
merged 3 commits into from
Nov 9, 2023

Conversation

jimmyzhai
Copy link
Contributor

@jimmyzhai jimmyzhai commented Oct 9, 2023

Fix the issue sonic-net/sonic-buildimage#16608, together with sonic-net/sonic-buildimage#16812.

Need separated channel for graeceful shutdown of syncd and gbsyncd.

ADO: 26284737

@kcudnik
Copy link
Collaborator

kcudnik commented Oct 9, 2023

aspell test check failed, please fix

@jimmyzhai
Copy link
Contributor Author

aspell test check failed, please fix

Fixed

* it on any level, including database numbers."
*/
#define SYNCD_NOTIFICATION_CHANNEL_RESTARTQUERY_PER_DB(dbName) \
((dbName) == "ASIC_DB" ? \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of hardcoding asic_db here you could use it define it should be in redisconfig.h

@@ -9,6 +9,18 @@

#define SYNCD_NOTIFICATION_CHANNEL_RESTARTQUERY "RESTARTQUERY"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also remove this since from this point we always want to have specified dB name and probably we can always join dB name to restart query, no need for special case

@jimmyzhai jimmyzhai merged commit 7acd028 into sonic-net:master Nov 9, 2023
13 checks passed
@jaganbal-a
Copy link

Unit test from multi ASIC line card.

command : systemctl stop gbsyncd@0
root@sonic:/home/cisco# grep -i "stopped|requested" /var/log/syslog
Nov 16 18:14:09.990051 sonic NOTICE gbsyncd0#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:14:09.990635 sonic INFO gbsyncd.sh[2675289]: requested COLD shutdown
Nov 16 18:14:12.878760 sonic NOTICE root: Stopped gbsyncd0 service...
Nov 16 18:14:12.892066 sonic INFO systemd[1]: Stopped gbsyncd service.
Nov 16 18:14:13.381114 sonic INFO swss0#supervisord 2023-11-16 18:14:13,380 INFO stopped: tunnelmgrd (terminated by SIGTERM)
Nov 16 18:14:13.381704 sonic INFO swss0#supervisord 2023-11-16 18:14:13,381 INFO stopped: fdbsyncd (terminated by SIGTERM)
Nov 16 18:14:13.382275 sonic INFO swss0#supervisord 2023-11-16 18:14:13,382 INFO stopped: vxlanmgrd (terminated by SIGTERM)
Nov 16 18:14:13.382890 sonic INFO swss0#supervisord 2023-11-16 18:14:13,382 INFO stopped: nbrmgrd (terminated by SIGTERM)
Nov 16 18:14:13.383604 sonic INFO swss0#supervisord 2023-11-16 18:14:13,383 INFO stopped: vrfmgrd (terminated by SIGTERM)
Nov 16 18:14:14.385226 sonic INFO swss0#supervisord 2023-11-16 18:14:14,384 INFO stopped: buffermgrd (terminated by SIGTERM)
Nov 16 18:14:14.385885 sonic INFO swss0#supervisord 2023-11-16 18:14:14,385 INFO stopped: portmgrd (terminated by SIGTERM)
Nov 16 18:14:14.386607 sonic INFO swss0#supervisord 2023-11-16 18:14:14,386 INFO stopped: intfmgrd (terminated by SIGTERM)
Nov 16 18:14:14.387278 sonic INFO swss0#supervisord 2023-11-16 18:14:14,387 INFO stopped: vlanmgrd (terminated by SIGTERM)
Nov 16 18:14:14.388043 sonic INFO swss0#supervisord 2023-11-16 18:14:14,387 INFO stopped: neighsyncd (terminated by SIGTERM)
Nov 16 18:14:15.390046 sonic INFO swss0#supervisord 2023-11-16 18:14:15,389 INFO stopped: coppmgrd (terminated by SIGTERM)
Nov 16 18:14:16.394492 sonic INFO swss0#supervisord 2023-11-16 18:14:16,394 INFO stopped: orchagent (terminated by SIGTERM)
Nov 16 18:14:16.395228 sonic INFO swss0#supervisord 2023-11-16 18:14:16,394 INFO stopped: portsyncd (terminated by SIGTERM)
Nov 16 18:14:17.524969 sonic NOTICE root: Stopped swss0 service...
Nov 16 18:14:19.526115 sonic INFO teamd0#supervisord 2023-11-16 18:14:19,525 INFO stopped: tlm_teamd (exit status 0)
Nov 16 18:14:21.521382 sonic INFO teamd0#supervisord 2023-11-16 18:14:21,520 INFO stopped: teamsyncd (exit status 0)
Nov 16 18:14:23.457731 sonic INFO teamd0#supervisord 2023-11-16 18:14:23,457 INFO stopped: teammgrd (exit status 0)
Nov 16 18:14:24.591383 sonic NOTICE cisco: Stopped teamd0 service...
Nov 16 18:14:24.594717 sonic INFO systemd[1]: Stopped TEAMD container.
Nov 16 18:14:26.741991 sonic NOTICE cisco: Stopped radv service...
Nov 16 18:14:26.745415 sonic INFO systemd[1]: Stopped Router advertiser container.
Nov 16 18:14:28.128199 sonic INFO bgp0#supervisord 2023-11-16 18:14:28,127 INFO stopped: staticroutebfd (exit status 0)
Nov 16 18:14:28.128806 sonic INFO bgp0#supervisord 2023-11-16 18:14:28,128 INFO stopped: fpmsyncd (terminated by SIGTERM)
Nov 16 18:14:28.130290 sonic INFO bgp0#supervisord 2023-11-16 18:14:28,130 INFO stopped: bgpmon (terminated by SIGTERM)
Nov 16 18:14:37.171706 sonic NOTICE cisco: Stopped bgp0 service...
Nov 16 18:14:37.226587 sonic INFO systemd[1]: Stopped BGP container.
Nov 16 18:14:39.449608 sonic NOTICE root: Stopped dhcp_relay service...
Nov 16 18:14:39.452873 sonic INFO systemd[1]: Stopped dhcp_relay container.
Nov 16 18:14:39.854369 sonic NOTICE syncd0#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:14:39.854767 sonic INFO syncd.sh[2676965]: requested COLD shutdown
Nov 16 18:14:50.286946 sonic NOTICE root: Stopped syncd0 service...
Nov 16 18:14:50.296978 sonic INFO systemd[1]: Stopped syncd service.
Nov 16 18:14:50.434829 sonic INFO gbsyncd.sh[2677898]: requested COLD shutdown
Nov 16 18:14:52.831375 sonic NOTICE root: Stopped gbsyncd0 service...
Nov 16 18:14:52.841611 sonic INFO systemd[1]: Stopped gbsyncd service.
Nov 16 18:15:30.368511 sonic INFO snmp#supervisord 2023-11-16 18:15:30,368 INFO stopped: snmp-subagent (exit status 0)
Nov 16 18:15:30.372981 sonic INFO snmp#supervisord 2023-11-16 18:15:30,372 INFO stopped: snmpd (exit status 0)
Nov 16 18:15:31.097424 sonic NOTICE root: Stopped snmp service...
Nov 16 18:15:31.101081 sonic INFO systemd[1]: Stopped SNMP container.
Nov 16 18:15:31.101812 sonic INFO systemd[1]: Stopped Delays snmp container until SONiC has started.
Nov 16 18:15:31.101896 sonic INFO systemd[1]: Stopped switch state service.
Nov 16 18:15:37.782461 sonic INFO snmp#supervisord 2023-11-16 18:15:30,975 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:15:37.782461 sonic INFO snmp#supervisord 2023-11-16 18:15:30,977 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:15:39.872582 sonic INFO radv#supervisord 2023-11-16 18:14:26,622 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:15:39.872582 sonic INFO radv#supervisord 2023-11-16 18:14:26,624 INFO stopped: supervisor-proc-exit-script (terminated by SIGTERM)
Nov 16 18:15:41.843451 sonic INFO dhcp_relay#supervisord 2023-11-16 18:14:39,326 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:15:41.843451 sonic INFO dhcp_relay#supervisord 2023-11-16 18:14:39,328 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:16:30.842123 sonic INFO syncd0#supervisord 2023-11-16 18:14:49,162 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:16:30.842123 sonic INFO syncd0#supervisord 2023-11-16 18:14:50,165 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:16:31.745489 sonic INFO swss0#supervisord 2023-11-16 18:14:17,410 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:16:31.745489 sonic INFO swss0#supervisord 2023-11-16 18:14:17,412 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:16:34.550183 sonic INFO gbsyncd0#supervisord 2023-11-16 18:14:12,758 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:16:34.550183 sonic INFO gbsyncd0#supervisord 2023-11-16 18:14:12,760 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:16:34.550245 sonic INFO gbsyncd0#supervisord 2023-11-16 18:14:52,714 INFO stopped: rsyslogd (terminated by SIGTERM)
Nov 16 18:16:34.550252 sonic INFO gbsyncd0#supervisord 2023-11-16 18:14:52,716 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:16:38.471457 sonic INFO teamd0#supervisord 2023-11-16 18:14:24,474 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:16:38.471457 sonic INFO teamd0#supervisord 2023-11-16 18:14:24,476 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)

@jaganbal-a
Copy link

Config reload log snippet:
root@sonic:/home/cisco# grep -i "stopped|requested|removed" sys11 | grep "syncd|swss"
Nov 16 18:23:33.068629 sonic NOTICE gbsyncd1#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:33.071400 sonic NOTICE gbsyncd1#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy6, switch_oid:21000000000006 removed#012
Nov 16 18:23:33.075356 sonic NOTICE gbsyncd1#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy7, switch_oid:21000000000007 removed#012
Nov 16 18:23:33.079202 sonic NOTICE gbsyncd1#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy8, switch_oid:21000000000008 removed#012
Nov 16 18:23:33.083549 sonic NOTICE gbsyncd1#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy9, switch_oid:21000000000009 removed#012
Nov 16 18:23:33.088407 sonic NOTICE gbsyncd1#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phyA, switch_oid:2100000000000a removed#012
Nov 16 18:23:33.092349 sonic NOTICE gbsyncd1#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phyB, switch_oid:2100000000000b removed#012
Nov 16 18:23:33.107135 sonic INFO gbsyncd.sh[2691110]: requested COLD shutdown
Nov 16 18:23:33.210998 sonic NOTICE gbsyncd0#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:33.230706 sonic INFO gbsyncd.sh[2691111]: requested COLD shutdown
Nov 16 18:23:33.238483 sonic NOTICE gbsyncd0#GBSAI[13]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy0, switch_oid:21000000000000 removed#012
Nov 16 18:23:33.246543 sonic NOTICE gbsyncd0#GBSAI[13]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy1, switch_oid:21000000000001 removed#012
Nov 16 18:23:33.262595 sonic NOTICE gbsyncd0#GBSAI[13]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy2, switch_oid:21000000000002 removed#012
Nov 16 18:23:33.277568 sonic NOTICE gbsyncd0#GBSAI[13]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy3, switch_oid:21000000000003 removed#012
Nov 16 18:23:33.279676 sonic NOTICE gbsyncd0#GBSAI[13]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy4, switch_oid:21000000000004 removed#012
Nov 16 18:23:33.281086 sonic NOTICE gbsyncd0#GBSAI[13]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy5, switch_oid:21000000000005 removed#012
Nov 16 18:23:33.451797 sonic NOTICE gbsyncd2#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:33.464723 sonic NOTICE gbsyncd2#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phyC, switch_oid:2100000000000c removed#012
Nov 16 18:23:33.469500 sonic NOTICE gbsyncd2#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phyD, switch_oid:2100000000000d removed#012
Nov 16 18:23:33.470332 sonic INFO gbsyncd.sh[2691150]: requested COLD shutdown
Nov 16 18:23:33.471909 sonic NOTICE gbsyncd2#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phyE, switch_oid:2100000000000e removed#012
Nov 16 18:23:33.473635 sonic NOTICE gbsyncd2#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phyF, switch_oid:2100000000000f removed#012
Nov 16 18:23:33.474782 sonic NOTICE gbsyncd2#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy10, switch_oid:21000000000010 removed#012
Nov 16 18:23:33.475602 sonic NOTICE gbsyncd2#GBSAI[12]: GBSAI: phy_remove_switch:359: GBCRUD-RES:REMOVE-PHY: phy11, switch_oid:21000000000011 removed#012
Nov 16 18:23:35.455193 sonic INFO bgp2#supervisord 2023-11-16 18:23:35,454 INFO stopped: fpmsyncd (terminated by SIGTERM)
Nov 16 18:23:35.754553 sonic INFO bgp1#supervisord 2023-11-16 18:23:35,754 INFO stopped: fpmsyncd (terminated by SIGTERM)
Nov 16 18:23:36.114765 sonic NOTICE root: Stopped gbsyncd0 service...
Nov 16 18:23:36.128945 sonic INFO systemd[1]: Stopped gbsyncd service.
Nov 16 18:23:36.153058 sonic INFO bgp0#supervisord 2023-11-16 18:23:36,152 INFO stopped: fpmsyncd (terminated by SIGTERM)
Nov 16 18:23:36.863733 sonic NOTICE root: Stopped gbsyncd2 service...
Nov 16 18:23:36.876223 sonic INFO systemd[1]: Stopped gbsyncd service.
Nov 16 18:23:37.331724 sonic INFO swss0#supervisord 2023-11-16 18:23:37,331 INFO stopped: tunnelmgrd (terminated by SIGTERM)
Nov 16 18:23:37.332282 sonic INFO swss0#supervisord 2023-11-16 18:23:37,332 INFO stopped: fdbsyncd (terminated by SIGTERM)
Nov 16 18:23:37.332868 sonic INFO swss0#supervisord 2023-11-16 18:23:37,332 INFO stopped: vxlanmgrd (terminated by SIGTERM)
Nov 16 18:23:37.333394 sonic INFO swss0#supervisord 2023-11-16 18:23:37,333 INFO stopped: nbrmgrd (terminated by SIGTERM)
Nov 16 18:23:37.333960 sonic INFO swss0#supervisord 2023-11-16 18:23:37,333 INFO stopped: vrfmgrd (terminated by SIGTERM)
Nov 16 18:23:37.360383 sonic INFO teamd2#supervisord 2023-11-16 18:23:37,360 INFO stopped: teamsyncd (exit status 0)
Nov 16 18:23:37.545712 sonic NOTICE root: Stopped gbsyncd1 service...
Nov 16 18:23:37.556007 sonic INFO systemd[1]: Stopped gbsyncd service.
Nov 16 18:23:37.578662 sonic INFO teamd1#supervisord 2023-11-16 18:23:37,578 INFO stopped: teamsyncd (exit status 0)
Nov 16 18:23:37.582360 sonic INFO swss2#supervisord 2023-11-16 18:23:37,582 INFO stopped: tunnelmgrd (terminated by SIGTERM)
Nov 16 18:23:37.582897 sonic INFO swss2#supervisord 2023-11-16 18:23:37,582 INFO stopped: fdbsyncd (terminated by SIGTERM)
Nov 16 18:23:37.583750 sonic INFO swss2#supervisord 2023-11-16 18:23:37,583 INFO stopped: vxlanmgrd (terminated by SIGTERM)
Nov 16 18:23:37.584299 sonic INFO swss2#supervisord 2023-11-16 18:23:37,584 INFO stopped: nbrmgrd (terminated by SIGTERM)
Nov 16 18:23:37.584965 sonic INFO swss2#supervisord 2023-11-16 18:23:37,584 INFO stopped: vrfmgrd (terminated by SIGTERM)
Nov 16 18:23:37.851397 sonic INFO teamd0#supervisord 2023-11-16 18:23:37,851 INFO stopped: teamsyncd (exit status 0)
Nov 16 18:23:38.336313 sonic INFO swss0#supervisord 2023-11-16 18:23:38,335 INFO stopped: buffermgrd (terminated by SIGTERM)
Nov 16 18:23:38.337322 sonic INFO swss0#supervisord 2023-11-16 18:23:38,336 INFO stopped: portmgrd (terminated by SIGTERM)
Nov 16 18:23:38.338248 sonic INFO swss0#supervisord 2023-11-16 18:23:38,337 INFO stopped: intfmgrd (terminated by SIGTERM)
Nov 16 18:23:38.338955 sonic INFO swss0#supervisord 2023-11-16 18:23:38,338 INFO stopped: vlanmgrd (terminated by SIGTERM)
Nov 16 18:23:38.339719 sonic INFO swss0#supervisord 2023-11-16 18:23:38,339 INFO stopped: neighsyncd (terminated by SIGTERM)
Nov 16 18:23:38.586983 sonic INFO swss2#supervisord 2023-11-16 18:23:38,586 INFO stopped: buffermgrd (terminated by SIGTERM)
Nov 16 18:23:38.587461 sonic INFO swss2#supervisord 2023-11-16 18:23:38,587 INFO stopped: portmgrd (terminated by SIGTERM)
Nov 16 18:23:38.588072 sonic INFO swss2#supervisord 2023-11-16 18:23:38,587 INFO stopped: intfmgrd (terminated by SIGTERM)
Nov 16 18:23:38.588582 sonic INFO swss2#supervisord 2023-11-16 18:23:38,588 INFO stopped: vlanmgrd (terminated by SIGTERM)
Nov 16 18:23:38.589163 sonic INFO swss2#supervisord 2023-11-16 18:23:38,589 INFO stopped: neighsyncd (terminated by SIGTERM)
Nov 16 18:23:38.840111 sonic INFO swss1#supervisord 2023-11-16 18:23:38,839 INFO stopped: tunnelmgrd (terminated by SIGTERM)
Nov 16 18:23:38.840746 sonic INFO swss1#supervisord 2023-11-16 18:23:38,840 INFO stopped: fdbsyncd (terminated by SIGTERM)
Nov 16 18:23:38.841383 sonic INFO swss1#supervisord 2023-11-16 18:23:38,841 INFO stopped: vxlanmgrd (terminated by SIGTERM)
Nov 16 18:23:38.842045 sonic INFO swss1#supervisord 2023-11-16 18:23:38,841 INFO stopped: nbrmgrd (terminated by SIGTERM)
Nov 16 18:23:38.842661 sonic INFO swss1#supervisord 2023-11-16 18:23:38,842 INFO stopped: vrfmgrd (terminated by SIGTERM)
Nov 16 18:23:39.341727 sonic INFO swss0#supervisord 2023-11-16 18:23:39,341 INFO stopped: coppmgrd (terminated by SIGTERM)
Nov 16 18:23:39.590871 sonic INFO swss2#supervisord 2023-11-16 18:23:39,590 INFO stopped: coppmgrd (terminated by SIGTERM)
Nov 16 18:23:39.844514 sonic INFO swss1#supervisord 2023-11-16 18:23:39,844 INFO stopped: buffermgrd (terminated by SIGTERM)
Nov 16 18:23:39.845046 sonic INFO swss1#supervisord 2023-11-16 18:23:39,844 INFO stopped: portmgrd (terminated by SIGTERM)
Nov 16 18:23:39.845729 sonic INFO swss1#supervisord 2023-11-16 18:23:39,845 INFO stopped: intfmgrd (terminated by SIGTERM)
Nov 16 18:23:39.846359 sonic INFO swss1#supervisord 2023-11-16 18:23:39,846 INFO stopped: vlanmgrd (terminated by SIGTERM)
Nov 16 18:23:39.846937 sonic INFO swss1#supervisord 2023-11-16 18:23:39,846 INFO stopped: neighsyncd (terminated by SIGTERM)
Nov 16 18:23:40.347982 sonic INFO swss0#supervisord 2023-11-16 18:23:40,347 INFO stopped: orchagent (terminated by SIGTERM)
Nov 16 18:23:40.348611 sonic INFO swss0#supervisord 2023-11-16 18:23:40,348 INFO stopped: portsyncd (terminated by SIGTERM)
Nov 16 18:23:40.595019 sonic INFO swss2#supervisord 2023-11-16 18:23:40,594 INFO stopped: orchagent (terminated by SIGTERM)
Nov 16 18:23:40.595789 sonic INFO swss2#supervisord 2023-11-16 18:23:40,595 INFO stopped: portsyncd (terminated by SIGTERM)
Nov 16 18:23:40.848680 sonic INFO swss1#supervisord 2023-11-16 18:23:40,848 INFO stopped: coppmgrd (terminated by SIGTERM)
Nov 16 18:23:41.446085 sonic NOTICE root: Stopped swss0 service...
Nov 16 18:23:41.699397 sonic NOTICE root: Stopped swss2 service...
Nov 16 18:23:41.851622 sonic INFO swss1#supervisord 2023-11-16 18:23:41,851 INFO stopped: orchagent (terminated by SIGTERM)
Nov 16 18:23:41.852184 sonic INFO swss1#supervisord 2023-11-16 18:23:41,851 INFO stopped: portsyncd (terminated by SIGTERM)
Nov 16 18:23:42.963728 sonic NOTICE root: Stopped swss1 service...
Nov 16 18:23:46.419271 sonic NOTICE syncd2#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:46.419930 sonic INFO syncd.sh[2692882]: requested COLD shutdown
Nov 16 18:23:46.886495 sonic NOTICE syncd0#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:46.886956 sonic INFO syncd.sh[2693109]: requested COLD shutdown
Nov 16 18:23:47.082983 sonic NOTICE syncd1#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:47.083459 sonic INFO syncd.sh[2693186]: requested COLD shutdown
Nov 16 18:23:56.397667 sonic NOTICE root: Stopped syncd1 service...
Nov 16 18:23:56.407317 sonic INFO systemd[1]: Stopped syncd service.
Nov 16 18:23:57.089366 sonic NOTICE root: Stopped syncd2 service...
Nov 16 18:23:57.099113 sonic INFO systemd[1]: Stopped syncd service.
Nov 16 18:23:57.446899 sonic NOTICE root: Stopped syncd0 service...
Nov 16 18:23:57.466814 sonic INFO systemd[1]: Stopped syncd service.
Nov 16 18:24:59.563142 sonic INFO swss0#supervisord 2023-11-16 18:23:41,354 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:24:59.563142 sonic INFO swss0#supervisord 2023-11-16 18:23:41,356 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:25:00.427732 sonic INFO swss1#supervisord 2023-11-16 18:23:42,870 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:25:00.427732 sonic INFO swss1#supervisord 2023-11-16 18:23:42,872 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:25:04.356564 sonic INFO syncd2#supervisord 2023-11-16 18:23:55,994 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:25:04.356564 sonic INFO syncd2#supervisord 2023-11-16 18:23:56,997 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:25:04.379524 sonic INFO swss2#supervisord 2023-11-16 18:23:41,602 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:25:04.379524 sonic INFO swss2#supervisord 2023-11-16 18:23:41,604 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:25:05.180749 sonic INFO syncd0#supervisord 2023-11-16 18:23:56,350 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:25:05.180749 sonic INFO syncd0#supervisord 2023-11-16 18:23:57,354 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:25:05.849683 sonic INFO syncd1#supervisord 2023-11-16 18:23:55,282 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:25:05.849683 sonic INFO syncd1#supervisord 2023-11-16 18:23:56,285 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:25:21.148370 sonic INFO gbsyncd1#supervisord 2023-11-16 18:23:37,426 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:25:21.148370 sonic INFO gbsyncd1#supervisord 2023-11-16 18:23:37,428 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:25:21.776221 sonic INFO gbsyncd2#supervisord 2023-11-16 18:23:36,747 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:25:21.776221 sonic INFO gbsyncd2#supervisord 2023-11-16 18:23:36,748 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 16 18:25:21.912902 sonic INFO gbsyncd0#supervisord 2023-11-16 18:23:35,959 INFO stopped: rsyslogd (exit status 0)
Nov 16 18:25:21.912902 sonic INFO gbsyncd0#supervisord 2023-11-16 18:23:35,961 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)

@kcudnik
Copy link
Collaborator

kcudnik commented Nov 17, 2023

what about those logs ? syncd is not behaving in the expected way after this change ? @jimmyzhai seems like syncd is getting SIGTERM instead of gracefully exits, like the notification is not received byt syncd

@kcudnik
Copy link
Collaborator

kcudnik commented Nov 17, 2023

@jaganbal-a since you are having gbsyncd1 here you probably need to specify -g and -x parameters in syncd_request_shutdown command, please take a read of wiki https://github.com/sonic-net/sonic-sairedis/wiki/Context-configuration, and you will probably need to call shutdown twice for each global config 0 and 1 if you have 2 syncd processes running

currently i don't know how shutdown scripts in syncd are prepared, but i think automatically supporting shutdown both syncd processes is not yet supported in scripts

@jaganbal-a
Copy link

jaganbal-a commented Nov 17, 2023

what about those logs ? syncd is not behaving in the expected way after this change ? @jimmyzhai seems like syncd is getting SIGTERM instead of gracefully exits, like the notification is not received byt syncd

@kcudnik , I see the config reload/gbsyncd docker restart/shutdown works as expected.
In the below logs of config reload, the syncd receives the graceful request which removes the switch on all 3 namespace.

Nov 16 18:23:41.446085 sonic NOTICE root: Stopped swss0 service...
Nov 16 18:23:41.699397 sonic NOTICE root: Stopped swss2 service...
Nov 16 18:23:41.851622 sonic INFO swss1#supervisord 2023-11-16 18:23:41,851 INFO stopped: orchagent (terminated by SIGTERM)
Nov 16 18:23:41.852184 sonic INFO swss1#supervisord 2023-11-16 18:23:41,851 INFO stopped: portsyncd (terminated by SIGTERM)
Nov 16 18:23:42.963728 sonic NOTICE root: Stopped swss1 service...
Nov 16 18:23:46.419271 sonic NOTICE syncd2#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:46.419930 sonic INFO syncd.sh[2692882]: requested COLD shutdown
Nov 16 18:23:46.421103 sonic NOTICE syncd2#syncd: :- removeAllSwitches: Removing all switches
Nov 16 18:23:46.886495 sonic NOTICE syncd0#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:46.886956 sonic INFO syncd.sh[2693109]: requested COLD shutdown
Nov 16 18:23:46.888292 sonic NOTICE syncd0#syncd: :- removeAllSwitches: Removing all switches
Nov 16 18:23:47.082983 sonic NOTICE syncd1#syncd_request_shutdown: :- send: requested COLD shutdown
Nov 16 18:23:47.083459 sonic INFO syncd.sh[2693186]: requested COLD shutdown
Nov 16 18:23:47.084965 sonic NOTICE syncd1#syncd: :- removeAllSwitches: Removing all switches
Nov 16 18:23:50.570633 sonic NOTICE syncd1#syncd: :- removeAllSwitches: removing switch RID oid:0x2100000000000000 took 3.484878 sec
Nov 16 18:23:50.773215 sonic NOTICE syncd2#syncd: :- removeAllSwitches: removing switch RID oid:0x2100000000000000 took 4.351490 sec
Nov 16 18:23:51.492529 sonic NOTICE syncd0#syncd: :- removeAllSwitches: removing switch RID oid:0x2100000000000000 took 4.603371 sec
Nov 16 18:23:56.397667 sonic NOTICE root: Stopped syncd1 service...
Nov 16 18:23:56.407317 sonic INFO systemd[1]: Stopped syncd service.
Nov 16 18:23:57.089366 sonic NOTICE root: Stopped syncd2 service...
Nov 16 18:23:57.099113 sonic INFO systemd[1]: Stopped syncd service.
Nov 16 18:23:57.446899 sonic NOTICE root: Stopped syncd0 service...
Nov 16 18:23:57.466814 sonic INFO systemd[1]: Stopped syncd service.

@kcudnik
Copy link
Collaborator

kcudnik commented Nov 17, 2023

Is that not expected behavior?

@StormLiangMS
Copy link

@jimmyzhai could you update ADO in the descritption?

@abdosi
Copy link
Contributor

abdosi commented Dec 29, 2023

@jimmyzhai could you update ADO in the descritption?

@StormLiangMS : Updated

mssonicbld pushed a commit to mssonicbld/sonic-sairedis that referenced this pull request Jan 4, 2024
* Use different channel RESTARTQUERY for syncd/gbsyncd

* change restartquery to restartQuery due to aspell error
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202305: #1337

mssonicbld pushed a commit that referenced this pull request Jan 5, 2024
* Use different channel RESTARTQUERY for syncd/gbsyncd

* change restartquery to restartQuery due to aspell error
@yxieca
Copy link
Contributor

yxieca commented Mar 21, 2024

@jimmyzhai can you help create 202311 cherry-pick PR?

@yxieca
Copy link
Contributor

yxieca commented Mar 22, 2024

Change is already in 202311 at creation time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants