Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don/packet io - Update to cover library #2

Open
wants to merge 199 commits into
base: master
Choose a base branch
from
Open
Changes from 2 commits
Commits
Show all changes
199 commits
Select commit Hold shift + click to select a range
1e05640
This document describes the functionality and high level desigh of br…
mohan-selvaraj Aug 16, 2019
e607a9f
Distributed forwarding in a VOQ architecture HLD
kartik-arista May 23, 2020
9819493
SONIC broadcast, unknown-unicast and unknown-multicast (BUM) storm-co…
mohan-selvaraj Jun 1, 2020
8459987
Updates after review meeting dated Jun-18 2020
kartik-arista Jun 23, 2020
ddf468a
Second set of edits from review at VOQ working group.
kartik-arista Jun 26, 2020
88a798a
Changes for Rev 2.0 of the document
kartik-arista Aug 6, 2020
90c1289
More edits after community review
kartik-arista Aug 8, 2020
fcf7e0c
Moving VOQ arch documents to voq directory instead of chassis as
kartik-arista Sep 8, 2020
2190c9a
Renaming Global DB to chassis db and misc cleanups
kartik-arista Sep 22, 2020
e716907
Complete all the renaming of global to chassis
kartik-arista Sep 28, 2020
7bc809c
MPLS HLD initial revision
qbdwlr Nov 16, 2020
3a7b397
updated with MPLS bulking support
qbdwlr Jan 11, 2021
0324010
updates for MPLS community review comments
qbdwlr Jan 14, 2021
1ff614e
updates from MPLS subgroup feedback
qbdwlr Jan 19, 2021
df80078
updates from github comments
qbdwlr Feb 24, 2021
c953f5d
CRM config update
qbdwlr Mar 1, 2021
364fcae
minor tweaks
qbdwlr Mar 8, 2021
453ca2a
Updated few of the comments based on HLD review
mohan-selvaraj Apr 19, 2021
6211994
[cbf] Added CBF spec
abanu-ms Jun 8, 2021
496c6c3
[cbf] Update flow diagram to fit
abanu-ms Jun 8, 2021
e65d05a
Update markdown formatting
abanu-ms Jun 8, 2021
dc4a7ae
MPLs HLD updates from code-review
qbdwlr Jun 14, 2021
6d9cdf8
Initial version of PINS HLD
bocon13 Jun 25, 2021
f619c03
update config DB table name and key separator
mohan-selvaraj Aug 2, 2021
7f22539
Update revision number and date, supported configuration modes
mohan-selvaraj Aug 2, 2021
fb74c29
[ACL] Flex Counters for ACL rules
stepanblyschak Aug 5, 2021
ec3b5bb
Update HLD with newest internal version
bocon13 Aug 11, 2021
078b908
Update pins_hld.md
bhagatyj Aug 12, 2021
774e54c
Added MPLS EXP to TC map HLD
abanu-ms Aug 16, 2021
70ba0f2
Update ACL-Flex-Counters.md
stepanblyschak Aug 17, 2021
76028d1
Update ACL-Flex-Counters.md
stepanblyschak Aug 17, 2021
c7a195c
[ACL] user defined ACL table types
stepanblyschak Aug 18, 2021
c71cf1b
update ACL table type HLD
stepanblyschak Aug 19, 2021
ade9a61
update ACL table type HLD
stepanblyschak Aug 19, 2021
e3ef045
Update dynamic buffer calculation HLD
keboliu Aug 20, 2021
8858b7a
update ACL table type HLD
stepanblyschak Aug 25, 2021
fe1bf2e
update ACL table type HLD
stepanblyschak Aug 26, 2021
63c28cf
Updates for new SAI model
abanu-ms Aug 26, 2021
5560588
update ACL table type HLD
stepanblyschak Aug 26, 2021
c0f071a
Update ACL table type HLD
stepanblyschak Aug 30, 2021
ca0031f
update ACL table type HLD
stepanblyschak Aug 30, 2021
e113735
update ACL table type HLD
stepanblyschak Aug 31, 2021
085ff23
rephrase the description
keboliu Aug 31, 2021
959f266
update ACL flex counters HLD
stepanblyschak Aug 31, 2021
08da370
update ACL flex counters HLD
stepanblyschak Aug 31, 2021
04abc04
update ACL flex counter HLD
stepanblyschak Sep 6, 2021
7add0f7
update ACL flex counter HLD
stepanblyschak Sep 6, 2021
0c136e7
update ACL table type HLD
stepanblyschak Aug 31, 2021
4e94ae4
update HLD
stepanblyschak Sep 9, 2021
2bb22a2
Update doc/qos/dynamically-headroom-calculation.md
keboliu Sep 13, 2021
11fed3d
Update dynamically-headroom-calculation.md
keboliu Sep 13, 2021
0d2001f
Merge branch 'master' of github.com:azure/sonic into acl-table-type
stepanblyschak Sep 15, 2021
14d99d9
Update ACL table type HLD
stepanblyschak Sep 15, 2021
32c85f4
Update ACL table type HLD
stepanblyschak Sep 15, 2021
96a65f0
Renamed MPLS EXP to MPLS TC
abanu-ms Sep 21, 2021
62b8d99
update ACL table type HLD
stepanblyschak Sep 21, 2021
3c6d6eb
Merge branch 'Azure:master' into acl-table-type
stepanblyschak Sep 21, 2021
a13f7d2
Update Acl Table Type HLD
stepanblyschak Sep 22, 2021
5cf41a3
Merge branch 'acl-table-type' of github.com:stepanblyschak/sonic into…
stepanblyschak Sep 22, 2021
bb476c5
Add per-port binding configuration details
abanu-ms Sep 22, 2021
3f9d005
SYSTEM READY
sg893052 Oct 1, 2021
1a48477
Update pins_hld.md
bhagatyj Oct 19, 2021
255d61a
Update Json_Change_Application_Design.md
renukamanavalan Oct 26, 2021
60e3b6d
Updated the images for fetching the latest build
kannankvs Oct 27, 2021
8eb0b75
updated the link for naming convention (#889)
kannankvs Oct 27, 2021
655bd1a
Merge pull request #867 from stepanblyschak/acl-table-type
bingwang-ms Oct 28, 2021
b2b7a2b
Merge pull request #890 from kannankvs/supported_platform_html_update…
zhangyanzhao Oct 28, 2021
d1b6624
Updated a typo error
kannankvs Oct 29, 2021
39b92d7
Merge pull request #893 from kannankvs/supported_platform_html_update…
zhangyanzhao Oct 29, 2021
f6e4b21
[pbh] dynamic implementation inner hashing (#824)
AntonHryshchuk Oct 31, 2021
11d5eb5
Updated statement on breakout handling. Minor correction.
mohan-selvaraj Nov 3, 2021
cf30172
Routed Subinterface enhancements (#833)
preetham-singh Nov 4, 2021
decc8eb
Merge pull request #441 from mohan-selvaraj/bum_storm_control
zhangyanzhao Nov 9, 2021
984ab4b
Merge pull request #875 from sg893052/SYSTEM_READY
zhangyanzhao Nov 9, 2021
9939c54
EVPN VxLAN update for platforms using P2MP tunnel based L2 forwarding…
dgsudharsan Nov 10, 2021
fff5342
SONIC Multi Architecture Build Support (#632)
antony-rheneus Nov 10, 2021
157141b
Fix typo in path '/host/ztp/ztp_data_local.json' (#868)
vmorokhx Nov 10, 2021
eea5e60
Revert "SYSTEM READY"
lguohan Nov 10, 2021
f6e354d
Add QoS config update test case in design doc (#892)
qiluo-msft Nov 11, 2021
e6346b4
Updated image file for md file (#899)
kannankvs Nov 11, 2021
23545e2
[HLD] Event Driven TechSupport Invocation & CoreDump Mgmt (#818)
vivekrnv Nov 11, 2021
69b861a
Add "Add/Remove rack end-to-end test" for generic config updater HLD …
qiluo-msft Nov 11, 2021
079df5d
fix review comments
keboliu Nov 12, 2021
ead0cbe
[Dynamic buffer] Updating peer response from a fixed value to dynamic…
neethajohn Nov 15, 2021
9707aa2
Merge pull request #857 from stepanblyschak/acl-flex-counters
bingwang-ms Nov 17, 2021
4508ddb
Updated MoM page till Oct 21 (#902)
kannankvs Nov 17, 2021
577779c
initial commit
ds952811 Sep 28, 2021
602964f
Migrate to sfp-refactoring framework
ds952811 Nov 8, 2021
9eda506
cosmetic updates
ds952811 Nov 8, 2021
8f99da5
update the exmaple output of CMIS EEPROM
ds952811 Nov 8, 2021
4e31486
Add implementation details and CMIS media settings
ds952811 Nov 12, 2021
2fac62b
Add example logs of CMIS manager
ds952811 Nov 12, 2021
c4b7770
Update the scope
ds952811 Nov 15, 2021
d811eb0
Update the HLD to support state-based app init and default app only
ds952811 Nov 19, 2021
b04947f
Update Json_Change_Application_Design.md
renukamanavalan Nov 23, 2021
a07d964
Add a couple of more implementation details
ds952811 Nov 23, 2021
a91573c
Add more details for CMIS state transition
ds952811 Nov 23, 2021
09dc6bb
rephrase
ds952811 Nov 23, 2021
183067b
Address review comments
ds952811 Nov 29, 2021
bfb222a
New Dell Platforms Update (#905)
kannankvs Dec 1, 2021
0e4516d
Add 'Outside the scope' and a few cosmetic changes
ds952811 Dec 1, 2021
9d48008
Merge pull request #876 from ds952811/cmis
prgeor Dec 1, 2021
004b175
Update 202106 Release Note document (#904)
Yuval-Mellanox Dec 6, 2021
bc9b0a4
Reclaim reserved buffer for unused ports (#831)
stephenxs Dec 6, 2021
0a01667
Overlay ECMP with BFD support (#861)
prsunny Dec 6, 2021
4f89ea8
Adjust system health HLD due to output of 'monit summary -B' command …
Junchao-Mellanox Dec 8, 2021
d5edcec
Final updates to reflect approved/merged PR content
qbdwlr Dec 9, 2021
78f3f73
Order ECMP HLD. (#896)
abdosi Dec 10, 2021
74b334f
[flow counters] High level design for flow counters support (#858)
Junchao-Mellanox Dec 15, 2021
39c12c5
Created a file for supported platform md format to update build image…
kannankvs Dec 15, 2021
11f12a3
[auto-techsupport] Added the note regarding sonic-sonic upgrade (#910)
vivekrnv Dec 15, 2021
89938fe
[app-ext] add Application Extension guide (#878)
stepanblyschak Dec 15, 2021
dc05062
Create Interface-Link-bring-up-sequence.md
shyam77git Dec 22, 2021
323b161
Update Interface-Link-bring-up-sequence.md
shyam77git Dec 23, 2021
2341c35
updated the file with credentials (#923)
kannankvs Jan 12, 2022
7c25fbc
Policy Based Hashing: Edit Flows HLD (#909)
nazariig Jan 15, 2022
736b9a8
Update Interface-Link-bring-up-sequence.md
shyam77git Jan 19, 2022
28a67ee
Update Interface-Link-bring-up-sequence.md
shyam77git Jan 19, 2022
b802395
Update Interface-Link-bring-up-sequence.md
shyam77git Jan 19, 2022
4c735ca
Update Interface-Link-bring-up-sequence.md
jaganbal-a Jan 21, 2022
17f1e86
Merge pull request #1 from jaganbal-a/patch-2
shyam77git Jan 21, 2022
d9d1ba6
Update Interface-Link-bring-up-sequence.md (#2)
jaganbal-a Jan 22, 2022
74a7e66
Put port operational speed to STATE DB
Junchao-Mellanox Jan 20, 2022
87ae10f
Update Interface-Link-bring-up-sequence.md
shyam77git Jan 25, 2022
c8fcff2
Updated the HLD with route and neighbor command outputs in BGP Unnumb…
AkhileshSamineni Jan 26, 2022
9091931
SONiC NVGRE Tunnel HLD (#869)
vadymhlushko-mlnx Jan 27, 2022
c272efa
Update Interface-Link-bring-up-sequence.md
shyam77git Jan 28, 2022
faf8731
HLD for the SONiC Management Framework "show tech-support" command (#…
kerry-meyer Jan 31, 2022
6c85773
Update Interface-Link-bring-up-sequence.md
shyam77git Feb 2, 2022
4a39c08
Update Interface-Link-bring-up-sequence.md
shyam77git Feb 3, 2022
fdc4e15
Update Interface-Link-bring-up-sequence.md (#3)
jaganbal-a Feb 3, 2022
1bdd505
Update Interface-Link-bring-up-sequence.md
shyam77git Feb 3, 2022
e1744f1
Adjust system health HLD for SYSTEM READY feature (#930)
Junchao-Mellanox Feb 7, 2022
66277d7
Password Hardening HLD (#874)
davidpil2002 Feb 8, 2022
6a344c5
Merge pull request #916 from shyam77git/patch-1
prgeor Feb 8, 2022
9bc8390
Add files via upload
Kalimuthu-Velappan Feb 10, 2022
768a897
High level design for route flow counter (#908)
Junchao-Mellanox Feb 13, 2022
5a72943
Copp manager redesign test plan (#903)
JibinBao Feb 16, 2022
45c315b
Srv6 hld (#795)
heidinet2007 Feb 16, 2022
09454a3
Initial version of PINS P4RT App HLD (#826)
rhalstea Feb 16, 2022
e690fa5
PINS Packet I/O HLD (#850)
kishanps Feb 16, 2022
ef9ded9
TC remapping HLD
bingwang-ms Feb 18, 2022
1ffb3d4
HLD document on add/remove ports dynamically feature (#900)
tomer-israel Feb 24, 2022
c4df429
Merge pull request #932 from Junchao-Mellanox/port-oper-speed
prgeor Feb 25, 2022
37a50b5
HLD for DSCP remapping
bingwang-ms Feb 25, 2022
9492c49
Update doc
bingwang-ms Feb 25, 2022
48eec5c
Address commentes
bingwang-ms Feb 28, 2022
505135e
Merge pull request #809 from pins/pins-hld
zhangyanzhao Mar 1, 2022
66d3d1d
Add files via upload
Kalimuthu-Velappan Mar 1, 2022
2d6ea89
updated md.sh file (#951)
kannankvs Mar 2, 2022
42b4ff7
Created SONiC 202111 release notes (#929)
kannankvs Mar 2, 2022
9a5d90f
Updated MoM till Feb end (#953)
kannankvs Mar 2, 2022
12d907f
Add an img to cover user scenario
bingwang-ms Mar 4, 2022
b7980db
Fix new line
bingwang-ms Mar 4, 2022
5b6e794
Add traffic flow with PFC
bingwang-ms Mar 7, 2022
be5fd7a
Add files via upload
Kalimuthu-Velappan Mar 7, 2022
a1370cc
Add SAI version check to SONiC build system (#935)
stepanblyschak Mar 15, 2022
c3f49bb
Merge pull request #942 from Kalimuthu-Velappan/master
xumia Mar 16, 2022
c5d0b64
Merge branch 'Azure:master' into master
kartik-arista Mar 18, 2022
55024e8
Add detailed map
bingwang-ms Mar 18, 2022
9c4114f
Design changes to support configurable Maximum SAs per SC
kartik-arista Mar 18, 2022
31b8462
Update CoPP design with new always_enabled field info (#962)
noaOrMlnx Mar 20, 2022
61e6c87
[auto-techsupport] support techsupport generation on potential memory…
stepanblyschak Mar 20, 2022
30e0c23
Fixed review comments
kartik-arista Mar 22, 2022
25f6976
[MACSec] HLD updates for supporting configurable Max SAs per SC
Pterosaur Mar 22, 2022
2e585d8
Merge pull request #796 from Metaswitch/cbf
zhangyanzhao Mar 23, 2022
6db781a
Merge pull request #844 from Metaswitch/exp_to_tc
zhangyanzhao Mar 23, 2022
decb477
Merge pull request #706 from qbdwlr/master
zhangyanzhao Mar 23, 2022
bec3417
Fix incorrect table
bingwang-ms Mar 23, 2022
b41966b
Add full list
bingwang-ms Mar 24, 2022
76baf8a
Add deadlock img
bingwang-ms Mar 24, 2022
eadfc89
Address comments
bingwang-ms Mar 25, 2022
29e67ca
TC7 -> PG0
bingwang-ms Mar 28, 2022
909691c
Fix typo
bingwang-ms Mar 29, 2022
5f20d9a
Merge pull request #950 from bingwang-ms/pcbb_dscp_remapping
bingwang-ms Mar 29, 2022
d1159ca
Updated 2011 Release notes with SAI 1.9.1 Release notes link (#969)
kannankvs Mar 30, 2022
e47901d
Library Update
donNewtonAlpha Mar 31, 2022
412b503
Fixing formatting
donNewtonAlpha Mar 31, 2022
0e86e9f
Update Overlay ECMP with BFD.md
prsunny Apr 3, 2022
a61cfc0
Update doc/pins/Packet_io.md
ai-cu Apr 4, 2022
64c5590
Update doc/pins/Packet_io.md
ai-cu Apr 4, 2022
f46667a
Update Packet_io.md
ai-cu Apr 4, 2022
3e00c1e
Updated the files for double braces error (#975)
kannankvs Apr 6, 2022
dd5c89b
Update transceiver-monitor-hld.md (#974)
keboliu Apr 6, 2022
bc4cd9d
Update due to new flags
ai-cu Apr 6, 2022
d0c8c57
fix version number
donNewtonAlpha Apr 7, 2022
a082f70
fix grammer
donNewtonAlpha Apr 7, 2022
a94913b
fixiing spelling
donNewtonAlpha Apr 7, 2022
ed48095
merging upstream updates
donNewtonIntel Apr 7, 2022
1aaa76c
Library Update
donNewtonAlpha Mar 31, 2022
4e7a173
Fixing formatting
donNewtonAlpha Mar 31, 2022
f5b4efd
Update doc/pins/Packet_io.md
ai-cu Apr 4, 2022
0a0a472
Update doc/pins/Packet_io.md
ai-cu Apr 4, 2022
8bc3ea4
Update Packet_io.md
ai-cu Apr 4, 2022
bbb4e30
Update due to new flags
ai-cu Apr 6, 2022
1767171
fix version number
donNewtonAlpha Apr 7, 2022
9f485bd
fix grammer
donNewtonAlpha Apr 7, 2022
5c2fb8d
fixiing spelling
donNewtonAlpha Apr 7, 2022
6941d92
Merge branch 'don/PacketIO' of github.com:pins/SONiC into don/PacketIO
donNewtonIntel Apr 7, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 52 additions & 2 deletions doc/pins/Packet_io.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,18 @@ _Rev v0.1_
Rev | RevDate | Author(s) | Change Description
---- | ---------- | ----------- | ------------------
v0.1 | 06/23/2021 | Google, ONF | Initial Version
v0.1 | 03/31/2022 | Google, ONF | Extract Library
donNewtonAlpha marked this conversation as resolved.
Show resolved Hide resolved



## Scope

This document covers the high level design aspects of Packet I/O in SONiC for P4Runtime application.
This document covers the high level design aspects of Packet I/O in SONiC for both the P4Runtime application and any subsequent application that would benifit from packet exchange using Generic Netlink Sockets.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

benifit->benefit



## Overview

SONiC supports Packet I/O on Linux netdev interfaces but this does not meet some unique requirements of P4Runtime application. This document details the requirements and captures the design changes necessary to meet the new requirements.
SONiC supports Packet I/O on Linux netdev interfaces but this does not meet some unique requirements of P4Runtime application. This document details the requirements and captures the design changes necessary to meet the new requirements. This document also explains the extraction of a library from the P4Runtime application in order to make the functionality available to other applications as required. In addition to the new library this document will explain a new application which serves as a model for using the library and allows for TCPDump like sniffing capability and packet generation using Generic Netlink Sockets.


**Requirements**
Expand Down Expand Up @@ -176,5 +177,54 @@ Vendor work is needed to enable the creation of the “submit_to_ingress” port

![drawing](images/p4rt_flow.png)

## New Library Usage

The library functionality is defined in genl-packet/receive_genetlink.h. First a callback function must be defined with the following signature:

```
using ReceiveCallbackFunction = std::function<void(
    const std::string& src_port_name, const std::string& target_port_name,
    const std::string& payload)>;
```
Then the StartReceive function is then called which returns a thread which calls the callback function on packet receipt. 
donNewtonAlpha marked this conversation as resolved.
Show resolved Hide resolved
```
std::thread StartReceive(
packet_metadata::ReceiveCallbackFunction callback_function,
nl_recvmsg_msg_cb_t process_callback_function);
```
A version of process_callback_function is implemented in genl-packet/receive_genenlink.cc and will be used if process_callback_function is NULL. The purpose of process_callback_function is to extract the netlink attributes: source port, destination port and payload and pass them to the ReceiveCallbackFunction. This can be implemented by the library user if the logic in the library is insufficient.

## Library Phases
- **Current:**
- The current kernel module, documented in PacketIO.md, remains unchanged and is supplied by the asic vendor or delegates.
- There is a single multicast group / queue that all producers and consumers can use.  All packets are sent to all consumers.  Filtering must be done post-consumption.
- Packet meta-data is fixed.

- **Dynamic generic netlink:**
- The kernel module will have an API that allows user space applications to manage multicast groups and queues at runtime.
- The library will allow applications to specify which multicast group / queue to listen to and allow for basic filtering.
- Packet meta-data continues to be fixed.
- **Dynamic meta-data:**
- Kernel module is likely unchanged.
- Multicast group / queue functionality is likely unchanged.
- Packet meta-data can be specified by the using application.  

## Sniffer Application:
The sniffer provides the means of a tcpdump-like tool to listen to the genetlink device. The sniffer/listener can be used for listening to traffic, as well as recording the traffic into a file or displaying to standard out. The resulting pcapng file can then be viewed using Wireshark. The sender can be used to send an example packet or packets from a pcap/pcapng file through genetlink. The sender also registers a new genetlink family and group called genl_packet and packets respectively. Both sniffer and sender use the pcapplusplus library which is an actively maintained open source library.
ai-cu marked this conversation as resolved.
Show resolved Hide resolved

Both the sender and the sniffer can be compiled via bazel or sonic-buildimage. Either way, once compiled or the necessary binary installed the following commands can be used to use the two applications ([sniffer] indicates the listener application and [sender] indicates the sender application):
- [sniffer] : launches the listener and records all packets into a file named out.pcapng
- [sniffer] -a : will either append the packets to out.pcapng or to a custom filename if given
- [sniffer] -filename=hello.pcapng : will write the packets into hello.pcapng
- [sniffer] -verbose : will print out verbose information about the packets received including metadata and packet contents.
- [sniffer] -=true : will simply print the payload into standard out.
ai-cu marked this conversation as resolved.
Show resolved Hide resolved

The packet metadata carried with process_callback_function gets put into a comment. If the sniffer is to be run outside of P4Runtime the user might want to construct their own custom receive thread using customCallbackReceive found in the header file for the sniffer, since the carried metadata might be different.
ai-cu marked this conversation as resolved.
Show resolved Hide resolved

- sudo [sender] : will send a sample packet using genetlink.
- sudo [sender] -inputfile=hello.pcapng : will read the packets from a given file and send them via genetlink.
- sudo [sender] -packet=AABBCCDD : will send the given packet in hex representation via genetlink.