Networking/NIC Software: Difference between revisions

From OpenCompute
Jump to navigation Jump to search
(initial summary)
(Add OCP 2022 global summit presentation)
(10 intermediate revisions by 2 users not shown)
Line 4: Line 4:
:Welcome to the OCP '''Networking NIC Software''' Sub-Project.
:Welcome to the OCP '''Networking NIC Software''' Sub-Project.


Our goal is to bring together NIC vendors and hyperscaler and other users to spur discussion about relevant NIC features, and arrive at agreed concrete specs for interface and behavior.
Our goal is to bring together NIC vendors and hyperscaler and other users to spur discussion about relevant NIC features, and then define concrete unambiguous specifications for the API and behavior of each feature.


We aim to ensure that all vendor implementations conform to the actual business requirements of large users. These requirements have historically often been underspecified or shared only with individual vendors.
We aim to ensure that all vendor implementations support the true business requirements of large users. These requirements have historically often been underspecified or shared only with individual vendors (possibly under NDA). Which may lead to feature implementations that prove inapplicable, e.g., due to constraints on scale or lack of necessary telemetry.


For each feature, we aim to arrive at a clear specification that covers both driver API and behavioral elements, such as scalability and performance expectations. Additionally, we aim to publish concrete conformance tests that vendors can use to self-certify.
For each feature, after reaching consensus, we publish a specification that covers both driver API and behavior in terms of metrics and their threshold values. We aim to share conformance tests that vendors can use to self-certify.


===Charter===
===Charter===


The focus is on foundational NIC features.
The focus is on traditional NIC networking features.


Significant developments are taking place in SmartNICs. For now, features not strictly related to the core network transmit and receive path are out of scope for this subproject. This includes on-board co-processors, control plane offload and offload of non networking interfaces, such as NVME (storage).
Significant developments are taking place in SmartNICs. But features not strictly related to the core network transmit and receive path are out of scope for this venue. This includes on-board co-processors operating systems and control plane offload.


===Public===
===Public===
Line 21: Line 21:
Disclaimer: Please do not submit any confidential information to the Project Community. All presentation materials, proposals, meeting minutes and/or supporting documents are published by OCP and are open to the public in accordance to OCP's Bylaws and IP Policy. This can be found on the OCP [http://www.opencompute.org/about/ocp-policies/ OCP Policies] page.  If you have any questions please contact OCP.
Disclaimer: Please do not submit any confidential information to the Project Community. All presentation materials, proposals, meeting minutes and/or supporting documents are published by OCP and are open to the public in accordance to OCP's Bylaws and IP Policy. This can be found on the OCP [http://www.opencompute.org/about/ocp-policies/ OCP Policies] page.  If you have any questions please contact OCP.


==Documents==
==Presentations==


:- [https://2021ocpglobal.fnvirtual.app/a/event/1996 NICs for Hyperscale Deployments]: 2021 OCP Global Summit introductory presentation
:- '''2021 OCP Global Summit''': introducing the effort: '''NICs for Hyperscale Deployments''' [https://fntech.sfo2.digitaloceanspaces.com/PresentationMediaUploads/31/1996/NICs-for-hyperscalers-submission-1-c24cc1465b868b21deb33aec8a5af4d6.pdf slides][https://2021ocpglobal.fnvirtual.app/a/event/1996 video]
 
:- '''2022 OCP Tech Talks''': crypto offload deep dive: '''Scalable NIC inline crypto for hyperscale workloads''' [https://146a55aca6f00848c565-a7635525d40ac1c70300198708936b4e.ssl.cf1.rackcdn.com/images/47c145a883e583f9d97ac7636bbf21956e9f9295.pdf slides]
 
:- '''2022 OCP Global Summit''': core offloads: '''Hyperscale NICs: performance and features''' [https://fntech.sfo2.digitaloceanspaces.com/PresentationMediaUploads/40/2649/OCP-2022-NIC-foundational-offloads-67413e5b30eb9d78cad353bd9968ea63.pdf slides]


==Topics==
==Topics==
Line 31: Line 35:
The current list of active topics are
The current list of active topics are


====Telemetry====
===Telemetry===
Define a standard set of network device and queue statistics, with standard names and unambiguous definition of each statistic's meaning.
Define a standard set of network device and queue statistics, with standard names and unambiguous definition of each statistic's meaning.


Line 39: Line 43:
Standardize state-of-the-art TE mechanisms.
Standardize state-of-the-art TE mechanisms.


Develop common support for stateless [https://www.kernel.org/doc/html/latest/networking/segmentation-offloads.html#generic-segmentation-offload generic tunneling offload]: combine TCP segmentation offload (an indispensible offload for many workloads) with arbitrary tunnel protocols, instead of building an offload for each tunnel protocol. Because hyperscalers may use protocols, protocol stacks and protocol variants --proprietary or not-- that the vendor cannot always anticipate.
Develop common support for stateless [https://www.kernel.org/doc/html/latest/networking/segmentation-offloads.html#generic-segmentation-offload generic tunneling offload]. Support TCP segmentation offload, an offload that is indispensible for many workloads, in combination with any tunnel protocols, as opposed to building an offload for each tunnel protocol. Hyperscalers may use protocols, protocol stacks and protocol variants --proprietary or not-- that the vendor cannot always anticipate. Generic tunneling is a requirement for deployment at scale.


Develop industry standard support for [https://legacy.netdevconf.info/0x12/session.html?evolving-from-afap-teaching-nics-about-time Earliest Departure Time] (EDT) stateless rate limiting offload.
Develop industry standard support for [https://legacy.netdevconf.info/0x12/session.html?evolving-from-afap-teaching-nics-about-time Earliest Departure Time] (EDT) stateless rate limiting offload.


====Flow Steering====
===Flow Steering===
Device queues are the basis for scalable networking, as well as task isolation and userspace queues.
Device queues are the basis for scalable networking through load balancing (RSS), as well as task isolation and userspace network stacks.


Define
Define
# a common interface for configuring devices queues and RSS groups, including dynamically without device down.
# a common interface for configuring devices queues and RSS groups, including dynamic addition and removal without link downtime.
# a common interface for steering flows to queue (groups), including expectations on datapath performance and scalability.
# a common interface for steering flows to queue (groups), including expectations on datapath performance and scalability of number of steering rules.


Additionally, define
Additionally, define
# a common queue interface for userspace network stacks like DPDK and [https://research.google/pubs/pub48630/ Google SNAP].
# a common queue interface for userspace network stacks like DPDK and [https://research.google/pubs/pub48630/ Google SNAP].


===Crypto===
TLS, QUIC, IPSec and other cryptographic protocols cover a huge fraction of all traffic.


Hardware support for offload of these and other protocols is uneven.


==Project Leadership==
Define key requirements, such as support for specific cryptographic algorithms like ChaCha2020-Poly1305, specific protocols such as QUIC and resulting requirements, such as framing and stream re-synchronization.


===Incubation Committee Representative===
Current protocols under active work are
# QUIC: [https://www.rfc-editor.org/rfc/rfc9000.txt rfc 9000]
# PSP: scalable tunnel and transport mode protocol developed at Google ([https://github.com/google/psp/blob/main/doc/PSP_Arch_Spec.pdf spec on github])


:- [mailto:jason.forrester@ocproject.net Jason Forrester] (Target)
===Time Measurement===
Expand hardware capabilities from PTP to include high-rate, high-precision timestamps on every packet. Both Rx and Tx.


===Sub-Project Leads===
Accurate time measurement is essential to advanced congestion control (e.g., [https://dl.acm.org/doi/pdf/10.1145/3387514.3406591 Swift]), fleetwide latency SLO monitoring and even applications such as globally distributed databases ([https://research.google/pubs/pub39966/ Spanner]), among potentially others.
:- [mailto:kicinski@fb.com Jakub Kicinski] (Meta)
:- [mailto:willemb@google.com Willem de Bruijn] (Google)


==Get Involved==
==Get Involved==


:- [https://ocp-all.groups.io/g/OCP-Networking Main Networking Project Mailing List]
:- [https://ocp-all.groups.io/g/OCP-Networking Mailing List]
:- [http://www.opencompute.org/projects/networking/ OCP website]
:- [http://www.opencompute.org/projects/networking/ Networking parent project]
:- [http://opencompute.org/projects/projects-calendar/ OCP Calendar]
:- [https://www.opencompute.org/projects/project-and-ic-meetings-calendar Meetings Calendar]


==Regular Project Calls==
==Regular Project Calls==


TBD
We join the monthly OCP Networking meetings, every second Monday of the month at 10am PT. See the  [https://www.opencompute.org/projects/project-and-ic-meetings-calendar Meetings Calendar] for more details.
 
==Specs and Designs==
Link to the Specs and Designs page -http://www.opencompute.org/wiki/Networking/SpecsAndDesigns


'''[[Networking/MonthlyCalls|This page]]''' has:
==Project Leadership==
:- the agenda for upcoming calls
:- complete information on how to join the call
:- minutes from past calls.


==Recordings from Past Calls==
====Incubation Committee Representative====


==Specs and Designs==
:- [mailto:jason.forrester@ocproject.net Jason Forrester] (Target)
Link to the Specs and Designs page -http://www.opencompute.org/wiki/Networking/SpecsAndDesigns


===Past Workshops===
====Sub-Project Leads====
:- [mailto:kicinski@fb.com Jakub Kicinski] (Meta)
:- [mailto:willemb@google.com Willem de Bruijn] (Google)

Revision as of 02:29, 21 October 2022

OCP-networking-v1-17a3x.png

Welcome

Welcome to the OCP Networking NIC Software Sub-Project.

Our goal is to bring together NIC vendors and hyperscaler and other users to spur discussion about relevant NIC features, and then define concrete unambiguous specifications for the API and behavior of each feature.

We aim to ensure that all vendor implementations support the true business requirements of large users. These requirements have historically often been underspecified or shared only with individual vendors (possibly under NDA). Which may lead to feature implementations that prove inapplicable, e.g., due to constraints on scale or lack of necessary telemetry.

For each feature, after reaching consensus, we publish a specification that covers both driver API and behavior in terms of metrics and their threshold values. We aim to share conformance tests that vendors can use to self-certify.

Charter

The focus is on traditional NIC networking features.

Significant developments are taking place in SmartNICs. But features not strictly related to the core network transmit and receive path are out of scope for this venue. This includes on-board co-processors operating systems and control plane offload.

Public

This Sub-Project is open to the public and we want to welcome all those who would like to be involved.

Disclaimer: Please do not submit any confidential information to the Project Community. All presentation materials, proposals, meeting minutes and/or supporting documents are published by OCP and are open to the public in accordance to OCP's Bylaws and IP Policy. This can be found on the OCP OCP Policies page. If you have any questions please contact OCP.

Presentations

- 2021 OCP Global Summit: introducing the effort: NICs for Hyperscale Deployments slidesvideo
- 2022 OCP Tech Talks: crypto offload deep dive: Scalable NIC inline crypto for hyperscale workloads slides
- 2022 OCP Global Summit: core offloads: Hyperscale NICs: performance and features slides

Topics

The area is open for innovation, and new topics are encouraged.

The current list of active topics are

Telemetry

Define a standard set of network device and queue statistics, with standard names and unambiguous definition of each statistic's meaning.

For Linux based deployments, this builds on and extends the standard rtnl_link_stats64 and replaces much of the free-form current driver interface exposed through ethtool -S.

Traffic Engineering

Standardize state-of-the-art TE mechanisms.

Develop common support for stateless generic tunneling offload. Support TCP segmentation offload, an offload that is indispensible for many workloads, in combination with any tunnel protocols, as opposed to building an offload for each tunnel protocol. Hyperscalers may use protocols, protocol stacks and protocol variants --proprietary or not-- that the vendor cannot always anticipate. Generic tunneling is a requirement for deployment at scale.

Develop industry standard support for Earliest Departure Time (EDT) stateless rate limiting offload.

Flow Steering

Device queues are the basis for scalable networking through load balancing (RSS), as well as task isolation and userspace network stacks.

Define

  1. a common interface for configuring devices queues and RSS groups, including dynamic addition and removal without link downtime.
  2. a common interface for steering flows to queue (groups), including expectations on datapath performance and scalability of number of steering rules.

Additionally, define

  1. a common queue interface for userspace network stacks like DPDK and Google SNAP.

Crypto

TLS, QUIC, IPSec and other cryptographic protocols cover a huge fraction of all traffic.

Hardware support for offload of these and other protocols is uneven.

Define key requirements, such as support for specific cryptographic algorithms like ChaCha2020-Poly1305, specific protocols such as QUIC and resulting requirements, such as framing and stream re-synchronization.

Current protocols under active work are

  1. QUIC: rfc 9000
  2. PSP: scalable tunnel and transport mode protocol developed at Google (spec on github)

Time Measurement

Expand hardware capabilities from PTP to include high-rate, high-precision timestamps on every packet. Both Rx and Tx.

Accurate time measurement is essential to advanced congestion control (e.g., Swift), fleetwide latency SLO monitoring and even applications such as globally distributed databases (Spanner), among potentially others.

Get Involved

- Mailing List
- Networking parent project
- Meetings Calendar

Regular Project Calls

We join the monthly OCP Networking meetings, every second Monday of the month at 10am PT. See the Meetings Calendar for more details.

Specs and Designs

Link to the Specs and Designs page -http://www.opencompute.org/wiki/Networking/SpecsAndDesigns

Project Leadership

Incubation Committee Representative

- Jason Forrester (Target)

Sub-Project Leads

- Jakub Kicinski (Meta)
- Willem de Bruijn (Google)