Difference between revisions of "Hardware Management/Hardware Fault Management"

From OpenCompute
Jump to: navigation, search
m (Working Documents)
(46 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[[File:OCP-hardware-management-v1-17a3x.png|right]]
+
[[File:OCP-Open-compute-hardware-fault-management-icon-3x-v1-1a.png|right]]
 
==Welcome==
 
==Welcome==
  
Line 8: Line 8:
 
:'''Disclaimer''': Please do not submit any confidential information to the Project Community. All presentation materials, proposals, meeting minutes and/or supporting documents are published by OCP and are open to the public in accordance to OCP's Bylaws and IP Policy. This can be found on the OCP [http://www.opencompute.org/about/ocp-policies/ OCP Policies] page.  If you have any questions please contact OCP.
 
:'''Disclaimer''': Please do not submit any confidential information to the Project Community. All presentation materials, proposals, meeting minutes and/or supporting documents are published by OCP and are open to the public in accordance to OCP's Bylaws and IP Policy. This can be found on the OCP [http://www.opencompute.org/about/ocp-policies/ OCP Policies] page.  If you have any questions please contact OCP.
  
===Sub-Project Leadership===
+
==Project Overview==
 +
The Hardware Fault Management subproject’s goal is to address the Large scale data center pain-points in managing hardware faults effectively.
 +
Objective of the subproject: to define hardware fault management solution such that it
 +
 
 +
- Minimizes performance degradation
 +
- Reduces No Trouble Found (NTF) in RMA
 +
- Scales the solution across heterogeneous fleet
  
:- [mailto:zhengyuyang@fb.com Zhengyu Yang] (Facebook), [mailto:ramab@microsoft.com Rama Bhimanadhuni] (Microsoft), & [mailto:anil.agarwal@intel.com Anil Agarwal] (Intel)
+
Activities of the sub-project include:
 +
- Develop running list of pain-points related to HW fault handling in the fleet(Facebook): [https://drive.google.com/file/d/1zrML07aKGBALBFzClBaM3pZMRZ_K-65r/view?usp=sharing OCP-PainpointList]
 +
- Develop hardware fault management solution best practices to address the above pain-points(Facebook): [https://drive.google.com/file/d/1SquZ9DS-BtJ7PnPfth1JtbvMUagNeozF/view?usp=sharing OCP-BestPractice]
 +
- Hardware Error reporting format Standardization(Microsoft): [https://docs.google.com/document/d/1JuxxsrUb5K4JkTxOLTk04fHF1IX_Lauy/edit HW error reporting format]
 +
- Define hardware error classification(HPE): [https://docs.google.com/document/d/1xQzeYlesh6SV57EA_yxKI30w8H-6Je7B/edit HW Error Classification]
 +
- OCP Fault Management Infrastructure Proposal(Google): [https://docs.google.com/document/d/1LOaVJb8oD1AzjCOhlen0rFsRi63rRoX7kyU01SKlblI/edit OCP Fault Management Infrastructure Proposal]
 +
- Standardizing system behavior under hardware failures (future topic)
 +
- Provide reference and guidance on system hardware failure management (future topic)
 +
 
 +
==Sub-Project Leadership==
 +
 
 +
:- [mailto:zhengyuyang@fb.com Zhengyu Yang] (Facebook), [mailto:anilagrawal@fb.com Anil Agrawal] (Facebook), [mailto:ramab@microsoft.com Rama Bhimanadhuni] (Microsoft), & [mailto:yogesh.varma@ocproject.net Yogesh Varma] (Intel)
  
 
==Get Involved==
 
==Get Involved==
:[http://opencompute.org/projects/projects-calendar/ OCP Calendar]
 
:[https://global.gotomeeting.com/join/865768677 Call Link]
 
  
:Regular meetings are held 8-9AM PDT on the last Friday of each month.  
+
:Participate in the discussion, mailing list: OCP-HWFaultMgt@OCP-All.groups.io [https://ocp-all.groups.io/g/OCP-HWFaultMgt/ Mailing List Info]
: - You can also dial in using your phone.
+
:Participate in HW Fault Management sub-project meetings
: - United States (Toll Free): 1 877 309 2073
+
: - [http://opencompute.org/projects/projects-calendar/ OCP Calendar]
: - United States: +1 (646) 749-3129
+
: - [https://global.gotomeeting.com/join/865768677 Call Link]
 +
 
 +
: - Regular meetings are held every other Friday from 2-3pm PT.
 +
:   - You can also dial in using your phone.
 +
:   - United States (Toll Free): 1 877 309 2073
 +
:   - United States: +1 (646) 749-3129
 +
:  - (Access Code: 865-768-677)
 +
 
 +
:Provide feedback and comments to working documents
  
: (Access Code: 865-768-677)
+
==Agendas and Minutes==
  
  
==Agendas and Minutes== 
 
  
 +
=== Meeting Date 07/22/21 ===
 +
 +
Participant: Anil (FB), Drew (Google), Mark (HPE), Rama (MS), Yogesh (Intel), Antonio (Intel)
 +
 +
Proposed Agenda:
 +
 +
1. OCP global summit planning (Anil)
 +
 +
2. Logistics of core meetings (Antonio/Yogesh)
 +
        a. Meeting time and duration change
 +
        b. Update pain-point spreadsheet
 +
 +
3. EDAC-extensions proposal (Anil)
 +
 +
Meeting Notes:
 +
 +
1. The following proposals were submitted for OCP Global Summit:
 +
        a. A study of 'Host-based Hardware Corrected Fault Reporting' methods in a compute platform for large-scale cloud infrastructure
 +
        b. Study of memory corrected error profile in cloud infrastructure
 +
        c. Mark: Error classification standardization
 +
2. Proposals getting ready to submit:
 +
        a. Rama/Antonio/Jeff Autor: Error formatting standardization proposal (plan to submit)
 +
          i.  Antonio to take lead for getting requirements published at the OCP portal
 +
          ii. Rama took the lead in submitting the proposal and presenting.
 +
        b. Drew: Error reporting architecture for cloud infrastructure
 +
        c. Yogesh: Status of HW fault management sub-project
 +
 +
3. Logistics of core meetings (Antonio/Yogesh)
 +
        a. Meeting time is agreed by the core team. Yogesh/Michael sent the revised meeting invitation. Alternate Fridays 2-3 PM PST
 +
        b. Pending Task - review the updates in pain-point spreadsheet
 +
4. EDAC-extensions proposal (Anil)
 +
        - Reviewed briefly. Need to discuss further in the following meeting.
  
 
==Working Documents==
 
==Working Documents==
  
 +
- Running list of pain-points related to HW fault handling: [https://drive.google.com/file/d/1zrML07aKGBALBFzClBaM3pZMRZ_K-65r/view?usp=sharing OCP-PainpointList]
  
==Presentations & Other Documents==
+
- Hardware fault management solution best practices to address the above pain-points: [https://drive.google.com/file/d/1SquZ9DS-BtJ7PnPfth1JtbvMUagNeozF/view?usp=sharing OCP-BestPractice]
OCP2020 Global Virtual Summit presentation: [https://onlinexperiences.com/Launch/Event.htm?ShowKey=86206&DisplayItem=E369254 Overview of HW Fault Management Sub-project]
+
 
 +
- Hardware Error reporting format Standardization: [https://docs.google.com/document/d/1JuxxsrUb5K4JkTxOLTk04fHF1IX_Lauy/edit HW error reporting format]
 +
 
 +
- Hardware error classification: [https://docs.google.com/document/d/1xQzeYlesh6SV57EA_yxKI30w8H-6Je7B/edit HW Error Classification]
  
OCP2019 Global Summit: [http://www.youtube.com/watch?v=hSHiN4XEJxc Minimizing the performance penalty associated with SMM based hardware error handling solution]
+
- OCP Fault Management Infrastructure Proposal : [https://docs.google.com/document/d/1LOaVJb8oD1AzjCOhlen0rFsRi63rRoX7kyU01SKlblI/edit OCP Fault Management Infrastructure Proposal]
  
OCP2019 Regional Summit: [https://www.youtube.com/watch?v=jBwIBD6izBs Platform Hardware Error Handling Requirements for Large Scale Data Center]
+
==Presentations & Other Documents==
 +
1. OCP Presentation Template - please contact [mailto:michael@opencompute.org Michael Schill] or [mailto:archna@opencompute.org Archna Haylock] for a copy
  
 +
2. OCP2020 Tech Week - Lighting Talk: [https://2020ocptechweek.fnvirtual.app/a/event/863 OCP HW Fault Management Sub-project Update]
  
==Project Overview==
+
3. OCP2020 Global Virtual Summit presentation: [https://onlinexperiences.com/Launch/Event.htm?ShowKey=86206&DisplayItem=E369254 Overview of HW Fault Management Sub-project]
The Hardware Fault Management subproject’s goal is to address the Large scale data center pain-points in managing hardware faults effectively.
 
Objective of the subproject: to define hardware fault management solution such that it
 
  
- Minimizes performance degradation
+
4. OCP2019 Regional Summit: [https://www.youtube.com/watch?v=jBwIBD6izBs Platform Hardware Error Handling Requirements for Large Scale Data Center]
- Reduces No Trouble Found (NTF) in RMA
 
- Scales the solution across heterogeneous fleet
 
  
Activities of the subproject include:
+
5. OCP2019 Global Summit: [http://www.youtube.com/watch?v=hSHiN4XEJxc Minimizing the performance penalty associated with SMM based hardware error handling solution]
- Define key baseline hardware fault management requirements to address the above pain-points
 
- Standardizing system behavior under hardware failures
 
- Provide reference and guidance on system hardware failure management
 
- Hardware Error reporting format Standardization.
 
  
 
==Past Calls - Recordings==   
 
==Past Calls - Recordings==   
:-
+
:- [https://www.youtube.com/watch?v=7pBzZKf7Nds August 20th, 2021]
 +
:- [https://www.youtube.com/watch?v=dku8mIa7Quc August 13th, 2021]
 +
:- [https://www.youtube.com/watch?v=ZupxY_yzHUY July 30th, 2021]
 +
:- [https://www.youtube.com/watch?v=DBfZ4VIlhdI June 25th, 2021]
 +
:- [https://www.youtube.com/watch?v=S9ANhKNFpk0 April 30th, 2021]
 +
:- [https://www.youtube.com/watch?v=52jkPU2UMeM March 26th, 2021]
 +
:- [https://www.youtube.com/watch?v=_Ua1GE7bY34 February 26th, 2021]
 +
:- [https://www.youtube.com/watch?v=cgC3M4t5C6c January 29th, 2021]
 +
:- [https://www.youtube.com/watch?v=gLsfA_uBPQE September 25th, 2020]
 +
:- [https://www.youtube.com/watch?v=jdDykEluzPc August 28th, 2020]
 +
:- [https://www.youtube.com/watch?v=kKpOoNkod4k July 31st, 2020]
 +
:- [https://www.youtube.com/watch?v=R4QnBKAgiZg June 26th, 2020]

Revision as of 12:31, 1 September 2021

OCP-Open-compute-hardware-fault-management-icon-3x-v1-1a.png

Welcome

Welcome to the OCP Hardware Fault Management Sub-project. This is a sub-project of the OCP Hardware Management Project
This Project is open to the public and we welcome all those who would like to be involved.
Disclaimer: Please do not submit any confidential information to the Project Community. All presentation materials, proposals, meeting minutes and/or supporting documents are published by OCP and are open to the public in accordance to OCP's Bylaws and IP Policy. This can be found on the OCP OCP Policies page. If you have any questions please contact OCP.

Project Overview

The Hardware Fault Management subproject’s goal is to address the Large scale data center pain-points in managing hardware faults effectively. Objective of the subproject: to define hardware fault management solution such that it

- Minimizes performance degradation
- Reduces No Trouble Found (NTF) in RMA
- Scales the solution across heterogeneous fleet

Activities of the sub-project include:

- Develop running list of pain-points related to HW fault handling in the fleet(Facebook): OCP-PainpointList
- Develop hardware fault management solution best practices to address the above pain-points(Facebook): OCP-BestPractice
- Hardware Error reporting format Standardization(Microsoft): HW error reporting format
- Define hardware error classification(HPE): HW Error Classification
- OCP Fault Management Infrastructure Proposal(Google): OCP Fault Management Infrastructure Proposal
- Standardizing system behavior under hardware failures (future topic)
- Provide reference and guidance on system hardware failure management (future topic)

Sub-Project Leadership

- Zhengyu Yang (Facebook), Anil Agrawal (Facebook), Rama Bhimanadhuni (Microsoft), & Yogesh Varma (Intel)

Get Involved

Participate in the discussion, mailing list: OCP-HWFaultMgt@OCP-All.groups.io Mailing List Info
Participate in HW Fault Management sub-project meetings
- OCP Calendar
- Call Link
- Regular meetings are held every other Friday from 2-3pm PT.
- You can also dial in using your phone.
- United States (Toll Free): 1 877 309 2073
- United States: +1 (646) 749-3129
- (Access Code: 865-768-677)
Provide feedback and comments to working documents

Agendas and Minutes

Meeting Date 07/22/21

Participant: Anil (FB), Drew (Google), Mark (HPE), Rama (MS), Yogesh (Intel), Antonio (Intel)

Proposed Agenda:

1. OCP global summit planning (Anil)

2. Logistics of core meetings (Antonio/Yogesh)

       a. Meeting time and duration change
       b. Update pain-point spreadsheet

3. EDAC-extensions proposal (Anil)

Meeting Notes:

1. The following proposals were submitted for OCP Global Summit:

        a. A study of 'Host-based Hardware Corrected Fault Reporting' methods in a compute platform for large-scale cloud infrastructure
        b. Study of memory corrected error profile in cloud infrastructure
        c. Mark: Error classification standardization

2. Proposals getting ready to submit:

       a. Rama/Antonio/Jeff Autor: Error formatting standardization proposal (plan to submit)
          i.  Antonio to take lead for getting requirements published at the OCP portal
          ii. Rama took the lead in submitting the proposal and presenting.
       b. Drew: Error reporting architecture for cloud infrastructure
       c. Yogesh: Status of HW fault management sub-project

3. Logistics of core meetings (Antonio/Yogesh)

       a. Meeting time is agreed by the core team. Yogesh/Michael sent the revised meeting invitation. Alternate Fridays 2-3 PM PST
       b. Pending Task - review the updates in pain-point spreadsheet

4. EDAC-extensions proposal (Anil)

       - Reviewed briefly. Need to discuss further in the following meeting.

Working Documents

- Running list of pain-points related to HW fault handling: OCP-PainpointList

- Hardware fault management solution best practices to address the above pain-points: OCP-BestPractice

- Hardware Error reporting format Standardization: HW error reporting format

- Hardware error classification: HW Error Classification

- OCP Fault Management Infrastructure Proposal : OCP Fault Management Infrastructure Proposal

Presentations & Other Documents

1. OCP Presentation Template - please contact Michael Schill or Archna Haylock for a copy

2. OCP2020 Tech Week - Lighting Talk: OCP HW Fault Management Sub-project Update

3. OCP2020 Global Virtual Summit presentation: Overview of HW Fault Management Sub-project

4. OCP2019 Regional Summit: Platform Hardware Error Handling Requirements for Large Scale Data Center

5. OCP2019 Global Summit: Minimizing the performance penalty associated with SMM based hardware error handling solution

Past Calls - Recordings

- August 20th, 2021
- August 13th, 2021
- July 30th, 2021
- June 25th, 2021
- April 30th, 2021
- March 26th, 2021
- February 26th, 2021
- January 29th, 2021
- September 25th, 2020
- August 28th, 2020
- July 31st, 2020
- June 26th, 2020