Project: Fault Management and Maintenance (Doctor)

Doctor is fault management and maintenance project to develop and realize the consequent implementation for the OPNFV reference platform.

The goal of this project is to build fault management and maintenance framework for high availability of Network Services on top of virtualized infrastructure. The key feature is immediate notification of unavailability of virtualized resources from VIM, to process recovery of VNFs on them.
Requirement survey and development of missing feature in NFVI and VIM are in scope of this project to fulfill requirements for fault management and maintenance in NFV.

NOTE: Doctor project has an implementation of the requirements which we identified. Doctor project contributes all functional codes to upstream projects, such as OpenStack. Only testing tools and document can be found in OPNFV Doctor repository.

• Doctor project "Solution Brief"
• Project Proposal (Presentation Slides, Old Version)
• Doctor Graduation as mature project
• Deliverables/Documents:
  • Arno (EOL): Requirement (PDF)
  • Brahmaputra (EOL): Requirement(PDF), Platform overview(PDF), User guide(PDF), Config Guide(PDF), Manual for OpenStack Nova API(PDF)
  • Colorado: Scenario description, User guide, Release notes
  • Danube: Release notes, Config guide, User guide, Development guide(How to test, API), Requirement, Design
  • Fraser:  Release notes, Config guide, User guide, Development guide(How to test, API), Testing
  • Latest build: Requirement
• Presentation Slides
  • Presentation & Demo at Prague Hackfest (Feb. 24, 2015)
  • Presentation at OpenStack Summit / OPNFV Hackfest (May 18, 2015)
  • Demo and Updates at Santa Clara Hackfest (July 30, 2015)
  • Controller, Presentation, Demo (video) at OPNFV Summit (Nov 9-12, 2015)
  • Maintenance API at Santa Clara Hackfest (March 15th, 2016)
  • Congress+Doctor, SFQM+Doctor at OpenStack Summit Austin (April 28, 2016)
  • PoC Demos (Handout, Congress, Vitrage-Recorded Demo), Inspector(Vitrage/Congress),  Doctor+SFQM, Deep Dive at OPNFV Summit 2016 (June 20-23, 2016)
  • Keynote demo, Inspectors, DPDK stats at OpenStack Summit Barcelona 2016 (Oct 25-28)
  • Updates, Doctor-QTIP: Faster, Higher, Stronger, Enabling Carrier-Grade Availability Within a Cloud Infrastructure, Maintenance PoC (recorded video) at OPNFV Summit Beijing 2017 (June 12-15, 2017)

  • Telco Scheduled Host Maintenance POC video (Oct 26, 2017), Telco Scheduled Host Maintenance POC ppt (Oct 10 2017)

  • Telco Scheduled Host Maintenance at OPNFV Plugfest Hillsboro (Dec 6 2017)

  • Infrastructure Maintenance & Upgrade: Zero VNF Downtime with OPNFV Doctor on OCP Hardware. OCP and ONS summit (March 2018) demo videoslides

  • How to gain VNF zero down-time during Infrastructure Maintenance and Upgrade OpenStack Vancouver summit presentation (May 2018)

  • Rolling maintenance and upgrade in interaction with VNFM OpenStack summit forum (May 2018)
  • New OpenStack project for rolling maintenance and upgrade in interaction with application on top of it OPNFV and OpenStack community call for participation e-mail (May 2018)

  • Rolling maintenance and upgrade in interaction with application on top of it OPNFV TC call presentation (May 2018)

• Workspace
  • Planned Maintenance Design Guideline.pdf 
  • Doctor Architecture
    • Figure sources: Opnfv_doctor_deliverable_v02.pptx
    • Discussion (Feb 6, 2015)
    • Proposed Implementation outside of Nova (Feb 10, 2015)
    • Relevance with ETSI NFV E2E Arch
    • SB API Discussion: 2015-04-14, 2016-02-23
  • Old Implementation Plan / Blueprints https://etherpad.opnfv.org/p/doctor_bps
    • Ceilometer: Rev.1, Rev.2, Rev.3, Rev.4
    • Nova: Original Plan (Mar 10, 2015), Updated Plan (Sep 1, 2015)
    • Nova Maintenance: May-10-2016, May-17-2016, June-21-2016
    • Old Doctor Dashboard https://etherpad.opnfv.org/p/doctor
  • Release Planning
    • Brahmaputra Planning (EOL)
    • Colorado Planning (EOL)
    • Danube Planning (EOL)
    • Euphrates Planning (EOL)
    • Fraser Planning (EOL)
    • Gambia Planning  (EOL)
    • Hunter Planning (EOL)
    • Iruya Planning (current stable release, bug and security-supported)
    • Jerma Planning (under development)
  •  Current priority work: Rolling infrastructure maintenance and upgrade in interaction with application on top of it
    
    • Implementation to OpenStack Fenix project
    • OpenStack Stein PTG, Fenix:
      • Upgrade SIG https://etherpad.openstack.org/p/upgrade-sig-ptg-stein
      • Blazar https://etherpad.openstack.org/p/blazar-ptg-stein
      • Self Healing SIG https://etherpad.openstack.org/p/self-healing-sig-stein-ptg
      • Airship https://etherpad.openstack.org/p/AirshipPTG4
    • OpenStack Berilin Forum, Fenix:
      • Planning etherpad: https://etherpad.openstack.org/p/fenix-forum-stein
      • Session etherpad: https://etherpad.openstack.org/p/BER-Forum-Fenix-Upgrade
    • ONAP DDF + OPNFV Plugfest, Paris-Sacley:
      
      • Doctor presentation for faster fault management and maintenance test using Fenix: Rolling Maintenance and Enhanced Fault Management Demo
      • Presentation about Fenix: OpenStack Fenix BoF
    • ONS America 2019, San Jose, CA:
      • Rolling OpenStack upgrade demonstrated by utilizing Doctor testing framework and OpenStack Fenix (video): Fenix rolling OpenStack upgrade in interactions with VNFM
      • Fenix latest design (video): Fenix - Rolling Upgrade, Maintenance and Scaling with Zero Impact to VNF
        
    • Open Infrastructure Summit + PTG, Denver May 2019.
      • Fenix Denver - Train - forum presentation
      • PTG:
        https://etherpad.openstack.org/p/DEN2019-fenix-PTG
        https://etherpad.openstack.org/p/DEN2019-fenix-ETSI-NFV-PTG
    • ONS Europe 2019, Antwerpen
      • In ONS there was a demonstration in LFN booth for Telco Edge healing and mainteannce using Fenix 
      • Presentation in ONS Europe 2019:
        Resiliency Closed Loop - Telco Self-healing and Maintenance
        Live demo in ONS Europe 2019:
        Resiliency Closed Loop - Telco Self-healing and Maintenance
        Unconference in ONS Europe 2019:
        Host maintenance with Fenix

• Meeting: Info & Logs, Note (Agenda), Join
• IRC: #opnfv-doctor @ Freenode (Web Chat)
• Mailing List: use opnfv-tech-discuss and tag your emails with [doctor] in the subject for easier filtering
• Review: https://gerrit.opnfv.org/gerrit/
• Repository: https://gerrit.opnfv.org/gerrit/
• Git Web: https://gerrit.opnfv.org/gerrit/gitweb?p=doctor.git;a=summary
• JIRA: https://jira.opnfv.org/browse/DOCTOR

Key Project Facts

Contributors:

• Dirk Kutscher (NEC, Dirk.Kutscher@neclab.eu)
• Palani Chinnakannan (Cisco, pals@cisco.com)
• Petri Kemppainen (Nokia, petri.kemppainen@nokia.com)
• Tapio Tallgren (Nokia, tapio.tallgren@nokia.com)
• Zhangyu (Huawei, zhangyu11@huawei.com)
• Michael Godley (Intel, michael.godley@intel.com)
• Andreas Ripke (NEC, Andreas.Ripke@neclab.eu)
• Maryam Tahhan (Intel, maryam.tahhan@intel.com)
• Gurpreet Singh (Spirent, Gurpreet.Singh@spirent.com)
• Iben Rodriguez (iben.rodriguez@vmsec.com)
• Victor Laza (Cloudbase Solutions, vlaza@cloudbasesolutions.com)
• Alessandro Tosti (Telecom Italia, alessandro.tosti@telecomitalia.it)
• Dan Druta (AT&T, dd5826@att.com)
• Balázs Gibizer (Ericsson, balazs.gibizer@ericsson.com)
• Uli Kleber (Huawei, ulrich.kleber@huawei.com) 
• Yuanzhen Li (ZTE, li.yuanzhen@zte.com.cn)
• Masanori Miyazawa (KDDI, ma-miyazawa@kddilabs.jp)
• Tomonobu Niwa (KDDI, to-niwa@kddilabs.jp)
• Yujun Zhang (ZTE, zhang.yujunz@zte.com.cn)
• Woosung Jeon (KT DT, woosung.jeon@kt.com)
• Tarek EL-Basyouny (Deutsche Telekom)
• Trinh Nguyen (Individual contributor, dangtrinhnt@edlab.xyz)

If you would like to contribute to this project please add yourself to the contributors list and contact us. See also Developer Getting Started .