^ Top

NANOG Meeting Presentation Abstract

NetOps Coding 201 - Building Facebook's FBAR for Network Devices
Meeting: NANOG66
Date / Time: 2016-02-10 2:30pm - 4:00pm
Room: Grande Ballroom
Presenters: Speakers:

David Swafford, Facebook

David Swafford is a Network Engineer who focuses on automation & tooling for Facebook's production network. Prior to FB, he worked in both the enterprise and academic space and has been involved with computer networking since 1997! Beyond networking and software, he's a Linux nerd and also loves diving into databases every now and again. Outside of tech, he can be found anywhere where both sunlight and competition might be - training for an alternate career in the sport of triathlon :-D! He also loves teaching and cooking for friends!
Abstract: Continuing with our theme of automating day-to-day operations, we'll dive into building your very own FBAR! FBAR is a system used by Facebook to handle server and network fault detection & repair -- which offsets much of our traditional NOC through software.

While FBAR on it's own is a massive system tightly integrated into all aspects of FB infrastructure, we're going to start fresh here and build a simplified version that focuses on the network side. The version we'll build follows the same model of parsing standard syslog messages into faults / events, and then running remediation scripts against those to further diagnose and potentially repair / mitigate the issue (i.e. moving traffic away for example from a bad path by changing BGP policy so that it's no longer user impacting).

In NetOps Coding 101, we focused heavily on regular expression parsing to build two example remediation scripts. We'll use those and additional ones here, but the focus of this session will not be on regular expressions or additional remediation scripts -- instead, we'll focus on the system itself.

The system we'll build should be more production ready by the end. It'll be structured in such a way that we can act on and remediate many devices at time (learning of Python topics such as threading / queues / and parallelism in general). We'll also focus on how to keep track of events, actions taken and their results (touching into the topics of storing and querying data) -- because we don't want to continuously run the same remediation script all day on the same device!

Note: This session builds from "NetOps Coding 101" -- but attendance of that is not required. If you have the basics of Python and Regex Parsing down, you'll be right at home!

Come have fun and hack with us and walk away with the knowledge to automate the mundane and shift into the new hybrid network engineer!
Files: pdfNetOps Coding 201 - Building Facebook's FBAR for Network Devices(PDF)
youtubeNetOps Coding 201 - Building Facebook's FBAR for Network Devices
Sponsors: None.

Back to NANOG66 agenda.

NANOG66 Abstracts

  • Conference Opening
    Speakers:
    Tony Tauber, Comcast; Greg Dendy, Equinix; Raj Khurana.
    Al Burgio, IIX;
  • Conference Opening
    Speakers:
    Tony Tauber, Comcast; Greg Dendy, Equinix; Raj Khurana.
    Al Burgio, IIX;
  • Conference Opening
    Speakers:
    Tony Tauber, Comcast; Greg Dendy, Equinix; Raj Khurana.
    Al Burgio, IIX;
  • Conference Opening
    Speakers:
    Tony Tauber, Comcast; Greg Dendy, Equinix; Raj Khurana.
    Al Burgio, IIX;
  • Coding BOF
    Speakers:
    Matt Griswold, United Internet Exchange; Job Snijders, NTT Communications; Jesse Sowell, MIT; Elisa Jasinska, BigWave;
  • Research and Education Track
    Speakers:
    Manish Karir, QuadMetrics; Seyed K. Fayaz, Carnegie Mellon University; Alberto Dainotti, CAIDA, UC San Diego; Luca Sani, IIT-CNR; Ruwaifa Anwar, Stony Brook University; Vicente De Luca, Zendesk;
  • Research and Education Track
    Speakers:
    Manish Karir, QuadMetrics; Seyed K. Fayaz, Carnegie Mellon University; Alberto Dainotti, CAIDA, UC San Diego; Luca Sani, IIT-CNR; Ruwaifa Anwar, Stony Brook University; Vicente De Luca, Zendesk;
  • Research and Education Track
    Speakers:
    Manish Karir, QuadMetrics; Seyed K. Fayaz, Carnegie Mellon University; Alberto Dainotti, CAIDA, UC San Diego; Luca Sani, IIT-CNR; Ruwaifa Anwar, Stony Brook University; Vicente De Luca, Zendesk;
  • Research and Education Track
    Speakers:
    Manish Karir, QuadMetrics; Seyed K. Fayaz, Carnegie Mellon University; Alberto Dainotti, CAIDA, UC San Diego; Luca Sani, IIT-CNR; Ruwaifa Anwar, Stony Brook University; Vicente De Luca, Zendesk;
  • Research and Education Track
    Speakers:
    Manish Karir, QuadMetrics; Seyed K. Fayaz, Carnegie Mellon University; Alberto Dainotti, CAIDA, UC San Diego; Luca Sani, IIT-CNR; Ruwaifa Anwar, Stony Brook University; Vicente De Luca, Zendesk;
  • Research and Education Track
    Speakers:
    Manish Karir, QuadMetrics; Seyed K. Fayaz, Carnegie Mellon University; Alberto Dainotti, CAIDA, UC San Diego; Luca Sani, IIT-CNR; Ruwaifa Anwar, Stony Brook University; Vicente De Luca, Zendesk;
  • Coding BOF
    Speakers:
    Matt Griswold, United Internet Exchange; Job Snijders, NTT Communications; Jesse Sowell, MIT; Elisa Jasinska, BigWave;
  • Coding BOF
    Speakers:
    Matt Griswold, United Internet Exchange; Job Snijders, NTT Communications; Jesse Sowell, MIT; Elisa Jasinska, BigWave;
  • Coding BOF
    Speakers:
    Matt Griswold, United Internet Exchange; Job Snijders, NTT Communications; Jesse Sowell, MIT; Elisa Jasinska, BigWave;
  • Peering Track
    Speakers:
    Brad Raymo, Microsoft; Aaron Hughes6connect; .
    Ciprian Marginean, AMS-IX; Daniel KoppDE-CIX; .
  • Peering Track
    Speakers:
    Brad Raymo, Microsoft; Aaron Hughes6connect; .
    Ciprian Marginean, AMS-IX; Daniel KoppDE-CIX; .
  • Peering Track
    Speakers:
    Brad Raymo, Microsoft; Aaron Hughes6connect; .
    Ciprian Marginean, AMS-IX; Daniel KoppDE-CIX; .
  • Peering Track
    Speakers:
    Brad Raymo, Microsoft; Aaron Hughes6connect; .
    Ciprian Marginean, AMS-IX; Daniel KoppDE-CIX; .

 

^ Back to Top