^ Top

NANOG Meeting Presentation Abstract

Building a Scalable Telemetry Collection Pipeline using Big-Data Approach
Meeting: NANOG57
Date / Time: 2013-02-06 9:30am - 10:00am
This item is webcast
Room: Crystal Ballroom A-C
Presenters: Speakers:

Petr Lapukhov, Microsoft

Petr Lapukhov is a network engineer at Microsoft Global Networking Services (GNS) Search team. The team is dedicated to supporting Bing Search engine as well as other online services at Microsoft, such as Bing Maps, Ad center, and MSN among others. The primary focus of the team is on areas of Data-Center/WAN network design, implementation and operations. GNS Search works in close cooperation with Autopilot – a team that develops and operates the software to automate Bing’s large-scale data-center infrastructure. Petr has more than 14 years of experience in networking, starting back in 90’s with Novell Netware/IPX, coaxial cables, Cisco 2500 boxes and dialup networks. Prior to joining Microsoft he was working as a CCIE instructor and network engineer at various positions in Russian companies and his alma mater – Kazan State University where he got his MSc in Applied Mathematics.
Abstract: Netflow/sFlow are a critical tools for network infrastructure monitoring and capacity planning. However, collecting and analyzing flow and other telemetry data for large scale networks could be a challenging task due to vast volume of data that has to be stored and joined with large external datasets for analysis. Commonly, relational databases have been used to store flow data and run OLAP-style analysis queries, but with very large data-sets such systems become prohibitively complex to operate. In this presentation we are sharing experience on building a scalable system for various telemetry data collection leveraging horizontally-scalable software load-balancers and using Map-Reduce framework for data mining. We demonstrate how a Map-Reduce type system could be used to store both flow-type and BGP data and build reports joining large data-sets together. Though our system is built using in-house tools, this approach could be reproduced using open-source software.
Files: pdfBuilding a Scalable Telemetry Collection Pipeline using Big-Data Appro(PDF)
youtubeBuilding a Scalable Telemetry Collection Pipeline using Big-Data Approach
Sponsors: None.

Back to NANOG57 agenda.

NANOG57 Abstracts

  • DNS 101
    Speakers:
    John Kristoff, Team Cymru;
  • Super Storm Sandy: Infrastructure Impacts
    Moderators:
    Daniel Golding, Datacenter Insight; Panelists:
    Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;
  • Super Storm Sandy: Infrastructure Impacts
    Moderators:
    Daniel Golding, Datacenter Insight; Panelists:
    Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;
  • Super Storm Sandy: Infrastructure Impacts
    Moderators:
    Daniel Golding, Datacenter Insight; Panelists:
    Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;
  • Super Storm Sandy: Infrastructure Impacts
    Moderators:
    Daniel Golding, Datacenter Insight; Panelists:
    Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;
  • Super Storm Sandy: Infrastructure Impacts
    Moderators:
    Daniel Golding, Datacenter Insight; Panelists:
    Scott A. Davis, DuPont Fabros Technology; Michael Poleshuk, Equinix; Michael J. Parks, Datapipe; Neil Crowley, Internap;
  • Internet Impacts of Hurricane Sandy
    Moderators:
    Jim CowieRenesys; .
    Panelists:
    John HeidemannUSC/Information Sciences Institute; .
    Emile AbenRIPE NCC; .
    Patrick GilmoreAkamai; .
    Doug MadoryRenesys; .
  • Internet Impacts of Hurricane Sandy
    Moderators:
    Jim CowieRenesys; .
    Panelists:
    John HeidemannUSC/Information Sciences Institute; .
    Emile AbenRIPE NCC; .
    Patrick GilmoreAkamai; .
    Doug MadoryRenesys; .
  • Internet Impacts of Hurricane Sandy
    Moderators:
    Jim CowieRenesys; .
    Panelists:
    John HeidemannUSC/Information Sciences Institute; .
    Emile AbenRIPE NCC; .
    Patrick GilmoreAkamai; .
    Doug MadoryRenesys; .
  • Internet Impacts of Hurricane Sandy
    Moderators:
    Jim CowieRenesys; .
    Panelists:
    John HeidemannUSC/Information Sciences Institute; .
    Emile AbenRIPE NCC; .
    Patrick GilmoreAkamai; .
    Doug MadoryRenesys; .
  • Internet Impacts of Hurricane Sandy
    Moderators:
    Jim CowieRenesys; .
    Panelists:
    John HeidemannUSC/Information Sciences Institute; .
    Emile AbenRIPE NCC; .
    Patrick GilmoreAkamai; .
    Doug MadoryRenesys; .

 

^ Back to Top