Using TRILL, FabricPath, and VXLAN: Designing Massively Scalable Data Centers (MSDC) with Overlays

Using TRILL, FabricPath, and VXLAN

Designing Massively Scalable Data Centers with Overlays

TRILL, FabricPath, and VXLAN overlays help you distribute data traffic far more effectively, dramatically improving utilization in even the largest data center networks. Using TRILL, FabricPath, and VXLAN is the first practical and comprehensive guide to planning and establishing these high-efficiency overlay networks. The authors begin by reviewing today’s fast-growing data center requirements, and making a strong case for overlays in the Massive Scale Data Center (MSDC). Next, they introduce each leading technology option, including FabricPath, TRILL, LISP, VXLAN, NVGRE, OTV, and Shortest Path Bridging (SPB). They also present a chapter-length introduction to IS-IS, focusing on details relevant to the control of FabricPath and TRILL networks. Building on this foundation, they offer in-depth coverage of FabricPath: its advantages, architecture, forwarding, configuration, verification, and benefits in Layer-2 networks. Through examples, they explain TRILL’s architecture, functionality, and forwarding behavior, focusing especially on data flow. They also fully address VXLAN as a solution for realizing IP-based data center fabrics, including multi-tenant cloud applications.

Using TRILL, FabricPath, and VXLAN provides detailed strategies and methodologies for FabricPath, TRILL, and VXLAN deployment and migration, as well as best practices for management and troubleshooting. It also presents three detailed implementation scenarios, each reflecting realistic data center challenges. In particular, the authors show how to integrate multiple overlay technologies into a single end-to-end solution that offers exceptional flexibility, agility, and availability.

Sanjay K. Hooda is principal engineer in Catalyst switching software engineering at Cisco. He has more than 15 years of network design and implementation experience in large enterprise environments, and has participated in IETF standards activities. His interests include wireless, multicast, TRILL, FabricPath, High Availability, ISSU, and IPv6. He is co-author of IPv6 for Enterprise Networks.

Shyam Kapadia, Technical Leader at Cisco’s Data Center Group (DCG), was an integral part of the team that delivered the next-generation Catalyst 6500 Sup 2T (2 Terabyte) platform. Since then, he has focused on developing new solutions for data center environments. He holds a Ph.D. in computer science from USC, where his research encompassed wired, wireless, ad hoc, vehicular, and sensor networks.

Padmanabhan Krishnan has more than 12 years of experience in networking and telecommunications, including 7 at Cisco. His recent experience has included providing data path solutions for TRILL in the Catalyst 6500 Sup 2T Platform using FPGA, as well as design and development of platform core infrastructure and L2 features.

n  Discover how overlays can address data center network problems ranging from scalability to rapid provisioning

n  Examine popular data center overlay examples

n  Learn about extensions to IS-IS for TRILL and FabricPath

n  Use FabricPath, TRILL, and VXLAN to simplify configuration, improve performance and availability, optimize efficiency, and limit table size

n  Learn about FabricPath control and data plane architecture details

n  Review example FabricPath configurations on Cisco Nexus 7000/6000/5000 switches

n  Understand TRILL concepts and architecture, including overlay header, control and data plane, and MAC address learning

n  Learn about VXLAN architecture details and packet forwarding

n  Review example VXLAN configurations on a Cisco Nexus 1000V distributed virtual switch

n   Implement TRILL/FabricPath networks with VXLAN to virtualized servers in an intra-data center environment

n   Connect multiple traditional data centers using an OTV overlay as a Layer 2 extension

n   Use OTV overlays to connect sites running FabricPath, TRILL, or both

Table of Contents

Chapter 1 Need for Overlays in Massive Scale Data Centers 1

Evolution of the Data Center 1

    Changing Requirements of Data Centers 4

    Data Center Architectures 6

    CLOS 8

    Fat-Tree 9

    Single Fabric 9

    Need for Overlays 10

Summary 15

References 15

Overlay Technologies Overview 20

Cisco FabricPath 22

    FabricPath Requirements 22

    FabricPath Benefits 23

FabricPath Architecture 24

    FabricPath Encapsulation 24

    FabricPath Data Plane Operation 25

TRILL 26

    TRILL Requirements 27

    TRILL Frame Format 28

    TRILL Data Plane Operation 28

Locator ID/Separator Protocol 30

    LISP Frame Format 30

    LISP Routing 30

VXLAN 32

    VXLAN Frame Format 33

    VXLAN Data Path Operation 34

NVGRE 35

    NVGRE Frame Format 36

    NVGRE Data Path Operation 36

Overlay Transport Virtualization 38

    OTV Frame Format 39

    OTV Operation 40

Provider Backbone Bridges (PBB) 41

Shortest Path Bridging 43

    Shortest Path Bridging MAC 43

    Shortest Path Bridging VID 45

Summary 47

References 47

Introduction to IS-IS 49

Concepts 50

    Neighbor Discovery 51

    Topology Exchange 51

    Flooding 51

    Route Computation 52

    Link State Protocol Scaling 52

    Link State Protocol in a Local Area Network 53

IS-IS Architecture Details 55

TRILL and FabricPath Specific Changes in IS-IS 56

    Overview of TRILL and FabricPath 57

    IS-IS Frame Formats 58

    TRILL IS-IS Neighbor Discovery 59

TRILL and FabricPath Topology Exchange 67

    Flooding 69

    Nickname or SwitchID Resolution 70

    Shortest Path Computation 71

    Distribution Trees Computation 71

    ESADI 77

    Fine Grained Labeling 79

    Pseudo Node 81

    Multi Topology Routing 83

Summary 84

References 84

Additional Resources 84

FabricPath Overview 86

FabricPath Architecture 87

    Core and Edge 88

    Addressing Concepts 89

    VLANs 89

    vPC+ 89

    FabricPath Encapsulation 91

    FabricPath Control Plane Protocols 93

    FabricPath Dynamic Resource Allocation Protocol 97

        FabricPath MAC Address Learning 98

FabricPath STP Interaction 102

    Topology Change Notifications Forwarding 105

FabricPath Packet Forwarding 106

    Broadcast: ARP Request 108

    Unicast: ARP Reply 111

    Unicast: Data 113

    IP Multicast Forwarding 116

FabricPath Basic Configuration 119

    FabricPath Benefits 121

Summary 122

References 122

Need for TRILL 124

    Spanning Tree in Layer 2 Networks 124

    Issues with Spanning Tree Protocol 126

    Virtual Switching System 127

    Giant Virtual Switch 128

    Flat Network 129

    Layer 3 Network 130

Concepts and Terminologies 130

    RBridge 131

    Appointed Forwarder 132

    Addressing Concepts 133

TRILL Frame Format 134

TRILL Control Plane 136

    Unicast 136

    Multicast 137

TRILL Data Plane 141

    Unicast 141

    Multidestination 143

MAC Address Learning in TRILL-Based Networks 147

    Dynamic Learning 147

    Learning Through Control Protocols 147

Work in Progress 148

    Multitopology Routing 148

    Fine-Grained Labeling 149

    Pseudo Node 152

Case Studies 159

    Bidirectional Packet Flow 159

    Packet Flow for Pseudo Node 167

Summary 174

References 174

Additional Resources 175

VXLAN Overview 177

    Advent of VXLAN 178

VXLAN Architecture 179

    VXLAN Header Format 181

VXLAN Packet Forwarding 182

    Broadcast: ARP Request 183

    Unicast: ARP Reply 184

    Unicast: Data 186

    Unknown Unicast 187

    VM Mobility Case 188

    IPv6 Data Forwarding 190

    NS Request and NA Response 191

    VXLAN Gateway 192

    Inter-VXLAN Communication 196

    Layer 3 Multicast 198

    Multicast-Less VXLAN 200

    Floodless VXLAN Forwarding 203

    VXLAN as a Network Overlay 205

    Other VXLAN Considerations 207

VXLAN Basic Configuration 208

    VXLAN Gateway Configuration 210

Summary 211

References 211

vPC 214

    vPC Overview 214

    vPC Terminology 215

    vPC Benefits 216

    vPC Deployment Scenarios 217

    Double-Sided vPC 218

    vPC Operations 219

    vPC Traffic Flow 224

    Cisco Fabric Services over Ethernet 225

    vPC ARP Sync 225

    vPC Peer Gateway 225

    vPC Verification 227

vPC+ 231

    vPC+ Overview 231

    vPC+ Basics 232

    vPC+ Basic Packet Flow 236

    Active/Active HSRP Forwarding 238

FabricPath Topologies 241

Migration to FabricPath Network 242

    Conversion from Classical Layer 2 to FabricPath Network 242

    Conversion of vPC to vPC+ (Classical Ethernet to FabricPath) 244

    Configuring vPC+ on Secondary Switch 246

    Configuring vPC+ on Primary Switch 249

    Conversion of Access Switch (Sw3) Connecting to Secondary (Sw2) to FabricPath 251

    Converting Access Switch Sw3 Uplink Connecting to Sw1 to FabricPath 254

Monitoring and Troubleshooting in FabricPath Networks 257

    Loopback Message 258

    Path Trace Message 259

    Multicast Trace Message 259

    FabricPath OAM Configuration Model 261

Summary 270

References 270

Introduction 271

TRILL Deployment 271

    TRILL Between Access and Distribution 274

    TRILL Core 274

    Expanding the POD 285

    TRILL Everywhere 286

    Meshed Distribution with No Core 287

    Link Aggregation or Pseudo-Node Deployments 287

    CLOS Network Model 289

Migration Toward TRILL 290

    TRILL and Spanning Tree 291

    Bottom-up Migration Toward TRILL 296

    Top-down Migration Toward TRILL 298

Monitoring and Troubleshooting in TRILL Networks 299

    OAM Packet Format 300

    Connectivity Verification 302

    Path Tracing 303

    TRILL Configuration Model 304

Summary 304

References 305

Overview 307

Case Study 1: TRILL or FabricPath Network with VXLAN to Virtualized Servers 309

Case Study 2: Data Center Interconnect Using OTV 315

Case Study 3: Interconnecting TRILL or FabricPath Data Centers Using OTV 321

    Merging TRILL or FabricPath Networks 321

    Independent TRILL or FabricPath Networks 323

    Interconnection of TRILL and FabricPath Data Centers 325

    Packet Flow 325

Summary 327

References 328

Submit Errata