Preface

Authors and Contributors

This book represents a joint intense collaborative effort between Cisco’s Engineering, Technical Support, Advanced Services and Sales employees over a single week in the same room at Cisco Headquarters Building 20 in San Jose, CA.

Authors

Andres Vega - Cisco Technical Services
Bryan Deaver - Cisco Technical Services
Jerry Ye - Cisco Advanced Services
Kannan Ponnuswamy - Cisco Advanced Services
Loy Evans - Systems Engineering
Mike Timm - Cisco Technical Services
Paul Lesiak - Cisco Advanced Services
Paul Raytick - Cisco Technical Services

Distinguished Contributors

Giuseppe Andreello
Pavan Bassetty
Sachin Jain
Sri Goli
Contributors
Lucien Avramov
Piyush Agarwal
Pooja Aniker
Ryan Bos
Mike Brown
Robert Burns
Mai Cutler
Tomas de Leon
Luis Flores
Maurizio Portolani
Michael Frase
Mioljub Jovanovic
Ozden Karakok
Jose Martinez
Rafael Mueller
Chandra Nagarajan
Mike Petrinovic
Daniel Pita
Mike Ripley
Zach Seils
Ramses Smeyers
Steve Winters

Dedications

“For Olga and Victoria, my love and happiness, hoping for a world that continues to strive in providing more effective solutions to all problems intrinsic to human nature.” - Andres Vega

“An appreciative thank you to my wife Melanie and our children Sierra and Jackson for their support. And also to those that I have had the opportunity to work with over the years on this journey with Cisco.” - Bryan Deaver

“For my parents, uncles, aunts, cousins, and wonderful nephews and nieces in the US, Australia, Hong Kong and China.” - Jerry Ye

“To my wife, Vanitha for her unwavering love, support and encouragement, my kids Kripa, Krish and Kriti for the sweet moments, my sister who provided me the education needed for this book, my brother for the great times, and my parents for their unconditional love.” - Kannan Ponnuswamy

“Would like to thank my amazing family, Molly, Ethan and Abby, without whom, none of the things I do would matter.” - Loy Evans

“Big thanks to my wife Morena, my daughters Elena and Mayra. Thank you to my in-laws Guadalupe and Armando who helped watch my beautiful growing girls while I spent the time away from home working on this project.” - Michael Timm

“Dedicated to the patience, love and continual support of Amanda; my sprite and best friend” - Paul Lesiak

“For Susan, Matthew, Hanna, Brian, and all my extended family, thanks for your support throughout the years. Thanks as well to Cisco for the opportunity, it continues to be a fun ride.” - Paul Raytick

Acknowledgments

While this book was produced and written in a single week, the knowledge and experience leading to it are the result of hard work and dedication of many individual inside and outside Cisco.

Special thanks to Cisco’s INSBU Executive, Technical Marketing and Engineering teams who supported the realization of this book. We would like to thank you for your continuous innovation and the value you provide to the industry.

We want to thank Cisco’s Advanced Services and Technical Services leadership teams for the trust they conferred to this initiative and the support provided since the inception of the idea.

In particular we want to express gratitude to the following individuals for their influence and support both prior and during the book sprint:

Shrey Ajmera
Subrata Banerjee
Dave Broenen
John Bunney
Luca Cafiero
Ravi Chamarthy
Mike Cohen
Kevin Corbin
Ronak Desai
Krishna Doddapaneni
Mike Dvorkin
Tom Edsall
Ken Fee
Vikki Fee
Siva Gaggara
Shilpa Grandhi
Ram Gunuganti
Ruben Hakopian
Robert Hurst
Donna Hutchinson
Fabio Ingrao
Saurabh Jain
Praveen Jain
Prem Jain
Soni Jiandani
Sarat Kamisetty
Yousuf Khan
Praveen Kumar
Tighe Kuykendall
Adrienne Liu
Anand Louis
Gianluca Mardente
Wayne McAllister
Rohit Mediratta
Munish Mehta
Sameer Merchant
Joe Onisick
Ignacio Orozco
Venkatesh Pallipadi
Ayas Pani
Amit Patel
Maurizio Portolani
Pirabhu Raman
Alice Saiki
Christy Sanders
Enrico Schiattarella
Priyanka Shah
Pankaj Shukla
Michael Smith
Edward Swenson
Srinivas Tatikonda
Santhosh Thodupunoori
Sergey Timokhin
Muni Tripathi
Bobby Vandalore
Sunil Verma
Alok Wadhwa
Jay Weinstein
Yi Xue

We would also like to thank the Office of the CTO and Chief Architect for their hospitality while working in their office space.

We are truly grateful to our book sprint facilitators Laia Ros and Adam Hyde for carrying us throughout this collaborative knowledge production process, and to our illustrator Henrik van Leeuwen who took abstract ideas and was able to depict those ideas into clear visuals. Our first concern was how to take so many people from different sides of the business to complete a project that traditionally takes months. The book sprint team showed that this is possible and presents a new model for how we collaborate, extract knowledge and experience and present it into a single source.

Book Writing Methodology

The Book Sprint (www.booksprints.net) methodology was used for writing this book. The Book Sprint methodology is an innovative new style of cooperative and collaborative authorship. Book Sprints are strongly facilitated and leverage team-oriented inspiration and motivation to rapidly deliver large amounts of well authored and reviewed content, and incorporate it into a complete narrative in a short amount of time. By leveraging the input of many experts, the complete book was written in a short time period of only five days, however involved hundreds of authoring man hours, and included thousands of experienced engineering hours, allowing for extremely high quality in a very short production time period.

Who Should Read This Book?

Expected Audience

The intended audience for this book is those with a general need to understand how to operate and/or troubleshoot an ACI fabric. While operation engineers may experience the largest benefit from this content, the materials included herein may be of use to a much wider audience, especially given modern industry trends towards continuous integration and development, along with the ever growing need for agile DevOps oriented methodologies.

There are many elements in this book that explore topics outside the typical job responsibilities of network administrators. For example the programmatic manipulation of policy models can be viewed as a development-oriented task, however has specific relevance to networking configuration and function, taking a very different approach than traditional CLI-based interface configuration.

Organization of this Book

Section 1: Introduction to ACI Troubleshooting

The introduction covers basic concepts, terms and models while introducing the tools that will be used in troubleshooting. Also covered are the troubleshooting, verification and resolution methodologies used in later sections that cover the actual problems being documented.

Section 2: Sample Reference Topology

This section sets the baseline sample topology used throughout all of the troubleshooting exercises that are documented later in the book. Logical diagrams are provided for the abstract policy elements (the endpoint group objects, the application profile objects, etc) as well as the physical topology diagrams and any supporting documentation that is needed to understand the focal point of the exercises. In each problem description in Section 3, references will be made to the reference topology as necessary. Where further examination is required, the specific aspects of the topology being examined may be re-illustrated in the text of the troubleshooting scenario.

Section 3: Troubleshooting Application Centric Infrastructure

The Troubleshooting ACI section goes through specific problem descriptions as it relates to the fabric. For each iterative problem, there will be a problem description, a listing of the process, some verification steps, and possible resolutions.

Chapter format: The chapters that follow in the Troubleshooting section document the various problems: verification of causes and possible resolutions are arranged in the following format.

Overview: Provides an introduction to the problem in focus by highlighting the following information:

  • Theory and concepts to be covered
  • Information of what should be happening
  • Verification steps of a working state

Problem Description: The problem description will be a high level observation of the starting point for the troubleshooting actions to be covered. Example: a fabric node is showing “inactive” from the APIC by using APIC CLI command acidiag fnvread.

Symptoms: Depending on the problem, various symptoms and their impacts may be observed. In this example, some of the symptoms and indications of issues around an inactive fabric node could be:

  • loss of connectivity to the fabric
  • low health score
  • system faults
  • inability to make changes through the APIC

In some chapters, multiple symptoms could be observed for the same problem description that require different verification or resolution.

Verification: The logical set of steps to identify what is being observed will be indicated along with the appropriate tools and output. Additionally, some information on likely causes and resolution will be included.