Search
  • Mike Ghazaleh

How to Troubleshoot NSX-T (using built-in tools)

Whether you're brand new to NSX-T, or you've used it a while, you know the importance of being able to troubleshoot. If you're new, you might be concerned about the perceived lack of ability to troubleshoot NSX (compared to a traditional network switch/router). If you've used NSX a bit, you might have an active issue with it and you're looking for some help.


Either way, I gotcha covered. In this post, I'm going to discuss my top tools for troubleshooting NSX.


Need an actual NSX-T Troubleshooting Course?

Check our course out here (use coupon code NRDYNATION)


Tool #1: Troubleshooting with TraceFlow


If you're not familiar with traceflow, it's basically a built-in tool inside of the NSX Manager, that allows you to generate packets in your network - and trace what happens to them. It's basically traceroute on steroids. Using traceflow allows you to see if a packet was dropped to a routing issue, or if it was perhaps dropped due to a firewall issue (and it'll tell you exactly which rule is causing the headaches).


I have personally used traceflow to identify routing, MTU, firewall, and even suboptimal routing issues - it's awesome. The screenshot below shows what the output looks like in Traceflow. You can access this via the "Plan & Troubleshoot" tab in NSX-T Manager.




Tool #2: Packet Captures in NSX


One of my favorite methods to troubleshoot in NSX is to run a packet capture. Packet captures in NSX are a little different than you're probably used to, but VMware has been making it easier with every release. As of this writing, you can capture packets in your network via the GUI for both VM-VM flows, as well as VM-to-physical flows that traverse your NSX Edge nodes.


To get a little more technical, there's two places I'd recommend running packet captures in an NSX environment. The first, is on your NSX Edges (as I mentioned above). The capture point here is actually on your T0 uplink - which sits on your edge VMs.


The second capture point that would be useful is on your Transport Nodes (NSX lingo for a vSphere/ESXi host that has been prepared for NSX). This can be excellent as a way to see packets are even leaving the VM, hitting the VDS, or leaving the host. You can do all of this easily in NSX.


*NOTE: I've heard reports that doing packet captures on Active/Active T0's in NSX may not be supported as of this writing, so just be aware of that.


The video below shows you how to do a packet capture in NSX 4.0 using the GUI:



Tool #3: Ping & Traceroute


I know you probably came to this post looking for some secret tips on troubleshooting NSX, but just as ping and traceroute were excellent tools to troubleshoot traditional networks - they're also awesome for NSX environments as well!


You can confirm via ping that your problem VM is able to ping it's default gateway (remember, in NSX, this default gateway lives on the distributed router, which is on all of your hosts). You can then make sure your VM can ping other VMs, or even physical devices. Once the pings start failing, you've now helped isolate where the problem is.


As for traceroute, this helps a lot when troubleshooting North-South connectivity - in other words - traffic coming in and out of your NSX networks (ie: to the internet).


Tool/Method #4: Where are the logs at???


I have a saying - if you're looking for logs, you're having a bad day. I do believe that's true, but, if you don't have logs, your bad day is gonna last a lot longer! In NSX, you can view most logs pretty easily. First, there's a bunch of logs located on your NSX Manager (locations listed below). If you're looking for firewall logs, those are stored on your ESXi hosts by default - NOT on the NSX manager.


You can find your firewall rule logs at the following location on your ESXi hosts:

/var/log/dfwpktlogs.log

*NOTE on NSX Manager logs: you can access them via the linux shell, or you can use the NSX CLI to run "get log" - they're the same essentially, but the NSX CLI method means you don't have to memorize log file locations.


Log Locations on the NSX Manager:


You can find all logs on your NSX manager in this directory:

/var/log

Alternately, you can use the NSX CLI to view logs as such:


get log-file <auth.log | controller | controller-error | http.log | kern.log | manager.log | node-mgmt.log | policy.log | syslog> [follow]


Need more help learning how to Troubleshoot NSX-T?


If you found this article helpful, but want more hands-on examples and live video from an expert NSX trainer - we have a brand new NSX-T Troubleshooting Course. You can check it out by hitting the button below. Be sure to use the coupon code "NRDYTECH" to get 70% off.



One of the nice things about the course above is that while there are a few slides, a lot of tricks and tips are included in the course from actual experience with hundreds of NSX/NSX-T customers. Don't take our word for it - check it out!


Additional NSX-T Troubleshooting references:


NSX-T 3.0 Operations Guide

NSX-T 3.1 Administration Guide

NSX-T Command Line Reference


13 views

Recent Posts

See All