Welcome to the Edge, I’m at Your Service

I’ve mentioned, more than once by now, that the Edge is where the services routers exist. They do their thing, whether tied into a Tier-0 or Tier-1 router.

So what services do we have? I’ mentioned them in the last post, so let’s run through them again pretty quickly:

How much can we actually do with NAT? I mean, Source NAT and Destination NAT are pretty straightfoward functions, I think. Both have been supported in NSX for, well, quite a while now.
In NSX-T, stateful NAT is supported if the Edge is configured for Active-Standby HA.

We also support reflexive, or stateless NAT. This is useful when I’m running the Edge in Active-Active HA.

The Edge also has a firewall, much like we’re used to in NSX-V. This is a pretty straightfoward L3 firewall, and is used for the same things we use the ESG firewall for in NSX-V – controlling north/south traffic at the software-defined network perimeter.
But we get to add a new twist – the Edge firewall is available on both Tier 0 and Tier 1 routers, so tenants can have a little more control over what they allow into the tenant perimeter.

VPN Services
We also have L2 and L3 VPNs avaialble in the Edge, but they’re completely built and managed through the APIs at the moment. Sorry folks, no real VPN love in this series, just an honorable mention. No API-fu with this software-defined joker yet.

Load Balancer
Just remember, if you’re using a VM form factor Edge, it has to be Medium or Large to configure Load Balancing. Learn from my fail, where I deployed a Small Edge, and wondered for an hour or so why I couldn’t attach the LB to the Edge.

Ultimately, we recommend Large if you’re going to use Load Blaancing. Medium supports the LB function, but really only for proof of concept use cases.

The Load Balancer in NSX-T is still ultimately a function of the Edge. No real news here. The configuration is pretty straightforward, where a Load Balancer instance is created, at least one Virtual Server is attached to the Load Balancer, and the LB is attached to a Tier 1 Logical Router.
Server Pools need to be created – the Load Balancing Algorithm is defined here, with Round Robin and Least Connections settings (along with weighted versions for those), as well as a simple IP Hash algorithm. You decide whether you’re going to do TCP multiplexing.

Then you decide whether you’re doing a Transparent load balancer, or statically or dynamically mapping the NAT. Transparent load balancing is only supported if the LB is inline with the servers in the pool.
Then you statically or dynamically choose pool members. Dynamic pools use NSGroups, where static is just that – predefined and static.

Finally, you decide what health monitors you want for pool members – these can be active or passive (or both, if you really want). NSX-T ships with default Active Health Monitors for HTTP, HTTPS, ICMP, and TCP. You can create more if you want, and decide how you want them to check for health.

When you create Virtual Servers, you get to decide whether you’re load balancing L4 or L7, TCP or UDP, and what Application Profile you’re going to use, it’s IP address, associated Server Pools, and what kind of persistence you want.

Then you just toss all that together with the LB itself, and off you go! The biggest differences here (that I see) are the things we need to create to make a load balancer are somewhat consolidated, and we need to make sure we have large enough Edges to support the Load Balancer. And Application Rules are gone, but replaced with other rewrite and redirection policies called LB Rules, managed in the Virtual Server.

L2 Bridging
We can’t forget the venerable bridging capabilities. They haven’t been mentioned anywhere else yet. In NSX-T 2.1, we had to define a Bridge Cluster and populate it with ESXi Transport Nodes to get L2 bridging. That’s still around, but in NSX-T 2.2, a far better option is available (relatively speaking, of course – it’s 2018, bridging should be a niche use case these days, but it’s here if you need it).

Today, we can create Bridge Profiles and attach them to the Edge, which does a number of things for us:
* we can use Edges rather than ESXi hosts for bridging – that sounds less expensive already!
* Edges use DPDK for forwarding – so now bridging is (potentially) faster
* Edges have a firewall, which means we can limit traffic going to or from the bridged network

I think we have a winner here, for those rare occasions that L2 adjacency is necessary, especially for things like migrations from physical networks to logical.

So, there we have it, a quick rundown of things that need Services Routers on the Edge.


~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX
The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”
Three Controllers to Rule Them All (that just doesn’t have the same ring to it, does it?)
Beyond Centralization: The Local Control Plane
Transport Zones, Logical Switchies, and Overlays! Oh, My!
Which Way Do We Go? Let’s ask the Logical Router!
If You’re Not Living on the Edge, You’re Taking Up Too Much Room

If You’re Not Living on the Edge, You’re Taking Up Too Much Room

We talked about the Edge last time.  But we really didn’t get into the differences that we might expect.  I just said “we’ll get to that later”.  I suppose it’s later by now.

The Edge in NSX-T is little more than a container.  No, not “container” in a Docker kind of sense, but in a “pool of resources for network services” kind of sense.

See, the Edge is still a virtual machine, except when it isn’t.  In NSX-T, we have our choice of form factors – 3 sizes of virtual machine, and then bare metal.  Need a really, really big Edge?  Get out that server, we’re making a big network device!

So what are the form factors for an Edge these days?

  • Small – 2 vCPU, 4 GB RAM, 120 GB disk
  • Medium – 4 vCPU, 8 GB RAM, 120 GB disk
  • Large – 8 vCPU, 16 GB RAM, 120 GB disk
  • Bare Metal – 8 CPU, 32 GB RAM, 200 GB disk (minimums, naturally)

The virtual machine Edges come with 4 vmxnet3 vNICs installed.  Bare metal, well, we’ve got to watch out.  Chances are, your Intel X520, X540, X550, or X710s will work.  Check the docs for specifics – it’s all spelled out there in the System Requirements section of the Installation Guide

If you give a mouse an Edge, he’ll probably realize that it’s a single point of failure, and he’ll ask for another.

We can’t use Edges right after they’ve been deployed.  They’re really just Fabric Nodes at that point, joined to the Management Plane and just taking up resources.  They need to be promoted to Transport Nodes before they’ll be useful.

Because, while an Edge will still act as the North/South perimeter of our Software-Defined Network, it’s more than that.  An Edge is actually an active participant in the network – NSX host switches and TEPs will be installed on the Edge as part of this process.  And it runs distributed routing processes so that traffic coming through the Edge destined to a workload on a logical switch has an efficient routing path to get there. But TEPs and distributed routers are not all that are deployed to the Edge.

If you recall, from the logical routing post, the services router (SR) was mentioned.  The SR always lives on an Edge, whether the SR belongs to a Tier 0 or a Tier 1 router.  You certainly can influence on _which_ Edge a SR is running, but it’ll always be on one.

Let’s recap the kinds of things that might be deployed as a SR:

  • NAT
  • BGP (Tier 0 only)
  • Firewall
  • Load Balancer

That’s an awful lot of stuff in just 4 bullet points.  We’ll get into the specifics of those later.  These services are generally stateful, and generally need to be centralized.  So we did.

Because some things are centralized, there should probably be some measures taken for high availability (thus my nod to Laura Numeroff and Felicia Bond a couple of paragraphs ago, in case you missed the reference and thought I was simply going mad).  For HA of SRs, we need an Edge Cluster.

An Edge Cluster is simply a grouping of 1 or more identical form factor Edge transport nodes (maximum of 8) put together as a larger logical construct.  A container for your network services containers, if you will.  It’s pretty straightforward to create an Edge Cluster, simply add one in the Edge Clusters tab of Fabric > Nodes, add the Edge Transport Nodes you want participating, define the cluster profile, and off you go.

There’s not much to an Edge Cluster, really.  And not much to the cluster profile, either.  The profile simply defines the BFD probe interval, how many hops are allows, and how many probes have to be lost before we declare an Edge officially dead.

Most services operate statefully, and support only Active/Standby failover.  

We can configure routers as Active/Active or Active/Standby, and that will define what you can do with that router.  It’s worth noting that, if you configure a router as Active/Active, it is essentially a stateless device.  NAT is only available as Reflexive (or Stateless) NAT. The Edge Firewall is also stateless.

In Active/Standby, however, you can so stateful NAT and firewalling.  You can attach a Load Balancer.  But you have to choose your failover mode. If you ask me, you’re really choosing your fallback mode, but that’s not how it’s called out in the UI.  You have two failover mode choices:  Preemptive and Non-Preemptive.  What does that even mean?

It’s actually pretty simple.  In preemptive mode, you define a preferred member of your Edge cluster. This is the Edge we want to use.  If it fails, we’ve got others, so we’re not out for long.  What preemptive means, however, is that when our preferred Edge returns to service, NSX will preempt the service to move it back to the preferred node.  An automatic fallback, if you will.

In a non-preemptive configuration, we do not define a preferred node, so there’s no drive by NSX to move a service back to a preferred location.  No automatic fallback of the service.

Is there more to talk about? Of course there is.  We’ve still got services to talk about, and tooling, and all kinds of good stuff.  Stay tuned!


~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX
The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”
Three Controllers to Rule Them All (that just doesn’t have the same ring to it, does it?)
Beyond Centralization: The Local Control Plane
Transport Zones, Logical Switchies, and Overlays! Oh, My!
Which Way Do We Go? Let’s ask the Logical Router!

Which Way Do We Go? Let’s ask the Logical Router!

Moving around the data plane, we need to think about how, exactly, we’re going to get packets off of one logical switch and onto another.  Or out to the physical infrastructure, even.  

Logical routing in NSX-T is so very similar to what we have in NSX-V, but it’s entirely new at the same time.  

We still have the concept of a distributed router, embedded in the kernel to make routing decisions.  Pretty similar so far.  

Logical routers have interfaces, called Downlinks,  connected to the logical switches to enable routing between L2 domains. Still pretty much the same as we’re used to.

Logical routers no longer have a DLR Control VM. Well, there’s a bit of a departure from NSX-V.

Let’s reel things back into similarities.  We can still maintain two tiers of routing to keep tenants separated.  But we don’t refer to the tiers as the Distributed Logical Router and Edge Services Gateway any longer.  Now, it’s Tier-1 and Tier-0 routing, respectively.  This is where we have to start unlearning NSX-V things and relearning NSX-T things.

The NSX-T routers are all distributed, meaning the Tier-0 and Tier-1 routers are programmed on all the transport nodes throughout the transport zone.

So think about this logical topology:


Sample 2-Tier Routing Topology

Everything in the diagram, save for the physical router, is distributed across the transport zone.  

Except when they’re not.  Without diving too deep here, each of these routers, Tier 0, Tenant A Tier 1, and Tenant B Tier 1, are actually comprised of two objects: the distributed router (DR), and the services router (SR).

So let’s talk about these for a minute.  The distributed router component is the part that lives on each transport node.  This is the part that makes the routing decisions.  If I had two workloads: tenantA-Web and tenantB-Web, and those workloads were instantiated on the same hypervisor, traffic between them would not have to leave the host.  If tenantA-Web sent a ping to tenantB-Web, the traffic would go VM -> Tenant A Tier 1 -> Tier 0 -> Tenant B Tier 1 -> VM.

Perhaps breaking the environment up into tenants was a poor choice, as I likely wouldn’t allow traffic between them like that.  But that’s where other capabilities come in – the distributed firewall, for example.  We’ll talk about more of that kind of stuff as we continue through this series.

Anyway, we have our DR, but what’s this services router thing?  Simply put, it’s the component that we use for centralized or non-distributed services, such as dynamic routing, NAT, Firewalling, Load Balancing, L2 Bridging, etc.  

I know, you’re wondering, at this point, just what we’re thinking with a whole new routing structure when it sounds a lot like we’ve taken the model from NSX-V, and just distributed the ESG routing. Which is sort of true.  But it’s not.  Because the Tier 1 routers can run services just as well as the Tier 0 routers, which means everyone gets a services router component, assuming you’ve turned up one of these services.

That’s right, I can do NAT at Tier 0 or Tier 1, I can firewall at either tier, and on and on and on.  

But we’re not getting into the specifics of all of that yet.  

So where do these services routers live?  Since they’re centralized, we need someplace to put them.  How about on the Edge? We still have Edges in NSX-T, though they’re definitely no longer “Edge Services Gateways” – just Edges.  We’ll talk more about the specifics of them in the next installment.  For right now, just remember that all of the stateful or centralized services will live on an Edge node.

Another thing of note, is that I don’t need to deploy a two-tier routing topology.  I can attach a Tier 0 logical router to a logical switch.  If I don’t need a complex topology, you don’t have to build a complex topology.  Just cut Tier 1 completely out of the picture, and you’ll be fine.

This is a lot of information.  It may not seem like it, but it really is.  We’ll dive into the Edge next in an effort to complete the picture here.  


~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX
The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”
Three Controllers to Rule Them All (that just doesn’t have the same ring to it, does it?)
Beyond Centralization: The Local Control Plane
Transport Zones, Logical Switchies, and Overlays! Oh, My!

Transport Zones, Logical Switches, and Overlays! Oh, My!

Logical Switching

Everyone loves Logical Swtiching, right?! The ability to spin up a Layer 2 network whenever you need is pretty darned cool. Arguably, VXLAN is groovy, too, taking an L2 frame from a VM and wrapping it up in VXLAN goodness to shoot over the underlay network. So VMware left that alone, right?

Wrong. VXLAN is yesterday’s overlay protocol. The future is here with GENEVE.

GENEVE, you say? What in the world is GENEVE? GENEVE stands for GEneric NEtwork Virtualization Encapsulation, and is still being standardised. Wanna know more? Check is out here.

Let’s summarize the draft, shall we? Every other network virtualization overlay out there (VXLAN, NVGRE, etc.) is fixed. They all do one thing. Not to disparage any of the others – they do what they do very well. But the data center is not a fixed thing. It needs flexibility. It changes, it grows, it has to support new use cases. And that’s why GENEVE was born. See, GENEVE supports all the same stuff as everything else for network virtualization – taking that important L2 frame and wrapping it up in a new set of headers to send across the underlay “backplane” to the destination TEP (Tunnel End Point) so that delivery to the destination VM can occur. What makes GENEVE powerful, though, is extensibility through a proposed set of TLV options that can be set. Did I mention that the Options field in the GENEVE header is variable-length? So all manner of things could possibly be done.

From a day-to-day perspective, the difference between GENEVE and VXLAN is negligible. GENEVE uses UDP/6081, where we’re used to VXLAN using UDP/4789. The Tunnel Endpoints are referred to as “TEPs” or “Tunnel Endpoints” rather than “VTEPs” or “VXLAN Tunnel Endpoints”. Wireshark, as a packet analyzer example, already recognizes and understands GENEVE. So you just keep doing things the way you’ve been doing them. Except that now your logical networks can span both ESXi and KVM hypervisors. See, we’re growing.

In addition to the change in encapsulation protocol, NSX-T no longer has a dependency on the vSphere Distributed Switch. This does mean a couple of things, though.

First, we no longer have to worry about those crazy long vxw-dvs-83-virtualwire-7-sid-10007-transitNetwork kinda port group names. Logical Networks show up as simply the network name, though they do have a pretty new icon signifying that they’re opaque objects to vCenter (meaning that vCenter really can’t do anything with them, but it knows they exist).

Second, KVM virtual machine attachment to a logical switch isn’t quite as easy as in vSphere. For those, we have to identify the UUID of the virtual NIC (hint: look at the VM’s XML file – “virsh dumpxml <VM domain>”, and look for “interfaceId”), and then create the Logical Port by attaching the VIF (Virtual Interface) in NSX Manager > Switching > Ports > Add, where we specify the VIF UUID we discovered. Not difficult, just a little tedious, and probably worth scripting if you’re adding a bunch of KVM virtual machines to a Logical Switch.

Finally, don’t chew up all of your physical uplinks – NSX needs a couple of them for the NSX vSwitch that’s installed when you promote the host to a Transport Node.  I have a story I’ll tell about that later in this series.

If you recall from NSX-V, a logical switch exists in the concept of a transport zone.  NSX-T has transport zones as well, and they do precisely the same thing – define a compute scope for our logical networks.  But they’re not exactly the same all around.

In NSX-T we have two different types of transport zone.  One for overlay networks, and another for VLAN-backed networks. In other words, I have the option to build VLAN-backed logical switches.  That sounds a bit crazy, if you ask me.  Why in the world would we want to do that?! Well, the big reason is northbound connectivity to physical routers from the Edge.  I know, we haven’t talked about the Edge yet (that’s coming really soon, now that we’re talking about things in the data plane), so I’ll keep this brief.  The Edge will be configured with one or more VLAN transport zones to connect to upstream VLANs.  I’ll use this analogy again, but it’s essentially like creating VLAN-backed port groups on an NSX vSwitch.

So there you have it, a quick summary of what’s different in Logical Switching.

Beyond Centralization: The Local Control Plane

We tend to place our focus on the centralized management and control plane, but they are distributed, existing in both a centralized component (the NSX Manager for the management plane, the multi-node controller cluster for the Centra Control Plane), and a distributed component in the Management Plane Agent (MPA) and Local Control Plane (LCP) agents installed on the nodes.

The MPA communicates with NSX Manager over a RabbitMQ channel, and has a couple of purposes:
• Bootstrapping the Transport Node
• Forwarding statistics to NSX Manager

It also works with a service, nsxa, that brokers any communications to and from the kernel.

The LCP is simply the host-local control plane agent – netcpa. We should be familiar with netcpa from NSX-V. We use a proprietary protocol to communicate between netcpa and the controller nodes that uses TCP/1234. On KVM hosts, netcpa is paired with the nsx-agent service to cover the local control plane duties, including programming OVS and ConnTrack for L2, L3, and DFW services.

The LCP is bootstrapped by the Managment Plane Agent (MPA), and is responsible for programming the data plane, as we need a user space broker to communicate to the kernel modules installed on the hypervisor.

The LCP is installed as part of the host preparation process when we add Fabric Nodes to NSX Manager, and is responsible for L2 and L3 control data – things like VNIs, VTEP, MAC, and ARP tables, etc. The LCP programs the DFW as well, and6  also contains VIF (Virtual Interface) status and other such information.

These are critical to the healthy operation of NSX-T.


~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX
The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”
Three Controllers to Rule Them All (that just doesn’t have the same ring to it, does it?)


Three Controllers to Rule Them All (that just doesn’t have the same ring to it, does it?)

So we’ve got an NSX Manager now, and the uber-cool nsxcli, so what’s next? We need to build out the control plane, which means it’s NSX Controller time.

We can do this the easy way, or the hard way. We’ll talk about both. But first, let’s think about NSX and what the Controllers do.

The Control Plane is broken up into two main pieces: the Central Control Plane (CCP) and the Local Control Plane (LCP). The Control Plane programs the data plane and maintains the current, or realized state of the network.

That sounds rather similar to NSX-V controller nodes, doesn’t it? That’s because it maintains the same general purpose of the controllers in NSX-V. Except it does more. For example, when DFW rules are published, they are published from NSX Manager to the NSX CCP, and the CCP is then responsible sending the rules to the Local Control Plane for data plane programming.

Another difference is that the interaction between NSX Manager and the controller nodes is that they communicate via a RabbitMQ message bus, not private APIs.

So far, so good. Let’s think about another change: Controller sizing. In NSX-V, each controller had 4 vCPUs, 4 GB of RAM, and (according to my lab) a 28 GB virtual disk. NSX-T, again, provides some sizing flexibility:

  • Small: 2 vCPU, 8 GB RAM, 120 GB disk
  • Medium: 4 vCPU, 16 GB RAM, 120 GB disk
  • Large: 8 vCPU, 32 GB RAM, 120 GB disk.

Why would I use the different sizes, you might ask? Well, Small is generally for labs or proof of concept deployments. Generally, Medium is a good starting point. And Large, well, how big of a shop are you in? You might need it, though.

If you’re looking at your home lab or other small lab environment, you can get away with a single controller, just like NSX-V. Just don’t blame me when something doesn’t work.

For all the general similarities between the Controllers in NSX-V and NSX-T, T introduces some pretty significant under-the-hood changes that have enabled some pretty nice modularity in the Controllers themselves.

So how do we get these things up and running? I mentioned an easy way and a hard way earlier.

The easy way is to just have NSX Manager deploy the entire Controller cluster for you.

In the NSX Manager web UI, go to System > Components, and click on the “Add Controllers” link. NSX Manager will deploy the controller nodes and configure them for you based on your inputs. The downsides to this are that it only works with vCenter, and it only deploys Medium-sized Controllers. You’ve got to go to Fabric > Compute Managers and add your target vCenter server so that NSX Manager has someplace to deploy these virtual appliances. Great for quick deployment, terrible for a small lab environment.

So we have to look at things the hard way for a number of other use cases. Proof of concept, KVM-only shops, big shops that need Large controllers, etc.

To deploy your controller nodes manually, just grab the OVF (or .qcow2, if you’re in a KVM shop) and deploy it. If you’re deploying the OVF, you provide the same kinds of information you provided for the NSX Manager deployment – admin, auditor, and root passwors, network identity, etc. There are some options to add managment plane configuration, as well, should you choose to go that route.

Once they’re deployed, you need to join the controllers to the managment plane. Before we do that, we need the API thumbprint from NSX Manager (we collected that here).

Then it’s a simple matter of running the “join managment-plane” command and passing it the IP address of NSX Manager, the admin username and password, and the API thumbprint. On all of the controllers, we should set the controller cluster security model. Right now, the only option is a shared secret. We run the “set control-cluster security-model shared-secret secret” command to do that.

On the first controller, run the “initialize control-cluster” command to, well, initialize the controller cluster.

From the remaining two controllers, get the certificate thumbprint (“get control-cluster certificate thumbprint” on the controller node). With that information, we head back to the first controller (where we initialized the cluster), and join the other two nodes with the “join control-cluster” command, passing the target controller IP address and thumbprint.

We need to go back to the second and third controllers (one at a time) and run the “activate control-cluster” command. All this work, and we finally have our managment and control planes set up and ready to go. How about that?!

We’re almost ready to build logical network constructs. Almost.


~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX
The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”

The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”

As we start thinking about NSX Manager, we need to think about the CLI. There’s a lot of stuff we might do there. Configuration, for example. Or Controller Cluster creation. Or other information gathering for troubleshooting.

The nsxcli is nicely organized, and ported across devices, so you get a similar (not identical) set of CLI tooling wether you’re at the Manager CLI, Controller, Edge, ESXi, or Linux. The tooling implemented in nsxcli is context sensetive, so things like “get controller-cluster status” don’t exist on the Manager.

Another fantastic thing about nsxcli is that it’s tab-completable. So I can start a command, hit <tab> <tab> (yep, twice), and a list of suggestions pops up. And if I get stuck later in the command, I can do it again.

The nsxcli is structured pretty simply: VERB NOUN. Sort of like Powershell. The pieces of the command are space-separated, rather than hyphen separated. But there’s more than just verbs and nouns. Let’s take a quick look:

nsxmgr-01> <Tab>Tab>

  clear       Clear setting

  copy        Copy from one file to another

  del         Delete configuration

  detach      Detach from NSX cluster

  display     Display packet capture file

  exit        Exit from current mode

  get         Retrieve the current configuration

  help        Display help

  list        List all available commands

  nslookup    Name server lookup

  on          Run Central CLI command

  ping        Send echo messages

  reboot      Reboot system

  restart     Restart service

  resume      Resume node upgrade

  set         Change the current configuration

  shutdown    Shutdown system

  start       Start service

  stop        Stop service

  traceroute  Trace route to destination hostname or IP address

  verify      Verify upgrade bundle

When you get your suggestions, you even get  some nice descriptions to get started. What really starts getting interesting is when we look at the get (and set) commands.

nsxmgr-01> get <Tab><Tab>

  all                 All items

  arp-table           ARP entries

  auth-policy         Authentication policy

  capture             Packet capture

  certificate         X509 certificate

  cli-timeout         CLI timeout

  clock               Manage the system clock

  configuration       Configuration details

  cpu-stats           CPU statisticsa

  eula                End User License Agreement

  file                File

  files               Files

  filesystem-stats    Filesystem statistics

  hardening-policy    Hardening Policy

  hostname            System’s network name

  interface           Interface configuration

  interfaces          Interface status and configuration

  log-file            Log file

  logging-servers     Syslog logging servers

  management-cluster  Management cluster

  memory-stats        Memory statistics

  name-servers        Name servers

  network-stats       Show system network stats

  node                Node

  nodes               Nodes

  ntp-server          NTP server

  ntp-servers         NTP servers

  processes           System processes

  route               IP routing table

  routes              IP routing table

  search-domains      DNS search domains

  service             Node service

  services            Node services

  sockets             Open IP sockets

  support-bundle      Support bundle

  upgrade-bundle      Node Upgrade bundle

  uptime              Show system uptime information

  user                Configure system passwords

  version             System version


nsxmgr-01> set <Tab><Tab>

  auth-policy       Authentication policy

  banner            Login banner

  cli-timeout       CLI timeout

  eula              End User License Agreement

  hardening-policy  Hardening Policy

  hostname          System’s network name

  logging-server    Syslog logging server

  name-servers      Name servers

  ntp-server        NTP server

  route             IP routing table

  search-domains    DNS search domains

  service           Node service

  snmp              SNMP service

  timezone          Timezone

  user              Configure system passwords

This is where we do most of the work with NSX Manager at the CLI. For example, we’ll need the API certificate thumbprint to join nodes to the management plane. That’s pretty easy:

nsxmgr-01> get certificate api thumbprint


We can get the status of the managment plane (and a little bit of info on the control plane as well)

nsxmgr-01> get management-cluster status

Number of nodes in management cluster: 1

–     (UUID 4c832d42-2dbb-3e12-2174-ef514037e38e) Online


Management cluster status: STABLE


Number of nodes in control cluster: 3

–     (UUID 5aeb415e-8dd5-40d2-aec4-2ab96dfaac68)

–     (UUID 8c2404e7-2503-497f-9c29-4fc8c4b0b2cb)

–     (UUID fd1a94fb-833e-4533-9e6f-b4c324f7f495)


Control cluster status: STABLE

I recommend spending some time exploring what we have here. It’s a rather powerful set of tools that are very easily accessible.

But the Manager is not the only NSX component we can interact with here.  That’s right, we’ve implemented a centralized CLI in NSX-T!  This is not the same kind of central CLI we have with NSX-V, though, where there are a specific set of commands we can use.  This is better.  Immensely better!

In NSX-T, I essentially tell the CLI:

On <node> exec <insert nsxcli command here>

The very cool thing about this, like I said earlier, is that nsxcli exists on all of your nodes. When you log into NSX Manager, NSX Controller, or NSX Edge nodes as user “admin”, you’re using nsxcli.  When you’re logged into an ESXi host, simply type “nsxcli” (/bin/nsxcli if you’re curious about the full path).  On Linux KVM hosts, it’s in the same place.  Note that on ESXi and Linux, you need superuser privileges.

Also recall that I mentioned that it’s context-sensitive.  In other words, I don’t have “get management-cluster” from nsxcli on an ESXi host.  But I have “get logical-switches”, which isn’t available on NSX Manager.

Just like everything else in NSX-T, nodes have a UUID.  So how do I find those?  That’s easy!

nsxmgr-01> get nodes

UUID                                   Type  Display Name

8c2404e7-2503-497f-9c29-4fc8c4b0b2cb   ctl   nsxctrl-03

5aeb415e-8dd5-40d2-aec4-2ab96dfaac68   ctl   nsxctrl-01

fd1a94fb-833e-4533-9e6f-b4c324f7f495   ctl   nsxctrl-02

92fcc10c-cae7-4013-8948-62bb7a1c2538   edg   edge-01

a3e9bc0a-74a4-4ab2-b886-73ae05aed11b   edg   edge-02

59291ac7-203d-4d5c-bd57-10a0496d0db9   esx   esxi-01.sd.vclass.local

1ca3279f-5f5d-4009-9318-64dfb8e8841c   esx   esxi-02.sd.vclass.local

bb84cad3-00cf-45d3-b336-aee6ce5943f2   kvm   kvm-01.sd.vclass.local

65a7e954-6312-42e8-8ac5-4b352ae01db0   kvm   kvm-02.sd.vclass.local

4c832d42-2dbb-3e12-2174-ef514037e38e   mgr   nsxmgr-01


So let’s build that out a little bit.  Here’ I’m telling NSX Manager “ On node esxi-01.sd.vclass.local, execute”, and tab completion tells me the things I can do (by the way, you can even tab complete node UUIDs!):

nsxmgr-01> on 1ca3279f-5f5d-4009-9318-64dfb8e8841c exec <Tab>Tab>

  clear     Clear setting

  detach    Detach from NSX cluster

  exit      Exit from current mode

  get       Retrieve the current configuration

  help      Display help

  join      Join NSX cluster

  list      List all available commands

  reset     Reset settings

  set       Change the current configuration

  start     Start service

  <CR>      Execute command

  |         Output modifiers

If I add the “get” verb to the command and tab complete, I’ll get all kinds of information I can gather:

nsxmgr-01> on 1ca3279f-5f5d-4009-9318-64dfb8e8841c exec get <Tab>Tab>

  bridge               Bridge

  bridges              Bridges

  capture              Packet capture

  controllers          NSX controllers

  firewall             Firewall configuration

  host-switch          Host switch

  hyperbus             HyperBus configuration

  logical-router       Logical router

  logical-routers      Logical routers

  logical-switch       Logical switch

  logical-switch-port  Logical switch port

  logical-switches     Logical switches

  maintenance-mode     Maintenance Mode

  managers             NSX managers

  node-uuid            Node UUID

  service              Node service

  version              System version

  vif                  VIF

  vswitch              vswitch

Long story short, the NSX CLI is powerful, extensive, and it might be frequently used. Spend some time with it.  Explore it!

Now that we’ve got that out of the way, let’s dive into the rest of NSX-T!

~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX