Month: July 2018

Beyond Centralization: The Local Control Plane

We tend to place our focus on the centralized management and control plane, but they are distributed, existing in both a centralized component (the NSX Manager for the management plane, the multi-node controller cluster for the Centra Control Plane), and a distributed component in the Management Plane Agent (MPA) and Local Control Plane (LCP) agents installed on the nodes.

The MPA communicates with NSX Manager over a RabbitMQ channel, and has a couple of purposes:
• Bootstrapping the Transport Node
• Forwarding statistics to NSX Manager

It also works with a service, nsxa, that brokers any communications to and from the kernel.

The LCP is simply the host-local control plane agent – netcpa. We should be familiar with netcpa from NSX-V. We use a proprietary protocol to communicate between netcpa and the controller nodes that uses TCP/1234. On KVM hosts, netcpa is paired with the nsx-agent service to cover the local control plane duties, including programming OVS and ConnTrack for L2, L3, and DFW services.

The LCP is bootstrapped by the Managment Plane Agent (MPA), and is responsible for programming the data plane, as we need a user space broker to communicate to the kernel modules installed on the hypervisor.

The LCP is installed as part of the host preparation process when we add Fabric Nodes to NSX Manager, and is responsible for L2 and L3 control data – things like VNIs, VTEP, MAC, and ARP tables, etc. The LCP programs the DFW as well, and6  also contains VIF (Virtual Interface) status and other such information.

These are critical to the healthy operation of NSX-T.

 

~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX
The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”
Three Controllers to Rule Them All (that just doesn’t have the same ring to it, does it?)

 

Three Controllers to Rule Them All (that just doesn’t have the same ring to it, does it?)

So we’ve got an NSX Manager now, and the uber-cool nsxcli, so what’s next? We need to build out the control plane, which means it’s NSX Controller time.

We can do this the easy way, or the hard way. We’ll talk about both. But first, let’s think about NSX and what the Controllers do.

The Control Plane is broken up into two main pieces: the Central Control Plane (CCP) and the Local Control Plane (LCP). The Control Plane programs the data plane and maintains the current, or realized state of the network.

That sounds rather similar to NSX-V controller nodes, doesn’t it? That’s because it maintains the same general purpose of the controllers in NSX-V. Except it does more. For example, when DFW rules are published, they are published from NSX Manager to the NSX CCP, and the CCP is then responsible sending the rules to the Local Control Plane for data plane programming.

Another difference is that the interaction between NSX Manager and the controller nodes is that they communicate via a RabbitMQ message bus, not private APIs.

So far, so good. Let’s think about another change: Controller sizing. In NSX-V, each controller had 4 vCPUs, 4 GB of RAM, and (according to my lab) a 28 GB virtual disk. NSX-T, again, provides some sizing flexibility:

  • Small: 2 vCPU, 8 GB RAM, 120 GB disk

  • Medium: 4 vCPU, 16 GB RAM, 120 GB disk

  • Large: 8 vCPU, 32 GB RAM, 120 GB disk.

Why would I use the different sizes, you might ask? Well, Small is generally for labs or proof of concept deployments. Generally, Medium is a good starting point. And Large, well, how big of a shop are you in? You might need it, though.

If you’re looking at your home lab or other small lab environment, you can get away with a single controller, just like NSX-V. Just don’t blame me when something doesn’t work.

For all the general similarities between the Controllers in NSX-V and NSX-T, T introduces some pretty significant under-the-hood changes that have enabled some pretty nice modularity in the Controllers themselves.

So how do we get these things up and running? I mentioned an easy way and a hard way earlier.

The easy way is to just have NSX Manager deploy the entire Controller cluster for you.

In the NSX Manager web UI, go to System > Components, and click on the “Add Controllers” link. NSX Manager will deploy the controller nodes and configure them for you based on your inputs. The downsides to this are that it only works with vCenter, and it only deploys Medium-sized Controllers. You’ve got to go to Fabric > Compute Managers and add your target vCenter server so that NSX Manager has someplace to deploy these virtual appliances. Great for quick deployment, terrible for a small lab environment.

So we have to look at things the hard way for a number of other use cases. Proof of concept, KVM-only shops, big shops that need Large controllers, etc.

To deploy your controller nodes manually, just grab the OVF (or .qcow2, if you’re in a KVM shop) and deploy it. If you’re deploying the OVF, you provide the same kinds of information you provided for the NSX Manager deployment – admin, auditor, and root passwors, network identity, etc. There are some options to add managment plane configuration, as well, should you choose to go that route.

Once they’re deployed, you need to join the controllers to the managment plane. Before we do that, we need the API thumbprint from NSX Manager (we collected that here).

Then it’s a simple matter of running the “join managment-plane” command and passing it the IP address of NSX Manager, the admin username and password, and the API thumbprint. On all of the controllers, we should set the controller cluster security model. Right now, the only option is a shared secret. We run the “set control-cluster security-model shared-secret secret” command to do that.

On the first controller, run the “initialize control-cluster” command to, well, initialize the controller cluster.

From the remaining two controllers, get the certificate thumbprint (“get control-cluster certificate thumbprint” on the controller node). With that information, we head back to the first controller (where we initialized the cluster), and join the other two nodes with the “join control-cluster” command, passing the target controller IP address and thumbprint.

We need to go back to the second and third controllers (one at a time) and run the “activate control-cluster” command. All this work, and we finally have our managment and control planes set up and ready to go. How about that?!

We’re almost ready to build logical network constructs. Almost.

 

~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX
The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”

The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”

As we start thinking about NSX Manager, we need to think about the CLI. There’s a lot of stuff we might do there. Configuration, for example. Or Controller Cluster creation. Or other information gathering for troubleshooting.

The nsxcli is nicely organized, and ported across devices, so you get a similar (not identical) set of CLI tooling wether you’re at the Manager CLI, Controller, Edge, ESXi, or Linux. The tooling implemented in nsxcli is context sensetive, so things like “get controller-cluster status” don’t exist on the Manager.

Another fantastic thing about nsxcli is that it’s tab-completable. So I can start a command, hit <tab> <tab> (yep, twice), and a list of suggestions pops up. And if I get stuck later in the command, I can do it again.

The nsxcli is structured pretty simply: VERB NOUN. Sort of like Powershell. The pieces of the command are space-separated, rather than hyphen separated. But there’s more than just verbs and nouns. Let’s take a quick look:

nsxmgr-01> <Tab>Tab>

  clear       Clear setting

  copy        Copy from one file to another

  del         Delete configuration

  detach      Detach from NSX cluster

  display     Display packet capture file

  exit        Exit from current mode

  get         Retrieve the current configuration

  help        Display help

  list        List all available commands

  nslookup    Name server lookup

  on          Run Central CLI command

  ping        Send echo messages

  reboot      Reboot system

  restart     Restart service

  resume      Resume node upgrade

  set         Change the current configuration

  shutdown    Shutdown system

  start       Start service

  stop        Stop service

  traceroute  Trace route to destination hostname or IP address

  verify      Verify upgrade bundle

When you get your suggestions, you even get  some nice descriptions to get started. What really starts getting interesting is when we look at the get (and set) commands.

nsxmgr-01> get <Tab><Tab>

  all                 All items

  arp-table           ARP entries

  auth-policy         Authentication policy

  capture             Packet capture

  certificate         X509 certificate

  cli-timeout         CLI timeout

  clock               Manage the system clock

  configuration       Configuration details

  cpu-stats           CPU statisticsa

  eula                End User License Agreement

  file                File

  files               Files

  filesystem-stats    Filesystem statistics

  hardening-policy    Hardening Policy

  hostname            System’s network name

  interface           Interface configuration

  interfaces          Interface status and configuration

  log-file            Log file

  logging-servers     Syslog logging servers

  management-cluster  Management cluster

  memory-stats        Memory statistics

  name-servers        Name servers

  network-stats       Show system network stats

  node                Node

  nodes               Nodes

  ntp-server          NTP server

  ntp-servers         NTP servers

  processes           System processes

  route               IP routing table

  routes              IP routing table

  search-domains      DNS search domains

  service             Node service

  services            Node services

  sockets             Open IP sockets

  support-bundle      Support bundle

  upgrade-bundle      Node Upgrade bundle

  uptime              Show system uptime information

  user                Configure system passwords

  version             System version

 

nsxmgr-01> set <Tab><Tab>

  auth-policy       Authentication policy

  banner            Login banner

  cli-timeout       CLI timeout

  eula              End User License Agreement

  hardening-policy  Hardening Policy

  hostname          System’s network name

  logging-server    Syslog logging server

  name-servers      Name servers

  ntp-server        NTP server

  route             IP routing table

  search-domains    DNS search domains

  service           Node service

  snmp              SNMP service

  timezone          Timezone

  user              Configure system passwords

This is where we do most of the work with NSX Manager at the CLI. For example, we’ll need the API certificate thumbprint to join nodes to the management plane. That’s pretty easy:

nsxmgr-01> get certificate api thumbprint

88710fcd3fd84686cc6cc03b22298a1f84b9784b9f49bb869e889d632b3c2b22

We can get the status of the managment plane (and a little bit of info on the control plane as well)

nsxmgr-01> get management-cluster status

Number of nodes in management cluster: 1

– 172.20.40.42     (UUID 4c832d42-2dbb-3e12-2174-ef514037e38e) Online

 

Management cluster status: STABLE

 

Number of nodes in control cluster: 3

– 172.20.40.31     (UUID 5aeb415e-8dd5-40d2-aec4-2ab96dfaac68)

– 172.20.40.33     (UUID 8c2404e7-2503-497f-9c29-4fc8c4b0b2cb)

– 172.20.40.32     (UUID fd1a94fb-833e-4533-9e6f-b4c324f7f495)

 

Control cluster status: STABLE

I recommend spending some time exploring what we have here. It’s a rather powerful set of tools that are very easily accessible.

But the Manager is not the only NSX component we can interact with here.  That’s right, we’ve implemented a centralized CLI in NSX-T!  This is not the same kind of central CLI we have with NSX-V, though, where there are a specific set of commands we can use.  This is better.  Immensely better!

In NSX-T, I essentially tell the CLI:

On <node> exec <insert nsxcli command here>

The very cool thing about this, like I said earlier, is that nsxcli exists on all of your nodes. When you log into NSX Manager, NSX Controller, or NSX Edge nodes as user “admin”, you’re using nsxcli.  When you’re logged into an ESXi host, simply type “nsxcli” (/bin/nsxcli if you’re curious about the full path).  On Linux KVM hosts, it’s in the same place.  Note that on ESXi and Linux, you need superuser privileges.

Also recall that I mentioned that it’s context-sensitive.  In other words, I don’t have “get management-cluster” from nsxcli on an ESXi host.  But I have “get logical-switches”, which isn’t available on NSX Manager.

Just like everything else in NSX-T, nodes have a UUID.  So how do I find those?  That’s easy!

nsxmgr-01> get nodes

UUID                                   Type  Display Name

8c2404e7-2503-497f-9c29-4fc8c4b0b2cb   ctl   nsxctrl-03

5aeb415e-8dd5-40d2-aec4-2ab96dfaac68   ctl   nsxctrl-01

fd1a94fb-833e-4533-9e6f-b4c324f7f495   ctl   nsxctrl-02

92fcc10c-cae7-4013-8948-62bb7a1c2538   edg   edge-01

a3e9bc0a-74a4-4ab2-b886-73ae05aed11b   edg   edge-02

59291ac7-203d-4d5c-bd57-10a0496d0db9   esx   esxi-01.sd.vclass.local

1ca3279f-5f5d-4009-9318-64dfb8e8841c   esx   esxi-02.sd.vclass.local

bb84cad3-00cf-45d3-b336-aee6ce5943f2   kvm   kvm-01.sd.vclass.local

65a7e954-6312-42e8-8ac5-4b352ae01db0   kvm   kvm-02.sd.vclass.local

4c832d42-2dbb-3e12-2174-ef514037e38e   mgr   nsxmgr-01

 

So let’s build that out a little bit.  Here’ I’m telling NSX Manager “ On node esxi-01.sd.vclass.local, execute”, and tab completion tells me the things I can do (by the way, you can even tab complete node UUIDs!):

nsxmgr-01> on 1ca3279f-5f5d-4009-9318-64dfb8e8841c exec <Tab>Tab>

  clear     Clear setting

  detach    Detach from NSX cluster

  exit      Exit from current mode

  get       Retrieve the current configuration

  help      Display help

  join      Join NSX cluster

  list      List all available commands

  reset     Reset settings

  set       Change the current configuration

  start     Start service

  <CR>      Execute command

  |         Output modifiers

If I add the “get” verb to the command and tab complete, I’ll get all kinds of information I can gather:

nsxmgr-01> on 1ca3279f-5f5d-4009-9318-64dfb8e8841c exec get <Tab>Tab>

  bridge               Bridge

  bridges              Bridges

  capture              Packet capture

  controllers          NSX controllers

  firewall             Firewall configuration

  host-switch          Host switch

  hyperbus             HyperBus configuration

  logical-router       Logical router

  logical-routers      Logical routers

  logical-switch       Logical switch

  logical-switch-port  Logical switch port

  logical-switches     Logical switches

  maintenance-mode     Maintenance Mode

  managers             NSX managers

  node-uuid            Node UUID

  service              Node service

  version              System version

  vif                  VIF

  vswitch              vswitch

Long story short, the NSX CLI is powerful, extensive, and it might be frequently used. Spend some time with it.  Explore it!

Now that we’ve got that out of the way, let’s dive into the rest of NSX-T!

~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX

NSX-T: The Manager of All Things NSX

You’ve seen it before. The monolithic NSX Manager from which all VMware SDN is spawned. The API endpoint. Provider of the UI. NSX Manager is the centerpiece of the world of NSX.

Welcome back to my adventure in moving from NSX-V to NSX-T!

NSX-T, just like NSX-V, is split into three functional planes: Managment, Control, and Data.

The Managment Plane is mostly the NSX Manager, but it also includes Managment Plane Agents on the hosts. The Managment Plane is a lot of things: my source of truth for network configuration, the persistent repository for the network state that I want, the API and UI provider, and more.

Just like in NSX-V, you deploy the NSX Manager as a virtual appliance. VMware ships the appliance in two different formats now – OVF and qcow2. You see, NSX-T is not nearly as beholden to vSphere as its cousin NSX-V. NSX-T is perfectly happy without VMware’s hypervisor and management stack. You can run happily with only RHEL or Ubuntu as your KVM platform, should you desire. This makes NSX-T a great option for those driving OpenStack for their private SDDC plaftorm.

There are so many more options in the OVF deployment now – 4 different size options (Small, Medium, Medium Large, and Large)

Small – 2 vCPU, 8 GB RAM, 140 GB disk
Medium – 4 vCPU, 16 GB RAM, 140 GB disk
Medium Large – 6 vCPU, 24 GB RAM, 140 GB disk
Large – 8 vCPU, 32 GB RAM, 140 GB disk

You get to choose your managment network (as usual), and decide whether your managment will run on IPv4 or IPv6.

3 sets of passwords for the admin, root, and audit users (yep, you have easily accessible root access here!). You can also specify different usernames for the admin and audit roles, if you don’t like the defaults.

Then you’ve got the host identity and role. Standard IP address and hostname stuff here, with the addition of the NSX role. Here, again, you have choices:

nsx-manager: This is the NSX Manager we know and love. The focal point for UI and API interaction.

nsx-policy-manager: Want to start automating security policies and the like? You need one of these, too (yep, a second appliance).

nsx-cloud-service-manager: Got NSX Cloud? Then get one of these.

nsx-manager+nsx-policy-manager – this multi-role option is only supported on VMConAWS deployments – don’t try this on-prem.

Finally, you set up your DNS configuration, NTP, and whether you want to allow SSH logins. And then you wait a minute for everything to deploy.

Once your done deploying, you can power on that bad boy of a VM. BTW, the memory is all reserved , so watch out.

Next step, logging into the web interface. Just point a browser at your NSX Manager IP or (preferably) hostname, and login with the admin credentials you just set during deployment. You’ll be presented with a beautiful Clarity-driven UI, with a dozen tiles for varying functions at the landing page.

Here, we can get into all kinds of trouble, from configuring load balancers to logical switches. But we’ve got more setup to do by deploying the Central Control Plane. We’ll get to that in another segment.

Before we get into all that, however, stay tuned for the next part in this series – I’ll take you on a tour of the NSX Manager admin CLI and show off some useful tools.

~$ history
Introduction: From NSX-V to NSX-T. An Adventure

From NSX-V to NSX-T. An Adventure.

I posted a while ago that NSX-T is the future, and the future is now.

And I entirely stand by that statement. While NSX-V is currently the software-defined networking standard at VMware, its time will come.
NSX-T is the architecture of the future. It’s the platrform for both NSX Data Center and NSX Cloud. The tooling you will use to define your networking and security capabilities and policies consistently between on-prem and off.

As it stands, today, NSX-T is really more for developer clouds. It has different capabilities than NSX-V, though the gap is shrinking dramatically with each release – NSX-T 2.2.0 can do an awful lot of cool stuff, and much of the stuff you may be accustomed to doing now with NSX-V. The proverbial “tomorrow” is close, indeed, and tomorrow, NSX-T will take the crown as king of the VMware SDN kingdom. Fortunately, this is not going to be a coup, but rather a peaceful transition (well, maybe not if you want to migrate in-place, but that’s a whole different discussion).

What I want to do in this series is to lay out the similarities and differences of the two platforms, as they stand today (NSX-V 6.4.1 and NSX-T 2.2.0). I will not cover positively everything – just what I would consider the basics. That’s still a significant list – my outline is crazy right now. Maybe it’ll become more manageable, maybe I’m just going to spend an awful lot of time writing. Hopefully, I will provide the information you need to essentially translate your NSX-V vocabulary to NSX-T.

That’s the goal. Remember, this isn’t going to be deep technical content – just a whirlwind tour through the new platform with comparisons to what you’re already familiar with.

I won’t be getting into the API, as that’s just not my wheelhouse right now – I can spell JSON, but not much more than that. So I won’t be covering cool stuff like Dashboard customizations, but there’s plenty for me to work on without that.

Join me on my journey through the wilds of NSX, here’s to hoping that we’ll both learn something!

~$ history
Introduction: From NSX-V to NSX-T. An Adventure
NSX-T: The Manager of All Things NSX
The Hall of the Mountain King. or “What Loot do We Find in nsxcli?”
Three Controllers to Rule Them All (that just doesn’t have the same ring to it, does it?)
Beyond Centralization: The Local Control Plane
Transport Zones, Logical Switchies, and Overlays! Oh, My!
Which Way Do We Go? Let’s ask the Logical Router!
If You’re Not Living on the Edge, You’re Taking Up Too Much Room
Welcome to the Edge, I’m at Your Service
Tooling and Operations