Category: Lab

Keep your passwords, I’m installing SSH keys to my NSX-T appliances

I’m not a fan of passwords. Too easy to mistype, too complex to remember without sticky notes (or a password manager). Too much room for error.

But SSH keys? I like ‘em a lot. Create something secure once, use ssh_copy_id to get it on the target system, and blammo! No more passwords.

Except that I can’t use ssh_copy_id to install my SSH key to an NSX-T Manager appliance. Or an Edge node. So what’s a person to do when you don’t have a complete shell to work with on the target system?

I don’t know for all systems, that’s for sure, but in the NSX CLI shell we’re presented in NSX-T, it’s pretty straight forward. All you need is your SSH public key and credentials to get to your NSX appliance. I’ve even produced a video to show you how!

It’s pretty simple, though – use the following command in nsxcli.  Piece of cake!

set user <username> ssh-keys label <label> type <type> value <SSH public key>

Nested ESXi on an NSX-T Logical Switch

TIL (Today I Learned) (ok, full disclosure – it wasn’t actually today – this post has been sitting in my drafts for a couple of weeks now) that MAC Learning on a vSwitch (distributed or NSX) doesn’t like that ESXi automagically inherits the physical MAC address of vmnic0. If you’re curious about this, prepare for a wall of text.

============

Here’s the scenario (there are two options here, but both behave the same)

1a. you have a lab running ESXi 6.7 and are using the native MAC Learning function introduced to the DVS. (see here for some great info on getting that set up)

1b. You have a lab running NSX-T and are using a non-default Mac Managment switching profile that has MAC Learning enabled.  Generally, nested ESXi is the use case for this switching profile.

2. You have deployed ESXi VMs, and attached them to a MAC Learning enabled port group

Symptom: single-MAC VMs (think a nested vCenter server, or NSX Manager, or just a Linux or Windows VM) function just fine – they can communicate with all kinds of things. Except your ESXi managment interfaces.

Extra fun bonus configuration: You have a separate VMkernel port dedicated to vMotion, with a separate subnet address, though still using the same uplink (example: Virtual Switch has 2 port groups: Managment and vMotion. vmk0 is attached to Managment PG, and vmk1 is attached to vMotion PG. Uplink vmnic0 is attached to the virtual switch. vMotion interfaces can communicate with each other just fine (ping ++netstack vmotion -I vmk1 <other host vmk1 IP>)

So, this exact scenario has been driving me bonkers. When I upgraded my lab to vSphere 6.7, I tried the native MAC Learning on my port groups. That was a mistake, as all 4 of my nested environments just evaporated. So I scrambled to undo the MAC Learning config, and went back to enabling Promiscuous Mode, Forged Transmits, and MAC Address changes on the port groups.  So much for the new features.

In this configuration, with NSX-V logical switches, my nested environments continued to work just fine, so long as I remembered to set the security options on any newly-created logical switch port groups to Accept.  So I just ran with it.

Fast forward to a couple of weeks ago. I ripped NSX-V off of the physical environment, and rebuilt it with NSX-T. That’s another adventure. Anyway, MAC Learning switching profiles are the option – there’s no editable dvPortGroup for NSX-T logical switches. So I had to figure this one out.

As you may know, when you install ESXi, it takes the MAC address of vmnic0 and assigns it to vmk0. Normally, that’s not a big deal. But when you enable MAC Learning, something goofy happens and Ethernet frames don’t get forwarded from vmnic0 all the way up to vmk0. So, here’s what I did to work around that little challenge.

I set the /Net/FollowHardwareMac flag in esxfg-advcfg to 1, from the default 0 (https://kb.vmware.com/s/article/1031111). After a reboot, this didn’t change the MAC address of vmk0, as the purpose of the FollowHardwareMac setting is to define whether the VMkernel MAC address should change when the underlying vmnic is replaced.  I didn’t replace the vmnic, so nothing should have changed.  This is not a necessary change, but I wanted to try it for completeness sake.  However, this is also useful if you install ESXi to a USB stick, and then move that USB stick to a different physical host, or clone your ESXi VM.

So I had to wipe out and recreate vmk0 from each of my hosts. I was a bit concerned about this, as my VMkernel ports were on a distributed switch. The only gotcha there is that you have to specify the dvPort ID for both deleting AND creating the new VMkernel port with esxcfg-vmknic.  That’s easily identified with esxcfg-vswitch -l.

And now, that nested environment works again. On an NSX-T logical switch.

Hopefully, those of you with home labs won’t have to spin your wheels so much if you run into this.

One Does Not Simply Walk Into Mordor. A Migrationary Tale.

So let’s talk about the migration of my homelab from NSX-V to NSX-T. For the faint of heart, this is a daunting tale.  Brought to you by the NSX vSwitch, and a limited number of physical NICs in my ESXi hosts.

Here’s the deal – a physical NIC (vmnic) in ESXi can be owned by exactly one vSwitch. That vSwitch can be a vSphere Standard Switch, a vSphere Distributed Switch, or an NSX vSwitch. But only by one of them at any given time.

So your host has to have at least one unused vmmic to be assigned to the NSX vSwitch during implementation. If you’ve got more than two attached to any given vSwitch, you’re fine, since you can move devices around without creating a single point of failure.

But what if your hosts were built with 4 pNICs, and you have 2 for infrastructure traffic (managment, vMotion, vSAN, etc) and 2 for virtual machine traffic? That’s still not so bad, as you can deploy NSX during a maintenance window, take a NIC away from VM traffic, and be prepared for the jeopardy state you’ve put yourself in.

How about a worst-case scenario – hypervisors with only a pair of pNICs installed. Maybe you have blade servers. Maybe you have rack servers with a pair of 25 Gbps or 40 Gbps NICs. That’s an easy implementation for NSX-V, since it simply leverages the vSphere Distributed Switch for its logical switches. It’s not so simple on NSX-T, since we’d have to pull a NIC to assign to the NSX vSwitch, and then what? With only two NICs, there’s no redundancy. This, in the modern data center, is sort of a party foul.

This is the situation I find myself in. Only one of my ESXi hosts has more than two available NICs. So I had to read the NSX-T docs to find out more about the ability to move VMkernel ports to Logical Switches.  This feature was introduced as API-only in NSX-T 2.1.  2.2 brings a UI element to the table.

VMkernel ports on Logical Switches? That’s crazy talk! Isn’t that going to create some kind of horrible circular dependency? Well, yes, it would if those Logical Switches were overlay switches backed by Geneve.

But you have to remember that NSX-T can support both Geneve-backed Logical Switches and VLAN-backed Logical Switches. Think about it like this: on any other vSwitch, Port Groups are created, and one of the port group policies available is VLAN Tagging. All a VLAN-backed Logical Switch really is, is a Port Group on the NSX vSwitch with the VLAN Tagging policy set. It’s no more complicated than that.

So how did I accomplish this crazy migration? Being my home lab, I had some flexibility in shutting down all of my VMs, so I powered everything down. I had to think about how to do that well, though, as my nested environments use storage from a DellEMC Unity VSA, and my DNS server is a virtual machine. As is vCenter Server, and NSX Manager for the physical infrastructure.  I temporarily dumped all of the NICs for the VMs on some distributed port groups to get everything off the logical switches.  After juggling DNS, vCenter, and NSX Manager around a bit, I unprepared my management cluster, then unprepared the compute cluster, and finally the old T5610 that I’ve kept around for some extra resources.  I then followed these instructions from vswitchzero.com to make sure everything was gone, including the vSphere Web Client plugin.  There may still be some vSphere Client plugin baggage hanging around, but I didn’t worry too much about that.

I was about to cross the point of no return.  I took a few minutes to breathe, and to review Migrating Network Interfaces from a VSS Switch to an N-VDS Switch in the NSX-T documentation, only to realize that I needed the titular vSphere Standard Switch, and not the vSphere Distributed Switch I had my endive environment running on.  So I jumped in, and built out a vSphere Standard Switch infrastructure to support my lab environment.  I’d like to say that I was awesome and whipped out a quick PowerShell or Python script to knock that out for me, but I’ll admit that I did it the old-fashioned, brute force way of clicking my way, host by host, through the vSphere Client.  Good thing I only have 5 hosts and 4 networks that live outside of NSX.  Once everything was migrated to a single standard switch (with a single uplink attached), I tore down the vDS in my lab, deleted my old NSX manager, and started deploying NSX-T 2.2.

I’d like to say the story is over there, but it’s not.  I prepared all of my hosts as Transport Nodes (can you see where this is going yet?), then migrated all of my management VMkernel ports over to a VLAN Logical Switch.  No sweat, until I went ahead and migrated the remaining uplinks to the N-VDS.  That may seem innocent enough, but I hadn’t moved my NSX Manager over to the N-VDS.  So there I am, with NSX Manager hanging off what is now an internal virtual switch, just chilling out with no one but the Controllers to talk to.  Realizing my conundrum, I tried to move the virtual machine over to the logical switch hosting my management network, only to realize that it doesn’t show up in the list of networks to which I can connect the vNIC.  So I scrambled to move the NSX Manager to a host outside my management cluster (good thing I attacked that cluster first, I suppose), and proceeded to reverse the migration process.  At this point, the two hosts in my management cluster are happily chugging along with their two uplinks attached to a vSphere Standard Switch.

Which is actually better, because my management cluster also hosts my edges, and those need to connect to both my transport network, and a VLAN network, both of which are configured on my vSwitch.

Looking back at the trouble I’d gotten myself into, it makes a ton of sense to not allow NSX Manager on an N-VDS, because who wants to be in circular dependency hell?  Some things you just have to learn the hard way.

With all that said, everything else has gone smoothly.  Mostly.  I still need to do some digging to figure out why Mac Learning isn’t doing me any good for my nested ESXi hosts on my logical switches. I’m sure I’m missing a quick API-only configuration switch or something. But routing works in and out of the nested environment (I can at least get my nested vCenter and other virtual machines that aren’t ESXi hosts.

So, smooth migration?  Not exactly.

Lessons learned? Lots.  For example, planning is crucial.  I know this.  Hell, I preach this in my classes.  But, as my RN wife keeps telling me about her time as an ER nurse, “bad things only happen to other people”.  I still tell her to be careful, because they do happen to other people, until you _are_ other people.  In my lab, I was “other people”.  Fortunately, it’s a lab.  The most I would have been out is about 2 days of manually rebuilding stuff from vCenter up if things had gone any more sideways.  But what if this was a production environment?  That was way too big a risk to take without writing up the migration plan, test plan, and testing in a lab first.  I just jumped in, because I’m an instructor, and I get to live in my technical ivory tower devoid of maintenance windows and serious consequences.

Also learned – leave your management cluster alone.  All too often, we forget that management is actually a pretty important workload category.

All in all, it’s done.  I’m glad I did it, if nothing else for the experience of doing it.  And I got to tear down an entire nested NSX-T environment, so I freed up a ton of resources that I don’t have (my hosts live in a perpetual state of red memory alerts).  NSX-T is a big workload, even in a small lab environment.

And, my favorite part of the whole migration?  I can now manage my entire lab from my Linux box.  No more flash for me. Well, mostly.  The vSphere Client in 6.7 does enough that I don’t have to fire up the Flash-based Web Client on the Mac very often at all.  I haven’t in at least 3 weeks.  And I’m ok with that. Maybe when the next version of vSphere drops, I’ll be able to uninstall Flash Player from my Mac.

Lab Network

So I’ve had a few questions about the network in my lab, since I’m teaching almost nothing but NSX these days.  So let’s talk about it for a bit.

My network is purposefully simple.  And I’ve just rebuilt pretty much everything, so it seems like a good time to document it.

At the edge of my network is a Ubiquiti Networks EdgeRouter Lite (ERL).  It deals with all of my routing inside the network, as well as routing to the outside world.  It’s a 3 interface device – one to the outside world (cable modem), one to my default VLAN and home network, and the third interface is carved into a bunch of sub-interfaces for my VLANs in the lab.

The two internal-facing interfaces are attached to a Cisco SG300-20 that I could also use for routing, but I chose to let the router deal with that.  This is where I have several VLANs set up for my different environments, and that’s all I’ve done with the Cisco switch – no IGMP Snooping, no routing, just VLANs:

  • Local Management – this is where all of my common stuff lives – the vCenter for my physical hosts, vROps, Log Insight, etc
  • Production Management – this is where my GA-versioned vESXi hosts live, along with their relevant supporting pieces – vCenter, NSX Manager, etc
  • Production NSX Control – I set this up simply to have a dedicated network for my NSX 6.1 Controllers.  These could just as easily gone into my Production Management VLAN
  • Production NSX Transport – this is here to simulate a dedicated VXLAN transport network.  Currently, this is superfluous, as NSX 6.0/6.1 VTEPs don’t deal well with VLAN tagging in a nested environment.  Not sure what that’s all about, <sarcasm>I must be running in an unsupported config </sarcasm>
  • Production Management Branch – this network provides a simulation of a remote site
  • Production NSX Transport Branch – again, simulation of a remote site, but much like the Production NSX Transport, this one’s completely superfluous at the moment.
  • I’ve got a matching set of VLANs for my non-GA environment, so that I can have a stable and unstable environments and maintain some level of isolation.  

Since my lab is completely nested, I also have VSAN and vMotion VLANs configured on my distributed switches, but they don’t map to anything in the physical network.

On the NSX side of things, well, I’m rebuilding that right now.  My thought process, since this is a lab, is to attach my outside-facing Edge VMs to the relevant Management network, depending on where I need the Edge.  This sort of flies in the face of having a dedicated Edge cluster, but hey, this is a lab 🙂  

Inside the Edge, my DLR(s?) will attach to a common Transit network, as will the inside interfaces of the Edges.  I’ll set up some OSPF areas so that the EdgeRouter Lite can advertise some networks into the Edge.  The DLR will also advertise its routes up to the Edge, which will in turn advertise back to the ERL.  This should be a pretty simple OSPF config.  I could eliminate the need for OSPF between the Edge and the ERL simply by configuring a default route, but what fun is that?

Then my workloads will attach to whatever Logical Switches I want them attached.  The sky’s the limit inside the SDN.  

For simplicity’s sake at this point, each network segment (VLAN or VXLAN) will have its own /24, though many of them could make due with a /28 or /29 pretty easily.  But I’m not strapped for IP addresses, thanks to our friend RFC 1918, so I’m not going to make things any more complicated than I need to.  

Everything works pretty well.  Sure, I run into some goofy behavior once in a while (see the VTEP VLAN tagging thing above), but this environment is entirely unsupported.  Honestly, it’s a miracle that any of this works at all, and is a galvanizing testament to what VMware software is actually capable of doing.  

Someday, maybe I’ll draw this all up.  But today is not that day.  

New Lab Server

Here I am, procrastinating on other stuff, to talk about the new lab setup.  I promised in the last video, I’d do a writeup, and I figured “no time like the present”!

So what do I have going on?  I’ve ripped out the ML370 and ASUS RS500A boxes, and replaced them with a veritable steal from the Dell Outlet.  I decided to go all-in for a nested lab, since I can’t come up with a good reason to put together a physical lab.

So I found a Precision T5610 with a pair of Xeon E5-2620v2s.  Twelve hyper threaded cores of processor get up and go.  The Scratch and Dent unit I bought had 32 GB of RAM installed, along with a 1 TB spindle.  Not a bad start.  But I had gear to work with, and needed more RAM.  

So I ordered a nice 64 GB upgrade kit from Crucial (well, 4 16 GB kits, technically, hoping it’d give me a 96 GB box to work with.  The pre-installed RAM and the Crucial RAM didn’t play too nicely together (Windows wouldn’t load from the spindle, nor would the ESXi installer start).  Boo.  So I pulled out the factory RAM, ran memtest86 for about a day, just to be safe, and am proceeding, for the moment, with 64 GB.

Still just a start, though, as I needed storage.  I had an Icy Dock 4-bay SATA chassis and an IBM M1015 SAS RAID controller in my little HP Microserver.  That box didn’t need those things anymore, so a transplant was necessary.  4 screws later, I had lots of room for 2.5” SATA drives.  I already had a pair of 120 GB Intel 520 SSDs in Icy Dock trays, and I pulled the 2 1 TB Crucial M550 SSDs from the ASUS box.  Now I have plenty of lab storage.  If I need more, I can always take the performance hit and attach something from the DS412+ (like I’ve already done with my “ISO_images” NFS share.

**EDIT** I ran into another snag.  The LSI controller and Icy Dock combination seem to not be playing nicely with the host.  The SSDs seem to randomly just drop offline periodically, which makes me sad.  It could be a power thing (these big SSDs are kind of notorious for power problems, especially in small drive chassis or NAS units), but I’m not going to push it any further.  I pulled the controller and drive cage out of the system, and I should have a SATA power splitter after UPS shows up today (I was too lazy to leave the house LOL), so I’m just going to run the SSDs off the on-boad SATA controller channels, and be happy about it.  

At this point, I thought I was ready for ESXi, but upon installation, I hit a snag.  The T5610 has an Intel Gigabit NIC.  As such, I expected no issues, but the 82579 isn’t noticed by ESXi 5.5u2 for whatever reason.  No biggie – this thing has a nice tool-less case, and less than a minute later, I had a dual-port Intel NIC installed and ready to go (82571EB if you’re curious).  One more reboot and ESXi was installing.

Everything’s pretty happy at the moment.  Here’s what it looks like now that I’ve built up all my virtual ESXi hosts:

Screenshot 2014 09 24 08 47 22

 

Screenshot 2014 09 24 08 48 53

 

Oh, and the “Remote_External” cluster is a set of ESXi virtual machines I have running in VMware Workstation on a Precision M4800 laptop.

I’ll follow up with some network details shortly, since that’s also gotten _way_ complex recently.  All in the name of scenario-based play with NSX. More fun later!

-jk

Deploying NSX Manager

Well, another day, another post.  

Ok, that may be exaggerating a bit, but I’m trying.  Again.  Still.

I’m rebuilding my lab (more to come on that later), and with the big push toward VMware NSX in my life right now, I thought I’d capture my fun and excitement in deploying everything.  Here’s Part 1, where I deploy the NSX Manager and register it with vCenter.  Nothing earth-shattering here, but it might help someone.

Still here. Or “Meandering thoughts about the lab that time forgot”

So, I do still exist.  I am still here, and look at this!  I even update once in a while.

My focus has shifted a little bit over the past few months.  I’m working a lot with NSX, and have been teaching the NSX: Install, Configure, Manage for a few months now.  Like, a lot.  Q3 has been absolutely nuts for me, especially with the announcement of NSX 6.1 at VMworld.

So, what does all that mean for me, and this blog, exactly?

Well, for the blog, it means a little bit less vSphere stuff, and a little bit more networking.  

Which leads us to “What does this mean for me?”.  Well, I’ve got a stack of Cisco gear here in the office now.  That must mean Cisco certification prep.  Someday, I’ll have a break and be able to actually do that 🙂

So, what Cisco stuff is in the pipeline?  CCNA – Routing and Switching is first.  that’s the basic stuff.  CCNA Data Center is soon to follow.  I’ve been tossing around CCDA because design is just a thing I do.  Do I pursue those to the NP/ND level?  I don’t know.  My guess is no, but I never leave anything off the table.

I’ve also got the VCP-NV on my list, and the forthcoming VCIX-NV when it launches.  I’ve taken a swing at the VCP-NV – I did that while I was out at VMworld, but I just missed the passing mark.  I was caught completely off-guard by some of the stuff in the exam.  It was my own fault, however, as I spent a grand total of zero time preparing and studying for the exam.  I even talked to the cert devs about the Exam Blueprint, that I didn’t actually read until the evening _after_ I wrote the exam.

That’ll be easy to rectify, however.  I know what I need to study to bump my score up over the pass mark.  Now I just need what everyone needs – time.

I also have some new lab gear incoming.  I caught a steal from the Dell Outlet the other day.  I have a scratch & dent T5610 with dual 6-core Xeon E5s on its way, and an additional 64 GB of RAM just showed up on my front porch today.  I’ll talk through the build I’m planning as I’m working through it.  I figure with 24 threads and 96 total GB of RAM, I should be able to run many, many nested ESXi hosts, especially with the VMware Tools fling and the MacLearn dvfilter fling for nested ESXi.  My old ASUS RS500A ran pretty well with 2 6-node clusters and a 4-node management cluster, each nested ESXi host with 8 GB of RAM each.  And the RS500A only had 64 GB of RAM.  

So I’m going to consolidate the ASUS and the old ML370 into a single, more modern (and likely more power efficient) box.  The tradeoff is that I’m giving up the iKVM and iLO capabilities of my existing hosts.  It’ll be ok.  I guess I’ll just have to look at IP KVMs now.  

In other news, I’ve moved my edge to higher-end gear.  Nope, nothing so fancy or cost-prohibitive as Cisco, but definitely powerful.  I picked up a Ubiquiti Networks EdgeRouter Lite the other day, and it’s pretty slick.  I’ve got a complementary UniFi AC on the way, and that should be waiting for me when I get home.  The EdgeRouter is pretty slick.  EdgeOS is based on Vyatta Core 6.3 (just before the Brocade acquisition), so I’m familiar, in passing, with the OS.  This thing is fast!

So this will all end up in a home network / lab series as I do more cool stuff with it.

As it stands, I gotta get back to class – we’re just about done deploying Controllers…

New Lab is here!

I’m a happy camper!  My new lab gear is in.

I believe we all need some kind of lab environment to play with, otherwise we just don’t learn the hands-on stuff nearly as well or as quickly.  Some employers have lab environments in which to test.  My employer is no different, but I prefer to have control over what I deploy, and when I deploy it.  That way I have no one to blame about anything but myself 🙂

That said, I was running my lab in an old Dell Precision 390 with nothing but 4 cores, 8GB of RAM, and local storage.  That was great a couple of years ago when I put it together, but now, not so much.

The new gear is actually server-grade stuff.  And reasonably inexpensive, if you ask me.

For my storage, I stumbled on a great deal on a N40L Proliant MicroServer from HP.  after repurposing some disk I had laying around the house, I had a small, reasonable storage server.  I installed a bunch of SATA disk: 3 7200 RPM 500GB spindles and a 1TB 7200 RPM spindle in the built-in drive cage.  But that wasn’t quite enough for what I had in mind.  So I bought an IcyDock 4-bay 2.5″ drive chassis for the 5.25″ bay in the MicroServer, and added an IBM M1015 SAS/SATA PCI-e RAID controller to drive the 2.5″ devices.  I had an Intel 520 Series 120GB SSD (bought for the ESXi host, but it didn’t work out) and a WD Scorpio Black 750GB drive just hanging around.  So I added another SSD and Scorpio Black so I could mirror the devices and have some redundancy.

So there’s my SAN and NAS box.  I installed FreeNAS to a 16GB USB stick, and carved up 4 ZFS pools – platinum, gold, silver, and bronze.  Creative, I know LOL.

  • Platinum is a ZFS mirror of the 2 SSDs
  • Gold is a RAID-Z set of the 3 500GB spindles
  • Silver is a ZFS mirror of the 2 Scorpio Blacks
  • Bronze is a ZFS volume on the single 1TB spindle

ZFS Volumes

I debated on swapping Gold and Silver at length, but in the end, left the layout as described.

There are two things I don’t like about this setup, and they both revolve around the networking baked into the MicroServer.

 

  1. Jumbo Frames aren’t supported by the FreeBSD driver for the onboard BroadCom NIC.  This could be fixed in the future with a driver update or the official release of FreeNAS 8.2 (I’m running beta 2 at the moment)
  2. There’s only one onboard NIC.  I’d have liked two NICs, but for the price, maybe I’ll add a PCI-e dual-port Intel Gig card.  That would solve both dislikes.

Platinum, Gold, and Silver are presented via the iSCSI Target on the FreeNAS box as zVol extents.  Bronze is shared via NFS/CIFS, primarily for ISO storage.

As for the ESXi host itself, well here we go:

  • ASUS RS-500A-E6/PS4 chassis
  • 2 x AMD Opteron 6128 8-core CPUs
  • 64GB of Kingston ECC RAM
  • 250GB 7200RPM spindle from the MicroServer
  • 1TB 7200RPM spindle that was recycled from the old lab gear

I chose this seemingly overpowered setup for a few reasons (yep, another bullet point list)

  • Price (the server and its constituent parts only ran me ~$2100USD)
  • Nearly pre-assembled.  I’m not one for building machines anymore
  • Capacity.  Instead of running multiple physical ESXi hosts, I chose to run my lab nested.
  • Compatibility.  This server’s Intel counterpart is on the VMware HCL.  That didn’t mean this one would work, but I felt the odds were high.  The onboard NICs are also both Intel Pro 1000s, which helps.
  • LOM was included.  This is important to me, as I don’t want/need/have tons of extra monitors/keyboards hanging around

So all the parts came in, I put them installed the disks, CPUs, and RAM, dropped an ESXi CD in the drive, booted it up, and wondered – where’s the remote console?  I hadn’t thought about that, so I jacked in a monitor and keyboard only to find that the Delete key is necessary to get into the BIOS to configure the iKVM.  Well, in my case, that posed a little bit of a problem.  See, the only wired keyboards (or wireless, for that matter) are Apple keyboards, since I recently let the last physical Windows box leave my house.  So I had to see if the iKVM pulled DHCP.  I got out iNet, my trusty Mac network scanning utility, scanned my network, and there it was – a MAC address identifying as “ASUSTek Computer, Inc”.  That had to be it, so I fired up a web browser and plugged in the IP.  Now I just had to figure out the username and password.  Documentation to the rescue!  So I got everything configured up, and booted to the ESXi installer, and there you have it, one nice, 16-core 64GB of RAM ESXi host.

Host Summary

It’s doing rather well so far, I’ve got the storage attached, networking set up, and all kinds of VMs running right now, including vCenter Operations, View, SQL, vCloud Director, VMware Data Recovery, vShield Manager, a couple of Win7 desktops, and a few virtualized ESXi hosts, and this is what the box is doing:
Resource Usage
Just to reinforce the importance of Transparent Page Sharing, at the moment, this host is sharing ~17GB of RAM.
Shared Memory
Not to repeat myself, but I’m a happy camper.  I’ve got View set up, so I can work with the environment while I’m on the road, and my next step is to get vCD rolling and happy with a couple of virtualized ESXi hosts so I can start plugging away at building class-specific vApps so I can keep up with the different courses we run.
I hope this helps and perhaps even gives you some inspiration for your own lab environment.  I’m happy to answer any questions you may have about the setup, just drop me a line!

The Lion Sleeps Tonight

Well, not, really, but I couldn’t resist with the title.  I’m experiencing new things with Mac OS X Lion this weekend.  So here’s what’s going on.

I just bought a new iMac.  Sandy Bridge i7, 3.4GHz, loaded up with 16GB of RAM and a 256GB SSD.  Oh, and I have a 1TB spindle installed as well.  27 diagonal inches of beautiful Apple display all built in as well.

I bought this machine for two reasons.  First, so I could clear off any personal stuff from my work-issued MacBook Pro.  Second, this thing has enough horsepower to run a sizable chunk of my home lab, along side my old, unsupported loaded-as-i-can-make-it Dell Precision 390.  So some of my infrastructure is running on an ESXi host, and some in Fusion VMs.  I’m still working out the final details.  And I’m actually contemplating buying a new Thunderbolt Cinema display so I can have a 2nd GigE interface on the iMac.  May just be a pipe dream, but the thought is there.

This configuration will suffice until next year (probably closer to 9 months, if I have my way) when I can pick up and populate a couple of ASUS barebones 1U boxes (assuming, of course, that they or their equivalents are still available).  I’ve got them all picked out now, and assuming everything works out, I’ll have 32 AMD cores and 128GB of RAM to work with.  That should keep me happy for a few years from a compute perspective.  I can only complain about my lab for so long before I put the wheels in motion to make it happen.  I’m still trying to figure out what i want to do for the storage, though.  I’m thinking a stack of ~100GB SSDs, since they’re relatively cheap and will do me well for running VMs without worrying about tons of contention.  I don’t need much for a lab, 200 or 300GB will actually probably be ok, especially with vSphere Thin Provisioning.  I’m sure I’ll keep some spindles around for overflow, since they’re real cheap.  But anyway, I digress from the point here (not that I’ve ever been known to do that)

What I really came to talk about was Lion’s new Recovery Partition and doing a fresh Lion install on a Mac.  I’m wiping the MacBook Pro clean, as I had (just earlier this morning) a Mercury Pro SSD from Other World Computing  (for the OS and VMs) and a 750GB Scorpio Black from WD installed for my Dropbox folder, and media.  About 30 minutes ago, I ripped out the spindle, so now the laptop simply has the SSD, and the optical drive has been reinstalled.

So, hardware work out of the way, I booted into Lion’s Recovery HD (hold the “option” key while you reboot, just in case you didn’t know how to get there), and there it was – what looked like, well, like an OS X installer.  I went ahead and erased the SSD partition, and then told the installer I wanted a fresh install of OS X.  After confirming my eligibility, confirming the disk to which I wanted to install, and requesting my Mac App Store credentials, it’s now “Downloading additional components.”  I have an hour and a half to go, the installer has been doing this for ~15-20 minutes as it stands.  It appears that it really _is_ downloading Lion from Apple’s servers somewhere.  I’m connected to a relatively fat pipe, and it’s still looking like 1.5 hours.  I’m glad I connected to the Time Capsule connected to the cable modem, rather than the DSL.  yikes!

So over the course of the weekend, I’ll be resetting things on the MacBook Pro, but it’ll be nice, as I’ll be able to keep work and play (mostly) separate now, between the iMac, iPad, and new Lion install on the MacBook Pro.  I’m a happy camper.  Oh, and while I’m waiting, I get to put up more drywall in the basement office.  Things are coming together!