New Lab is here!

I’m a happy camper!  My new lab gear is in.

I believe we all need some kind of lab environment to play with, otherwise we just don’t learn the hands-on stuff nearly as well or as quickly.  Some employers have lab environments in which to test.  My employer is no different, but I prefer to have control over what I deploy, and when I deploy it.  That way I have no one to blame about anything but myself 🙂

That said, I was running my lab in an old Dell Precision 390 with nothing but 4 cores, 8GB of RAM, and local storage.  That was great a couple of years ago when I put it together, but now, not so much.

The new gear is actually server-grade stuff.  And reasonably inexpensive, if you ask me.

For my storage, I stumbled on a great deal on a N40L Proliant MicroServer from HP.  after repurposing some disk I had laying around the house, I had a small, reasonable storage server.  I installed a bunch of SATA disk: 3 7200 RPM 500GB spindles and a 1TB 7200 RPM spindle in the built-in drive cage.  But that wasn’t quite enough for what I had in mind.  So I bought an IcyDock 4-bay 2.5″ drive chassis for the 5.25″ bay in the MicroServer, and added an IBM M1015 SAS/SATA PCI-e RAID controller to drive the 2.5″ devices.  I had an Intel 520 Series 120GB SSD (bought for the ESXi host, but it didn’t work out) and a WD Scorpio Black 750GB drive just hanging around.  So I added another SSD and Scorpio Black so I could mirror the devices and have some redundancy.

So there’s my SAN and NAS box.  I installed FreeNAS to a 16GB USB stick, and carved up 4 ZFS pools – platinum, gold, silver, and bronze.  Creative, I know LOL.

  • Platinum is a ZFS mirror of the 2 SSDs
  • Gold is a RAID-Z set of the 3 500GB spindles
  • Silver is a ZFS mirror of the 2 Scorpio Blacks
  • Bronze is a ZFS volume on the single 1TB spindle

ZFS Volumes

I debated on swapping Gold and Silver at length, but in the end, left the layout as described.

There are two things I don’t like about this setup, and they both revolve around the networking baked into the MicroServer.

 

  1. Jumbo Frames aren’t supported by the FreeBSD driver for the onboard BroadCom NIC.  This could be fixed in the future with a driver update or the official release of FreeNAS 8.2 (I’m running beta 2 at the moment)
  2. There’s only one onboard NIC.  I’d have liked two NICs, but for the price, maybe I’ll add a PCI-e dual-port Intel Gig card.  That would solve both dislikes.

Platinum, Gold, and Silver are presented via the iSCSI Target on the FreeNAS box as zVol extents.  Bronze is shared via NFS/CIFS, primarily for ISO storage.

As for the ESXi host itself, well here we go:

  • ASUS RS-500A-E6/PS4 chassis
  • 2 x AMD Opteron 6128 8-core CPUs
  • 64GB of Kingston ECC RAM
  • 250GB 7200RPM spindle from the MicroServer
  • 1TB 7200RPM spindle that was recycled from the old lab gear

I chose this seemingly overpowered setup for a few reasons (yep, another bullet point list)

  • Price (the server and its constituent parts only ran me ~$2100USD)
  • Nearly pre-assembled.  I’m not one for building machines anymore
  • Capacity.  Instead of running multiple physical ESXi hosts, I chose to run my lab nested.
  • Compatibility.  This server’s Intel counterpart is on the VMware HCL.  That didn’t mean this one would work, but I felt the odds were high.  The onboard NICs are also both Intel Pro 1000s, which helps.
  • LOM was included.  This is important to me, as I don’t want/need/have tons of extra monitors/keyboards hanging around

So all the parts came in, I put them installed the disks, CPUs, and RAM, dropped an ESXi CD in the drive, booted it up, and wondered – where’s the remote console?  I hadn’t thought about that, so I jacked in a monitor and keyboard only to find that the Delete key is necessary to get into the BIOS to configure the iKVM.  Well, in my case, that posed a little bit of a problem.  See, the only wired keyboards (or wireless, for that matter) are Apple keyboards, since I recently let the last physical Windows box leave my house.  So I had to see if the iKVM pulled DHCP.  I got out iNet, my trusty Mac network scanning utility, scanned my network, and there it was – a MAC address identifying as “ASUSTek Computer, Inc”.  That had to be it, so I fired up a web browser and plugged in the IP.  Now I just had to figure out the username and password.  Documentation to the rescue!  So I got everything configured up, and booted to the ESXi installer, and there you have it, one nice, 16-core 64GB of RAM ESXi host.

Host Summary

It’s doing rather well so far, I’ve got the storage attached, networking set up, and all kinds of VMs running right now, including vCenter Operations, View, SQL, vCloud Director, VMware Data Recovery, vShield Manager, a couple of Win7 desktops, and a few virtualized ESXi hosts, and this is what the box is doing:
Resource Usage
Just to reinforce the importance of Transparent Page Sharing, at the moment, this host is sharing ~17GB of RAM.
Shared Memory
Not to repeat myself, but I’m a happy camper.  I’ve got View set up, so I can work with the environment while I’m on the road, and my next step is to get vCD rolling and happy with a couple of virtualized ESXi hosts so I can start plugging away at building class-specific vApps so I can keep up with the different courses we run.
I hope this helps and perhaps even gives you some inspiration for your own lab environment.  I’m happy to answer any questions you may have about the setup, just drop me a line!

The Lion Sleeps Tonight

Well, not, really, but I couldn’t resist with the title.  I’m experiencing new things with Mac OS X Lion this weekend.  So here’s what’s going on.

I just bought a new iMac.  Sandy Bridge i7, 3.4GHz, loaded up with 16GB of RAM and a 256GB SSD.  Oh, and I have a 1TB spindle installed as well.  27 diagonal inches of beautiful Apple display all built in as well.

I bought this machine for two reasons.  First, so I could clear off any personal stuff from my work-issued MacBook Pro.  Second, this thing has enough horsepower to run a sizable chunk of my home lab, along side my old, unsupported loaded-as-i-can-make-it Dell Precision 390.  So some of my infrastructure is running on an ESXi host, and some in Fusion VMs.  I’m still working out the final details.  And I’m actually contemplating buying a new Thunderbolt Cinema display so I can have a 2nd GigE interface on the iMac.  May just be a pipe dream, but the thought is there.

This configuration will suffice until next year (probably closer to 9 months, if I have my way) when I can pick up and populate a couple of ASUS barebones 1U boxes (assuming, of course, that they or their equivalents are still available).  I’ve got them all picked out now, and assuming everything works out, I’ll have 32 AMD cores and 128GB of RAM to work with.  That should keep me happy for a few years from a compute perspective.  I can only complain about my lab for so long before I put the wheels in motion to make it happen.  I’m still trying to figure out what i want to do for the storage, though.  I’m thinking a stack of ~100GB SSDs, since they’re relatively cheap and will do me well for running VMs without worrying about tons of contention.  I don’t need much for a lab, 200 or 300GB will actually probably be ok, especially with vSphere Thin Provisioning.  I’m sure I’ll keep some spindles around for overflow, since they’re real cheap.  But anyway, I digress from the point here (not that I’ve ever been known to do that)

What I really came to talk about was Lion’s new Recovery Partition and doing a fresh Lion install on a Mac.  I’m wiping the MacBook Pro clean, as I had (just earlier this morning) a Mercury Pro SSD from Other World Computing  (for the OS and VMs) and a 750GB Scorpio Black from WD installed for my Dropbox folder, and media.  About 30 minutes ago, I ripped out the spindle, so now the laptop simply has the SSD, and the optical drive has been reinstalled.

So, hardware work out of the way, I booted into Lion’s Recovery HD (hold the “option” key while you reboot, just in case you didn’t know how to get there), and there it was – what looked like, well, like an OS X installer.  I went ahead and erased the SSD partition, and then told the installer I wanted a fresh install of OS X.  After confirming my eligibility, confirming the disk to which I wanted to install, and requesting my Mac App Store credentials, it’s now “Downloading additional components.”  I have an hour and a half to go, the installer has been doing this for ~15-20 minutes as it stands.  It appears that it really _is_ downloading Lion from Apple’s servers somewhere.  I’m connected to a relatively fat pipe, and it’s still looking like 1.5 hours.  I’m glad I connected to the Time Capsule connected to the cable modem, rather than the DSL.  yikes!

So over the course of the weekend, I’ll be resetting things on the MacBook Pro, but it’ll be nice, as I’ll be able to keep work and play (mostly) separate now, between the iMac, iPad, and new Lion install on the MacBook Pro.  I’m a happy camper.  Oh, and while I’m waiting, I get to put up more drywall in the basement office.  Things are coming together!