ESXi UNMAP not working on Replicated EqualLogic Volume

Symptoms

  • The VMware vSphere ESXi UNMAP command doesn’t release space on some or all volumes on a Dell EqualLogic SAN array running v8 firmware (may apply to other versions too). Using the following command in an SSH session to a 6.0u2 host (again, will apply to other versions):
    esxcli storage vmfs unmap –l MYVOLUMENAME
  • The volumes are VMFS5 (and always have been- they haven’t been upgraded from VMFS3).
  • Replication is enabled for the volumes that won’t rethin.

image

Cause

UNMAP doesn’t work on the EqualLogic when Replication is enabled. It doesn’t return an error to the SSH session, and the temporary rethinning file is still created, but the disk is not thinned.

Solution

Disable replication on the volume, re-thin the volume using the UNMAP command, then re-configure replication. Unfortunately this means the entire volume must be re-copied to the replication partner and this may impact bandwidth usage and replication schedules on larger volumes.

image


Advert:

Learning with Minecraft

There’s been a lot of coverage in the press about teaching with Minecraft- Microsoft even releasing an Educational version. So when the kids were set a homework project about “Super Structures” this got me thinking- let’s give Minecraft a go to supplement the project work set by the teacher. The project brief was to produce an informational poster or model on a structure of their choice, in our case The Eiffel Tower and Big Ben (or at least the tower which contains the bell by that name to be pedantic). Suitable amounts of craft paper, lolly sticks, straws, matchsticks and PVA were obtained and they set about construction, but once that was done and the glue was drying they turned to the Xbox One.

image

I’d prepared for this and we sat down and spent some time building Minecraft impressions of our Super Structures. This led to discussions about the materials to use (or at least the colour and texture), how big to make the model, and the shapes of the buildings. For example we before building tall towers we noted that in this case they both have a square base, the Eiffel Tower is made of iron and looks dark grey/black, we need to make the base wide enough in each that we have space to slope in to a point at the top but not so wide that we spend all day piling blocks up to make them tall enough, and so on.

It also led to discussions about the differences between Minecraft and real life. Gravity and other forces aren’t as much of a factor in the Minecraft world, and Augustus Pugin and Charles Barry didn’t have to design their structures to withstand marauding Creepers.

A new superflat world had been set up in advance, using Creative Mode (so players have access to unlimited resources without spending hours digging underground) and with the difficulty set to Peaceful (so that players are not distracted by Zombies crossing Westminster Bridge). To give us a bit of setting a short length of both the Thames and the Seine were included along with some trees from Champ des Mars, although with more time I might have practised some better topiary.

Minecraft Screenshot

Big Ben, The Eiffel Tower, and some trees.

Once finished, flying around the landscape allowed some screenshots of their creations to be taken which could then be printed, cropped, and glued (real old-skool Cut-and-Paste) onto the posters.

I’m not a qualified teacher, and have no idea if this will directly get them “better marks” on their homework, but it definitely sparked some conversation about the design of the structures, their location and history, and the materials used, which is really the point of the exercise. For an hour or so we managed to play in Minecraft whilst getting a bit of extra education relevant to their schoolwork- remember Kids, Learning is Fun!

VMworld Europe 2016 Social, Community, and vPeople

WP_20161017_09_39_34_Pro_LI

be_Connected – The people-networking side of VMworld is vital

VMworld is not all about announcements and sessions, some of the most valuable content is found in the social and community side of the event. It brings together over ten thousand people in the IT industry and there’s always lots to talk about and plenty to learn outside of the scheduled keynotes and breakouts. The social side isn’t just about popping out for a few vBeers courtesy of a sponsor- it’s the opportunity to meet your fellow professionals, hear what their up to, what they’ve seen at the show, what problems they are facing back at the datacentre or office, and possibly what solutions they can offer for your own issues. It makes for a week of long days, but it’s definitely worth it.

The community side of VMworld started for me even before leaving for Barcelona, as I had an impromptu meetup at London Gatwick with a couple of other vExperts (great to meet you Mark and Giuliano) who were waiting for the same flight.

vRockstar2Upon touchdown at El-Prat there was just time to check-in to the hotel and change before heading off for the opening social gatherings organised by the awesome events team that is Patrick Redknap and Marco Broeken– the Cohesity “Pre-Beer Party” at the Obama English Pub and then the annual vRockstar party held at the Hard Rock Cafe. Thanks to the vRockstar sponsorsRubrik, Nutanix, Veeam, VMUG, Hytrust, EMC Elect, Zerto, and Exelerys. These events were a great opportunity to catch up with old friends and make some new ones before the conference officially started the next day.

 

WP_20161017_11_29_36_Pro_LIMonday morning saw my first opportunity to get into the Gran Fira venue, register, and get down to the VMVillage for my first taste of the Hands-On Labs. There were prizes to be won, and I finally met Noell Grier when collecting a SocialLabs T-shirt from the Cloud Credibility booth. This area was also the scene of a “Team London VMUG” photo opportunity later that morning.

 

For me, Monday evening started with a walk down the beach to the Dell EMC reception at W Barcelona, right up the tower at the Eclipse bar. This featured VMworld themed cocktails such as the “vMojito” and an “All-Flash Martini” and stunning views across the city as the sun set. Later on in the evening I got a taxi across town to the Nutanix event at Cafe Ocana. Again, there were some regular faces and I met a whole host of new people, including Stephen Foskett of TechFieldDay fame.

WP_20161020_14_01_54_Pro_LIAs I mentioned in my session write up of the Tuesday, the day kicked off with the Keynote and as I was lucky enough to have a bloggers pass (thanks again Corey Romero and the team) I managed to get a reserved seat right down at the front. Tuesday morning was also the first opportunity to get into the Solutions Exchange. Here there’s the opportunity not only to pick up some vendor SWAG but, more importantly, find out what the latest product developments in the VMware ecosystem are.

WP_20161018_20_13_23_Moment (2)

OpenTechCast podcasters Gareth and Amit interviewing (or being photobombed by) Dave Simpson

Tuesday evening was the vExpert Reception held at L’eggs with a brief appearance by Pat Gelsinger himself. Also present were the OpenTechCast team conducting interviews.

After that drew to a close a small group of us took a trip to a local pub with London VMUG’s man on the ground, Alaric. Whilst the idea of a late-night party in a beach-front club is all well and good, the opportunity to have a chat with friends without having to shout was much appreciated.

Between sessions, much of my Wednesday was spent in the Hang Space and Bloggers area in the VMvillage including some last minute preparation for by vBrownbag presentation and some time to catch up on both my own blog posts and what others had been writing.

 



WP_20161019_18_53_26_Pro_LIFinishing up Wednesday was the VMworld Party. held in the keynote hall. This year the headline band was Australian outfit “Empire of the Sun”. Again the networking continued, the hall is large enough that even with the band or DJ in full flow at the front there is space at the other end of the arena for meet-ups and conversations.

Thursday is generally a quieter day at VMworld, and it offered the chance to do one last trawl through the Solutions Exchange- looking up those products and services that people had mentioned were worth a look at. The VMvillage got quieter and quieter as the afternoon went on as people left for the airport. An ideal time to take stock of the week’s happenings and finally get a go on that circular pool table!

WP_20161019_12_02_47_Pro_LI

Circular Pool Table in the Hang Space

VMworld Europe 2016 Day 3

Wednesday morning and VMworld continues…

Here’s a look at my sessions from another day in Barcelona.

General Session

WP_20161019_09_00_49_Pro_LIsm

The big announcements of Day 2 gave way to some more detail in the Day 3 Keynote, kicked off by Sanjay Poonen focusing on the digital transformation happening in the EUC world. VMware has 66,000 EUC customers and is promoting an any-app, any-device strategy through the Workspace One platform. Today 50% of business apps are web based, 40% are Windows Client-Server, and Mobile picks up the remainder. We saw a demo of how VMware use workspace one internally to provide access to all these services through one portal.

Sanjay was followed on stage by Ray O’Farrell who gave us further detail about the announcements in the SDDC. The vSphere 6.5 announcement was of course at the forefront, with Ray pointing out the advances that made the interface six times faster than 5.5 and gave a demo using PowerCLI which showed the realtime response of the new HTML5 client. He ran through a demo of the new VM encryption facilities in 6.5 and covered some of the advances in vRealize automation- including new support for containers.

imageNext up was Yanbing Li to talk about vSAN (lower case v, upper case SAN) which also has some advantages in the 6.5 release. First up is Direct-Connect options for 2-node deployments (think remote office/ branch office installs or setups requiring strict isolation of data). In this model the management and witness traffic is split out from the data traffic allowing for the two vSAN nodes to be linked directly together using Ethernet cables.

Secondly, vSAN 6.5 enables iSCSI support, so the storage infrastructure can now be used as a target for physical workloads. Yanbing also talked about the ongoing vSAN beta- future versions of the product are likely to offer data at-rest encryption and nested fault domains. vSAN has passed 5000 customers in 2.5 years, and now VMware hopes more affordable licensing with the offer of all-flash support on that Standard SKU.

Networking and Security was the next topic with Rajiv Ramaswami with the focus on NSX which is now giving 1700 NSX customers Micro-segmentation and Distributed Load Balancing. From my point of view NSX is continuing to gain weight as the product underpinning VMware’s SDDC- most of the presentations this week mentioned NSX in some form – and I expect this market to grow as the Private/Public/Hybrid cloud model expands.

Kit Colbert brought the Wednesday Keynote to a close with insights into container hosting and management VMware style using the Photon Platform. As with everything else here Photon is continuing to evolve and will be offering Kubernetes-as-a-service in Q4 this year. The Photon Controller and Photon OS are both open source- available for download from Github.

If you want to watch the full session yourself, check out the video here:

Day 2 Operations: A vCenter Server Administrator’s Diary [INF9128]

Adam Eckerle and Emad Younis gave this talk, catching up on what’s new in vCenter and how to keep it running smoothly after the install process has finished. I picked up some great takeaways here, and I’ve distilled my pages of notes to come up with the following highlights:

  • There are 5 web based clients for the vSphere environment in 6.5 :  vCenter Client and Web Client, the Appliance Manager UI (formerly VAMI) and so on. There is no support for the legacy Windows C# client.
  • The vCenter appliance upgrade preserves the identity of the old Windows-based vCenter so all connected applications and plugins should continue to operate. If the upgrade needs to be rolled back it’s just a case of turning off and removing the new VCSA and then powering on the old Windows Server and rejoining it to the domain (although any configuration changes made under the VCSA’s stewardship would then be lost) . Upgrades are possible from Windows vCenter 5.5 or 6.0
  • We were shown a demo of extending the disk in the vCenter appliance using LVM autogrow. Also, as of 6.5, the appliance will warn when the disk reaches 80% capacity and will auto-shutdown at 95% to prevent corruption
WP_20161019_11_25_11_Pro_LIsm

LVM Autogrow for logs on VCSA

  • The second half of the session included a whiteboarded overview of PSC migrations and topology. The linear ring topology was highlighted as being preferable over hub/spoke (limited in failover) and full mesh(too complicated) in larger (and expandable) deployments. Check out the poster for more details on choosing a PSC topology.

vBrownBag “The Amazing World of IT in Higher Education”

This time it was my turn to give a presentation, and in this quick ten minute talk at the vBrownBag stage I covered some of the unusual practices that an IT Pro might experience if dropped into a University environment and how various forms virtualisation can be used to save the day. If you want some more details check out my post and video link here.

Support Bundle taking up log file space on VCSA

Symptoms
The log file disk was nearly full on a vCenter Server Appliance instance, showing a warning message in the vCenter console and the VAMI (Login to https://vCenterHostName:5480). After a bit of investigation (and I’d recommend looking at this article by Brandon Lee and Knowledge Base 2143565 ) I found an old Support Bundle was taking up 2GB of the log disk unnecessarily and tripping the alert threshold.

Solution
SSH into the VCSA using the username root (see KB2143565) and navigate to the folder

/storage/log/vmware/vsphere-client/logbrowser/public/

And look for files named “*_vmsupport.tgz“, check the timestamp on these files and remove any old ones that are no longer required.
Hopefully the warning should clear and the health status should return to green in the vSphere Appliance Management Interface
Healthy VCSA

Photon, Photon, Photon

imageVMware’s cloud-native application stack is here, and it uses several things called “Photon”. In this quick post, I’ll have a look at what’s what in the stack and how the components of this container-optimised enterprise cloud platform fit together.

At the base level is Photon OS. This is a Linux distribution created from the ground up by VMware for this infrastructure.

The Photon Controller runs on Photon OS. This is a web-scale control plane which manages the workloads.

Photon Platform is the entire stack. This includes the Photon Controllers but also encompasses the underlying ESX Hypervisor, Network Virtualisation, and Storage underneath. Photon Platform is designed to be multi-tenant, so within one platform multiple tenants can be allocated resources using a hierarchical model and then individual tenants can divide their resources up across projects.

image

The Amazing World of IT in Higher Education

There are around 17,000 IT Professionals working in the UK Higher Education industry supporting world-leading, cutting edge research and the development of the next generation of minds. They are also stewards of the (slightly less glamorous) systems for HR, payroll, email, printing and so on.
Whilst much of this environment will be familiar to those employed in a more corporate setting, there are a number of striking differences to IT operations found elsewhere in both the public and private sectors which Higher Education straddles.
This quick talk from VMworld Europe 2016 covers some of the surprises that encountered in a HE IT organisation, and how Virtualisation can save the day.

VMworld Europe 2016 Day 2

Day 2, Tuesday, is when the real meat of VMworld starts- kicking off with the General Session (a.k.a. Keynote) at 9am. We were expecting some big announcements and weren’t disappointed. Here’s a look at some of my sessions today- these are the highlights only and more detail will follow.

 General Session

WP_20161018_09_26_00_Pro_LI

 

 

 

 

 

 

 

Ten thousand attendees from ninety-six countries packed into the keynote arena for the opening General Session. It borrowed many parts from the US event, but with a European slant and the added announcements of the new vSphere and VSAN 6.5 platforms and the AWS deal revealed only last week. Mike Clayville, VP of AWS, joned VMware’s Pat Gelsinger on stage to talk about how that deal is going to join the leaders in the private cloud with the leaders in the public cloud. General Availability of that platform will be mid-2017.

We were treated to demos of some of the new features offered through the Cloud Foundation and Cross-cloud architecture, including spinning up a vSphere cluster sat on Amazon Web Services, and vMotioning a workload between an on-premises Datacentre and the public cloud. One of the interesting features for me was “Elastic DRS”- this harnesses the flexibility of the public cloud and allows vCenter to automatically, dynamically, expand and contract the size of a cluster by adding and removing hosts on the fly as workload requirements change. This could deliver that real promise of an elastic SDDC capable of handling “bursty” loads smoothly, it’s also got the potential of landing you a big bill at the end of the month so hopefully there’s some checks and balances in the configuration.

The Power Hour: Deep Dive, DevOps, and New Features of PowerCLI [INF8092]

Always a fun experience, this session didn’t disappoint. With a star-wars-esque intro with voice over by Alan Renouf, and further sci-fi references added throughout by himself and Luc Dekens, it offered an in depth look into some of the new happenings in the PowerCLI world without making anyone’s head hurt with the complexity.  Announced this morning was the release of PowerCLI-core which offers the opportunity to run PowerCLI on a Linux (including PhotonOS) or Mac platform.

There’s also the announcement of VMware (no longer “vSphere”) PowerCLI 6.5 which is offering many new features including a switch to modules from the traditional snapins, cross-vCenter vMotion using Move-VM (even to AWS!), and improvements around Virtual Disk Management and Horizon View controls. There’s too much to mention it all here, so I suggest keeping a lookout for the session online.

What’s New with vSphere [INF8375R]

With the new vSphere 6.5 announcement this session was really full, with people sitting on the floor round the sides of the room. The session served as a jumping-off platform for the more in-depth sessions this week, but summarised the new features in Migration, User Interfaces, Encryption, ESXi lifecycle, Secure Boot and the Universal App Platform to enable containers within vSphere.

 WP_20161018_14_36_34_Pro_LI

The All New vSphere 6.5

vmworldHighlights

  1. A new version of VMware vSphere, 6.5, will be released shortly
  2. Migration/Upgrade tools from previous versions (including Windows vCenter) to new VCSA.
  3. VCSA Native High Availability
  4. VCSA Integrated VMware Update Manager
  5. Native vCenter Backup and Restore
  6. Improved Appliance Management
  7. vSphere Clients
  8. Encryption

New vSphere coming soon

VMware has bucked the trend in versioning adopted by other major software companies and decided not to call it’s new vSphere version “10” and opted for the more traditional “vSphere 6.5” to succeed version 6.0 which was originally released back in March 2015. Announced at VMworld Europe 2016 with GA to follow, vSphere 6.5 is a continuation of the product which forms the core of the Software Defined Datacentre chunk of VMware’s “Any Cloud” Cross-Cloud Architecture portfolio. A lot of work has been put into making the experience of installing and operating a vSphere virtualised environment easier; Ignoring any improvements under the hood, and just looking at what’s on the surface there’s a whole bunch of features designed to make life run smoother for the IT Professional, some of which are highlighted in this post.

The new vCenter Server Appliance is a core part to this simplicity, and VMware have answered the requirements of anyone currently sticking to the Windows-based vCenter. If you can get more features and more reliability for less cost and less effort then it’s definitely the way forwards in my opinion. Some of the features discussed here- notably Native HA and Backup/Restore- will only be available in the appliance version of vCenter.

VCSA Upgrade and Migration

 

image Again out to both simplify the life of IT Professionals and encourage vCenter Appliance adoption, VMware has put a lot of effort into creating straightforward, and comprehensive, upgrade and migration tools. As more and more operations and data are handled by vCenter it becomes more and more important that the system can be smoothly navigated from version to version with minimal human effort.

Migrations are possible from Windows vCenters running version 5.5 or 6.0, and both the embedded and external database topologies are supported. Additionally, the new vCenter will assume the identity of the old Windows vCenter so any external interfaces, scripts, and automation should continue to work post-migration.

VCSA Native High Availability

VCSA 6.5 offers a built-in high availability deployment taking away the need for any 3rd party clustering or database solutions. The appliance deploys as an active/passive pair (plus witness) which automatically sets up replication of the integrated database and required vCenter files. The basic setup option also places these nodes intelligently using DRS and SDRS technology and automatically creates the necessary affinity rules and private IP comms, keeping everything simple. For infrastructures with unique and challenging topologies, there’s still an advanced workflow that can be used.

image

Integrated VMware Update Manager

Prior to 6.5 using VUM to manage the patching of a vSphere infrastructure based on the vCenter Appliance has been, how can we put it?, “annoying”. After deploying the slick appliance it was then necessary to spin up (and license) a separate Windows VM just to handle the update system. This requirement has been removed in the new version- VUM is now integrated into the VCSA, enabled by default, and shares the same database instance. The new VUM integration also leverages the VCSA High Availability and Backup functionality.

Native vCenter Backup and Restore

Also new to the vCenter Server Appliance is integrated backup and restore functionality. A great step forward in the simplification of deploying a system this provides a built in solution to backup vCenter to an external location (SCP, SFTP, HTTPS locations for example) and then be able to recover by deploying a clean OVA and choosing the Restore option. image

 

Improved Appliance Management and Monitoring

The vCenter Server Appliance Management Interface- VAMI – has also had a makeover, with many features being added. The 6.0 version had an interface limited to changing IP and NTP settings, rebooting the appliance, and little else. 6.5 adds in built in monitoring of Network, CPU, Memory and the vPostgres database. There is also the option to configure Syslog for deeper external monitoring of the vCenter infrastructure- this allows fully verbose logs to be kept for auditing and troubleshooting processes.

image

vCenter Server Appliance 6.0 Management Interface

image

vCenter Server Appliance 6.5 Management Interface

vSphere Client(s)

Work continues to focus on delivering a fully functioned HTML5 client, but in the interim vCenter 6.5 will come shipped with a new (limited) HTML5 based “vSphere Client”- evolved from the current fling – as well as an improved flash based “vSphere Web Client”. Expect the “vSphere Client” to see continuous improvement and feature addition through the lifetime of the platform –driven through the Fling programme.

Encryption

As with the other topics here encryption in the new vSphere could easily be a post in itself (or a whole series), but to summarise the new features in this area, vSphere is now offering built-in VM encryption. The encryption happens between the VM and the storage so is invisible to the guest.

Local keys are generated within vSphere, and encrypted using keys held in an external (third-party) KMS- this would usually be managed by the IT Security team. Back in vCenter encryption is implemented through Storage Policies, so a VM can be encrypted simply by assigning the correct policy to it. Through the GUI (or API/PowerCLI) it’s possible to set  encryption covering  the Disks, the VMX/Swap files, or the whole lot on a per-VM basis. Through the API/PowerCLI it’s also possible to arrange encryption on a per-VHD level, potentially encrypting different disks on a VM with different keys.

VSAN encryption is on the way- there’s currently an ongoing beta – but will not be available in the 6.5 release. Based on the recent cadence I’d expect to see something in Spring 2017, but that’s just my speculation.

Summary

In summary, there’s lots to look for in the new vSphere release and in particular the vCenter Server Applicance. This week’s VMworld should reveal a lot more in depth into these advances.

VMworld Europe

VMworld Europe 2016 Day 1

WP_20161017_09_39_34_Pro_LIMonday is always a quieter day at VMworld Europe; the Keynotes kick the event off fully tomorrow and most of the sessions on Monday are reserved for Partners and TAM clients. However, there’s still plenty going on for us regular attendees.

 

Sunday night once again saw vRockstar provide an “unofficial” opening to the event. Thanks to the team and sponsors (Rubrik, Nutanix, Veeam, and the rest) for putting the event on, it was great once again to see faces old and new- several who I hadn’t seen in person for a whole year.

 

WP_20161017_08_15_49_Pro_LI (2)Back over at Fira Grand Via, my Monday agenda featured a couple of sessions, plus the opportunity to explore the Hang Space and Hands-On-Labs area. Here’s a quick run down of some of the content, in all of my posts this week I’m planning on giving the highlights, and linking into deeper-dive pages (or session videos where available) so you can have a closer look if something piques your interest.

 

Hands-on Lab: SPL-1703-SDC-1: VMware NSX Feature Tour

I worked through the first couple of modules of this, and intend to come back to look at the rest after the event is over. It’s aimed at people looking at NSX and starting a deployment from scratch. The first module was a click-through interactive slideshow of a basic NSX deployment, getting the components installed on a virtual infrastructure. This was extended in the second module where an existing deployment was used to work through adding a logical switch, connecting up some workloads, and checking connectivity. I’d recommend having a look if you have an interest in NSX and aren’t sure where to start. (To find the lab, visit http://labs.hol.vmware.com/HOL/catalogs/catalog/123 and search for “HOL-1703-SDC-1”)

 

Virtual Networking: Moving from Hype to Reality with Deep Traffic Analytics [MGT8486-QT]

Staying on the NSX theme, this was a quick-talk format session which formed a useful introduction both on a technical level but also helped me warm up to the more intense sessions coming later in the week. A half-hour long version of an hour-long presentation it showed how important analytics is to making sure a virtual network is correctly configured and traffic is running as it should.

 

Hands-on Introduction to VMware Cloud Foundation [ELW-1799-FEL-1]

Cloud Foundation was announced at VMworld US in August (plus an announcement of an AWS partnership last week) and this was an ideal opportunity to dive in and get acquainted with some of the new technology through a Hands-On-Lab, with the added benefit of a knowledgeable guide to take the group through it. The product “integrates vSphere, VSAN, and NSX into a single unified stack” that can be run both in a private cloud on-premises but also on a public cloud provider – or both through a elastic hybrid cloud model.

This lab skips the initial setup of the Cloud Foundation and dives straight in with a guided walkthrough of the user interface followed by a practical look at creating and destroying work domains- essentially the independent vSphere clusters carved from the stack and given to a particular task (VDI, generic compute, management etc). A simulated version of this lab is available here – http://labs.hol.vmware.com/HOL/catalogs/lab/2715 .

 

Roll on tomorrow, with the “General Session” keynote kicking off the day at 9am. I’m expecting some big announcements there so it should be good.