Rolling Windows and Sticky Bytes

<tagline type="clever"/>

Who Do You Trust?

BJ Jenkins’ keynote on “Transforming Backup” started out today with the Tape Terminator being deployed to planet data because of duplicate data being detected. Tape Teminator, as it turns out, was BJ Jenkins himself.

It was an interesting talk overall. I live tweeted the session and you can find that series of tweets on my twitter profile.

I’ll provide a summary of the key points here. To keep things reasonably short, I’m going to skip the customer testimonials but I would encourage you to read the live tweet session or watch the video once it’s available.

De-duplication used to be the cutting edge of backup. When talking about a backup solution, de-duplication is no longer a feature that is nice to have, it is absolutely a must-have.

EMC estimates that over 1000 of our customers are now running completely tapeless for backup.

Newly Released:

  • Data Domain DD990
  • Avamar 6.1
  • Watch for more announcements around other BRS products (NetWorker, SourceOne, DPA) in the coming weeks

Key Features of DD990:

  • Expandable up to 65PB
  • Cross-site de-duplication and replication from up to 270 remote sites
  • Up to 540 concurrent backup jobs
  • New DD OS with improved features for “multi-tenant” grids

Key Features of Avamar 6.1:

  • Support for SAP and Sybase
  • 3x faster backup and 60x faster restore than nearest competitor (specifically for VMware Image utilizing change block restore, I believe)
  • Support for Hyper-V backups with a feature set nearly identical to VMware backups (including granular, a.k.a. file level restore)

At this point, BRS CTO Stephen Manley came to the stage and talked about EMC’s vision for backup.

Backup gets focus when it does not meet the business needs of the rest of the business. For example, there may be impact to a system while in production (such as a long backup window) or if the backup software suffers a failure and data can’t be recovered in a timely manner (or perhaps even at all).

The vision is the same as last year — backup as a service. So how do we get there? Let’s start with the old backup model:

  1. A new system is deployed to host an application.
  2. After a while (or — in the worst case — an incident), somebody notices that the system is not protected and calls the system administrators.
  3. The system administrators call the backup administrators to ask for the system to be backed up.
  4. The backup administrators install client software and the data enters the backup infrastructure.
  5. When a restore needs to be done, the system administrators call the backup administrators and ask for a restore.

This involves too many phone calls and too much effort for both the system administrators and the backup administrators.

The pressures above tend to push backup toward faster backup, smaller backup windows, faster recovery and self service. System administrators, especially DBAs, want visibility into their data. Where are my primary copies? Are they healthy? What about my backup copies? Where do they live? Can they be restored?

Some problems are solvable independently but may have other consequences:

  • Giving DBAs the ability to dump their databases to disk and then sweep them with backup software solves part of the self-service problem and the visibility problem but causes sprawl and increased overhead for the DBAs.
  • De-dupe on the client with DDBoost or Avamar will reduce pressure on backup administrators for smaller windows and faster backups.
  • This pressure for smaller backup windows and faster backups also applies to the applications, leading to features such as VMware Change Block Tracking and Oracle Block Change Tracking.

So how do you solve these problems? Make the backup team a service provider.

  1. Centralize backup. Use Data Domain NFS mounts, Avamar Client, or DDBoost to get all your data where the backup administrators can control it, have visibility into it and manage it.
  2. Delegate some control of the backups (control, visibility and management) directly to system administrators.

The BRS team is driven to change the world — to transform backup. Backup is all about trust. Who do you trust to get you to “Backup as a Service”? It’s “not just a job”. Your time and your energy are the most valuable commodities. To whom are you going to trust your time and energy in the transformation of backup for you?

No Comments »

RAMP and Gamification

After the EMC “All Hands” meeting wrapped up, I had the good fortune to be with @mjbrender when he bumped into @TylerAltrup. Both of these fine gentlemen are heavily involved in the Recognition, Award and Motivation Program (RAMP) on the EMC Community Forums.

I’ve been thinking a lot about RAMP and Badgeville and how it might be possible to extend the core concepts into the support space — not so much in terms of rewards but in terms of recognition. Even though our chat was short, I think it was valuable.

The main takeaways for me were:

  1. From the Community Forums side of things, it would be (relatively) easy to link together the Badgeville profiles used for RAMP and the Service Center interface used by support to accept and work cases, make notes, etc.
  2. Service Center could be used as an additional data source for Community Forums points and missions.
  3. When there is a spike in service request (SR) volume, it can be difficult for the support team to know which cases to prioritize. Prioritization of cases is largely a manual process at the moment and it can be time consuming for the workflow managers who manage the queues. If, on the other hand, the manager on duty (MOD) could assign additional points or mission milestones to a particular service request (for example, they could assign 10 “bonus” points to an SR if a call is received on the MOD phone for a low severity case), this would encourage Support Engineers to prioritize these more valuable cases.
  4. As with anything, one has to consider how the system could be gamed. If cases were to increase in value as they aged, for example, this would encourage people to leave the case until they could receive maximum possible benefit. Fortunately, the competitive nature of the system itself should do a good job to discourage this — leaving a case for too long hoping to maximize your own personal benefit could easily backfire since everybody on the team would be striving to pick up the most valuable cases first.

I think we’ll have to have a discussion with the Powers That Be for Service Center to see if this is something they would be able to support.

1 Comment »

Ask the Expert, Visit the World

Some months ago, I watched with (I must admit) a tinge of envy as my good friend Matthew Brender ran an “Ask the Expert” session the EMC Community Forums about VNXe Front-End Networks with VMware. A week or two after his own session had wrapped up, he approached me about getting involved with Ask the Expert and I jumped at the chance. After being briefed by Mark Browne, Matt, and Sean Thulin (and a quick consultation with my boss who was incredibly enthusiastic about the idea), we set up a session about Avamar on the EMC Community Forums. Since my Avamar specialty is distinctly server-side, I decided to make the topic of the session Avamar Server Management Best Practices.

Probably the last thing I anticipated when I agreed to run the session was an invitation to EMC World so you can imagine how surprised and flattered I was when I received an invitation to join the other Ask the Expert contributors in Las Vegas for live Ask the Expert events. If you will be at EMC World, please stop by the Ask the Expert booth and say hello! I plan to be in the booth daily for a couple of hours right after breakfast. Sadly, I will not be bringing my dragon hat (it’s been de-fanged) but you should be able to pick me out.

I will also be reporting in periodically from EMC World itself with anything interesting I pick up during the events and keynotes. I’ll be focusing my time outside Ask the Expert on Backup and Recovery (I do work for Avamar Engineering, after all). I plan to update daily while I’m at the conference so if you’re interested in BRS topics from an engineering perspective, please stay tuned!

No Comments »