Timothy

Finding Stale Recovery Points in DPM 2010 Using PowerShell

I recently had a disk failure that I was using for my Microsoft Data Protection Manager 2010 (DPM) protection groups.  Unfortunately, it wasn't a recoverable error so I had to remove the disk from the pool and reallocate the affected data sources.  DPM is supposed to error out the data sources when it can't perform the backup, but I've found this is not always the case.  Only about 25% of the data sources that were on that disk ever errored out.  The rest had the happy green "OK."  Looking through the protection groups, I noticed that any data source that was no longer protected would still show the correct last recovery point from when it last succeeded (which was days ago).  When I tried to run a manual express full backup at that point, I would get an error that stated that the disk was missing and it could not perform the backup.  However, it still showed the green "OK" symbol next to the data source.

I have several hundred protected data sources and I couldn't go through them one by one, so I whipped up a PowerShell script to show me stale DPM data.  Basically, it enumerates all the data sources and compares the latest recovery point date with 24 hours ago.  If it's older than that, it outputs the protected resource so I can removed it and add it back to the protection group with a fresh volume.

The biggest gotcha I ran into is that a lot of properties returned by Get-DataSource are asynchronous, which is annoying when you are scripting.  Luckily, a TechNet blogger had a solution to that problem.  His script contains an error (a missing parenthesis), though.  I notified him so hopefully it will get fixed.  I have confirmed my script below works.

This script is not just useful for finding stale data due to a failed disk.  It could also be adapted to notify you when failed backups is close to surpassing your defined Recovery Point Objective (RPO).  DPM's internal notification system is very noisy since it's not uncommon for a backup to fail, then recover on its own very quickly.  If you manage a large DPM deployment, you are probably used to hundreds to thousands of emails awaiting you after the weekend.   I'm not using it that way yet, but I think I just might.

The output of my script looks like this:

$ds[133] System State and BMR : server1.avianwaves.com : Computer\System Protection

The $ds variable is the array that stores all the data sources used in the script.  The 133 is the index, so you can quickly query more information about the data source if you need to.  Just type "$ds[133]" at the PowerShell prompt to do so.  Immediately after the variable name is the Protection Group.  Following that is the server holding the protected resource.  Then the last part is the protected resource itself.

I hope this helps somebody out there!

# Refresh the datasource metadata. Code taken from: http://blogs.technet.com/b/dpm/archive/2010/09/11/why-good-scripts-may-start-to-fail-on-you-for-instance-with-timestamps-like-01-01-0001-00-00-00.aspx
Disconnect-DPMserver #clear object caches
$ds = @(Get-ProtectionGroup (&hostname) | foreach {Get-Datasource $_})
$ds = $ds | ?{$_} #remove blanks
$global:RXcount=0
for ($i=0; $i -lt $ds.count;$i++) { [void](Register-ObjectEvent $ds[$i] -EventName DataSourceChangedEvent -SourceIdentifier "TEV$i" -Action { $global:RXcount++}) }
# touch properties to trigger events and wait for arrival
$ds | select latestrecoverypoint  > $null #do not use [void] coz does not trigger
$begin = get-date
$m = Measure-Command { while (((Get-Date).subtract($begin).seconds -lt 30) -and ($RXcount -lt $ds.count)) {sleep -Milliseconds 100} }
if ($RXcount –lt $ds.count) { write-host “WARNING: Less events arrived [$RXcount] than expected [$($ds.count)]” }
Unregister-Event *

# Look for stale data
$staleDate = (get-date).AddDays(-1) # 24 hours old is our limit
$count = 0
foreach ($dsi in $ds)
{
  if ($dsi.LatestRecoveryPoint -lt $staleDate)
  {
    write-host "`$ds[$count] $($dsi.ProtectionGroup.FriendlyName) : $($dsi.ProductionServerName) : $($dsi.LogicalPath)"
  }
  $count ++
}

Timothy

Microsoft Data Protection Manager: Start a Consistency Check on all Inconsistent Replicas with PowerShell

I'm a big fan of MS DPM and have been using it for years.  The one place where it really suffers, though, is the management console.  Even in DPM 2010, there are very few things you can do in a batch sequence, which is a common need for DPM administrators.

From time to time, your server may get a large amount of inconsistent replicas.  This is usually due to something outside of DPM's control.  For example, one DPM server I manage is a VM.  If the host server suspends the VM, then resumes and the target server it was backing up before suspend is suddenly offline (such as during a reboot cycle from updates), you can get inconsistent replicas.  There are numerous other scenarios, but you get the drift.  When this happens, the management console forces you to right click on every single alert individually and select "Run a synchronization job with consistent check."  (You can also just wait for the next automatic consistency check interval, but that's not always ideal.)

Luckily, PowerShell provides a better way to kick off a manual consistency check.  Behold!  Here is my DPM PowerShell script that kicks off a consistency check on every inconsistent replica.  This is tested with DPM 2010, but should also work with DPM 2007.

$pg = Get-ProtectionGroup
foreach ($pgi in $pg) { $ds = get-datasource $pgi; foreach ($dsi in $ds) { if ($dsi.State -eq 'Invalid') { Start-DatasourceConsistencyCheck $dsi | out-null; $dsi.ProductionServerName + " :: " + $dsi.DisplayPath } } }

Enjoy!

Timothy

Who Will Vote Against the Interests of the Downtrodden Wealthy?

House Democrats are forcing the Republicans to admit with their vote that they just want tax breaks for the rich.  The Democrats are trying a (sneaky or awesome, depending on your political preference) procedural move to force the hand of Republicans.  Speaker-in-waiting John Boehner is turning a brighter shade of orange.   Are Democrats finally learning how to play politics the same way as Republicans?  I hope so.  I'm basically an o-bot, it seems, by Liberal Intertubez Blog Commenting Standards, but I am oh-so-very-glad to see some hardball being played after several days of the media reporting that Obama apologized for not being more accommodating to the guys who got our country into the mess it's in.

So will the Republicans vote against tax cuts because the uber wealthy aren't getting their (unfair) share?  A lot of them will, is my prediction, but we'll see.  Either way, the media headlines will emphasize the delicate fee fees of the Republicans being hurt, and not the fact that (some?) Republicans vote against tax breaks for the middle class.  Just you wait and see.  Extra credit for lazy reporting for anybody in the media refers to this bill as a "tax hike" (on the wealthy) by Democrats.

Either way, I'm just happy to see the House Dems go out in a blaze of glory.  It is hte awesome!

Timothy

My New Pet Peeve

My new pet peeve is when technology pundits that use the phrase (or a variation thereof), "designing for 1985" when talking about traditional mouse/keyboard input modalities.  This is usually said as a derogatory comment to Microsoft, but also includes other companies that have not yet jumped headfirst into the latest fad of iPad computing (if you can call running applets on a multimedia presentation device "computing.")

I think touch screens a la cell phones and slate devices have their place and can be awesome, but just, for a minute, try imagining typing a long document on an iPad compared to basically any traditional computer setup.  The very thought of it is tiring to me.  I need a nap.  My point is that mouse+kb has been successful for so long because when you know how to type (probably not many tech magazine writers know how without staring at the keyboard and pecking with their index fingers, unfortunately), the keyboard is incredibly fast for input.  And for situations that you can't use the keyboard, the mouse is still faster than touch screens.  With a mouse you can quickly jump from one part of the screen to the other and precisely zero in with just the smallest flick of your wrist.  Touch screens require your arm and hand actually extend to each of the affected areas on the screen.  When you have two or three monitors, having to touch for all direct-contact input with controls on the screen would actually be quite slow, even if it is satisfying in a "I feel like I'm on Star Trek" kind of way.

All that said, kb+mouse+touch is the ultimate win.  You get the best of all possible worlds.

Which brings me back to the original article that sparked this rant.  I completely disagree with the unimaginative Lenovo Technology Director.  Windows 7 on a slate device sounds incredibly awesome.  I would take that in a heartbeat over iOS, Android, or the other proprietary OSes.  Apparently somebody hasn't gotten the memo about all the touch work that went into Windows 7.  It does everything your precious, precious iPad does, plus it can actually run real productivity software.

Timothy

America Spoke: No Socialists Win in Election

Now that the 2010 election is over, I think it's safe to say that Americans have clearly spoken: nobody on the Socialist Party ticket won.  Clearly this means America is a center-right country.  At the same time, Obama should have used the bully pulpit more, because progressive voters are fragile creatures who would rather let their opponents win and go scorched earth than support a moderate on their own ticket.

Yea, also, to, this is all stupid and wrong.  I can say this because I have as much knowledge and credentials as most of the lip-flapping sputter coming out of the beltway media.  I posted the following comment on Balloon Juice (one of the only sane political blogs on the intertubes), but had to share it here to.

Regarding this article.

My comment…

So much analysis. So much wrong!

Everybody keeps pointing to their favorite ponies and stating, “if only Obama would have petted this pony more often and with a more vigorous stroking technique, Democrats would have done better.”

No.

Despite what the Firebaggers say, Obama, Pelosi, and Reid passed a lot of pretty great stuff. Pretty great stuff can also be scary because it’s new and different. Could the pretty great stuff have been even awesomer? As always, yes, IF Republicans weren’t such total douchebags.

I will take pretty great stuff (that had a possibility of passing and did) over super awesome great stuff that could never pass. I will also definitely take the pretty great stuff over whatever the beltline Pony of the Day is.

I see a lot of pointing to the economy. That’s historically correct and correct today too. But I’m glad Obama didn’t spend the last two years primping a new stimulus pony. It could even have had sparkles, braided hair, and an “I love you” tattoo on its leg, but it never would have passed!

I think this shows really good leadership and a true commitment to this country for Obama, Pelosi, and Reid. They walked in knowing how much they were truly fucking themselves in the next election or two, but did the stuff they did anyway, because it was the right thing to do.

Like it or not, that’s how American politics works. Good stuff happens. When it does, there’s a backlash from entrenched interests and fear. Shit, there was a major backlash against social security (there still is today)! But it’s in no danger of going away because even conservative Americans rely on it. The same will be true for much of what was passed in the last two years.

Relish in the liberalism of the last two years. It will be back. Fear can’t last forever.

Also, too, THIS.

Timothy

WTF Has Obama Done So Far?!

The winningest site for those of us on the left that understand that governing this country is more than just a bully pulpit and getting angry: http://whatthefuckhasobamadonesofar.com/

NOW GET OUT AND VOTE!

Timothy

SCHANNEL.ADM Updated

A while back I created the SCHANNEL.ADM administrative template to allow SCHANNEL ciphers and protocols to be configured in a GPO and pushed out to all servers in a domain.  Basically, in any Windows Server version before 2008, the SSL 2.0 protocol was enabled, and so were a bunch of weak ciphers, like 40-bit RC2 or 56-bit DES.

Well, recently one of the sites I manage began failing a HackerSafe test for ciphers and it seems I missed a few ciphers in my template.  Why HackerSafe only discovered this now instead of years ago is anybody's guess.  It is run by McAfee now, so I wouldn't bet the farm on their audits…  But I digress.

I updated the template on my website for download.  In addition to having  a few more ciphers, I also put in the description "(Recommend Disabled)" next to all the weak ciphers.

Remember that these values are not fully managed policy entries and if you delete your GPO, the affected server will not automatically revert to default values – you will be left to clean up the registry.

More information on enabling/disabling protocols and cipher in Windows can be found here: http://support.microsoft.com/kb/245030

Download SCHANNEL.ADM here.

Timothy

Brad Miller 2010

My congressional representative is Brad Miller.  He’s done some great work and is generally well liked by his district.  However, just like many Democrats this year, he’s facing stronger competition than in many previous years.  Luckily, his opponent is a crazy conspiracy theorist (BP Truther) and “tea party” candidate.

I went to a house party for Miller’s campaign.  This is the first time I’ve done such a thing.  I’m always pretty politically motivated, but as far as campaigning goes, I’m usually just a money donator, bumper sticker/yard sign poster, and not much more.

The house party was encouraging.  There were quite a few disappointed progressives, but they were largely positive towards Miller and more critical of Obama.  This is interesting to me because most of the media reports about Democratic turn-out point to dissatisfied Obama supporters not turning out for their local Democrats.  I can’t speak for all Democrats and the house party certainly isn’t a scientific sampling, but it was encouraging to see quite a few people turn out for Miller, despite possibly being disappointed in Obama. 

Personally, I think Obama has done a great job considering what he has going against him.  Sure, I disagree with him from time to time (such as a lot of the bipartisan wishful thinking), but I’m not sure Obama would have accomplished what he had if he had taken that position, so I give him a pass.  I guess I’m just an O-Bot, but I don’t blame him for not getting my ponies and rainbows.  I blame the obstructionist Confederate Party of Big Business (aka Republican Party).  I think we need to channel our fear and anger away from minor intraparty mistakes and quibbles and channel it against the political party that is holding up all true progress for the American people.

If you are a liberal and are feeling unmotivated to vote, just think about what it would be like to have a tea partier as your representative.  That is what is at stake.  Do you want government to work for you or against you?  There is no other choice.

Vote.

Despite the overall outlook and common beltway wisdom, I’m feeling encouraged and can’t wait to vote for Brad Miller and Elaine Marshall.

Timothy

SCVMM: Mounting Shared ISOs From The Library on the Library Host

There's a lot of documentation out there on how to mount ISOs in Microsoft System Center Virtual Machine Manager for Hyper-V hosts.  It basically involves setting up permissions on the library shares so that the Hyper-V host machines can consume the share, then setting up constrained delegation in Active Directory.  Here's a link: http://blogs.technet.com/b/dutchpts/archive/2009/02/09/hyper-v-and-scvmm-2008-mounting-iso-s-from-a-network-share.aspx?wa=wsignin1.0

And here's the error…

Error (12700)
VMM cannot complete the Hyper-V operation on the HVSERVER1.domain.com server because of the error: 'NewServerHost' failed to add device 'Microsoft Virtual CD/DVD Disk'. (Virtual machine ID 119730D6-8939-4CB9-8456-7941F6925279)

'XSIntWeb1-V': The Machine Account 'XSINC\HVSERVER1$' does not have read access to file share '\\HVSERVER1.domain.com\ISOs\en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso'. Please add this computer account to the security group of file share. Error: 'General access denied error' (0x80070005). (Virtual machine ID 119230D6-7929-4EB9-9456-6946F6925279)
(Unknown error (0x8001))

Recommended Action
Resolve the issue in Hyper-V and then try the operation again.

If you follow those steps exactly and still have issues, then there might be one more gotcha.  Are you trying to connect to a VMM library that is hosted on the same server as both the VMM server and the Hyper-V host's parent partition?  (Yes, this is a supported configuration.)

If so, you need to add the NT AUTHORITY\Network Service user account to the share and NTFS permissions.  The reason is that the CIFS (SMB) file share is local to the computer needing access.  As such, instead of the user appearing as the Hyper-V parent server machine name (such as HVSERVER1$), it will appear as the local account that the Hyper-V server is running as, which is typically the built-in Network Service account.

What makes this a 'gotcha' is that the error message explicitly states the machine account as the culprit.  I only figured this out through the power of Process Monitor by SysInternals, which clearly showed the ACCESS DENIED is occurring when Network Service hits the share.

Process Monitor screenshot

Timothy

Mystery of the Failing Meeting Appointment Updates on Exchange 2003

A very brief tech mystery today!

We have not upgraded to Exchange 2010 yet at work and are still on Exchange 2003 (yes, 2003).  That upgrade is scheduled for a little later this year, but until then, I have to continue to support 2003.

I ran into a strange mail flow issue today that ends up being the result of a combination of Exchange 2003 and Outlook 2007 or newer.  One of my users was trying to send an update to a meeting request, but recipients outside of our domain (external SMTP recipients) were getting stuck in the queue.  The additional information in the queue just said "Unable to open the message for delivery."  That clued me in that it wasn't an external SMTP rejection of the message (invalid email address or whatever).  So I did some googling.

It turns out there's actually a known issue for Outlook 2007 or Outlook 2010 (or newer) clients sending meeting updates via Exchange 2003.  It must not happen all the time, because we've had Outlook 2007 deployed for years and never saw this issue before.  In fact, it didn't start happening until a user on Outlook 2010 sent a meeting update.  Maybe it's just coincidence – I'm not really sure.

Anyway, here's the KB with all the details.  The article isn't clear, but to double check if you have this error by enabling diagnostic logging, you need to set MSExchangeTransport\Exchange Store Driver to Maximum.  Once you do that, go back to the queue, select the outbound server the stuck message is sitting in, right click and select "Force Connection."  Then you can look in the event viewer for the specific error to confirm this is your issue.

Don't forget to set the registry entry after applying the hotfix!

Blog

Search Posts

Recent Comments

  1. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    ptbNPA: That should have been *ID 810*, not 820

  2. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    ptbNPA: For anyone else coming across this in the future and have an ID 820 error: For some strange reason...

  3. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Timothy: Sorry, I don't have any other insight. I'm sure you ran into all the same articles I did about the error...

  4. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    ptbNPA: FYI, after posting the issue on technet + windows-noob.com with no replies I decided to do a fresh Windows...

  5. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    ptbNPA: Hi and thanks for your post. I originally installed SQL 2016 with SP1 on a Windows server 2016 and got...

  6. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Timothy: Update Rollup 2.

  7. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Catwiesl: You write "Microsoft System Center Data Protection Manager (DPM) officially supports SQL 2016 with U...

  8. Re: PowerShell: Quickly Finding the Oldest and Newest Files in a Folder
    Neki: i would like to compare a local file with a remote file and download the remote file if its newer than...

  9. Re: RD Tabs: Tabbed Windows Remote Desktops
    lukas: Fantastic software though I am having issues saving passwords. As soon as I kill the program the setting...

  10. Re: Easy Way to Change Permissions on the Windows Server Scheduled Tasks Folder (C:\Windows\Tasks)
    George: From another server, using an admin account, I opened \\SERVER\c$\windows in Windows Explorer. Tasks...

Archive

Tag Cloud