SharePoint 2013 Slowing to a Crawl Due to Quick Launch

This morning we ran into a really weird issue with SharePoint 2013.  Out of nowhere, our intranet SharePoint portal would not load.  It just timed out.  After two minutes, you would see the "Sorry, something went wrong" text and a correlation ID.  When I looked up that ID in the logs folder, I saw that /default.aspx was timing out. 

My first thought is that it was a misbehaving web part, so I tried to load the web part editing page with the ?contents=1 in the URL trick.  That also timed out.  I did some research and found you can extend the timeout by modifying executionTimeout on the httpRuntime tag in the web.config.  I set it to 24 hours.  Nothing was going to time out now!

Well, after reloading the page and waiting about five minutes, the page did finally finish loading.  But the quick launch just said "error."  I couldn't find any information on the internet or in the logs what was going on, but at least now I had it narrowed down to (probably) the Quick Launch.  We also were able to access all the sub-sites that had different Quick Launches.  The evidence was growing.

I tried to go into the site settings navigation section.  It took nearly an hour and I just gave up.  I wondered if I could peek into the Quick Launch with a SQL query.  It was at this time that my colleague informed me that the other day he noticed entries duplicating in the Quick Launch any time one user published a particular document.  He didn't know why and since it was just a visual issue and not pressing, he left it alone.  Could this be a clue?

Well, I poked around in the SharePoint contentdb for the site.  Surprisingly, the Quick Launch is actually somewhat simple to query.  You can SELECT * from the NavNodes table and there you go.  When I did this, there were over 3500 rows.  That's a lot for a navigation bar, but if you think about, it's not an obscenely high number.  But who knows what sort of joins this table has with other tables as a matter of course in SharePoint operation.  I scrolled through the results and one entry kept repeating.  Yup, the same one my colleague had seen the other day.  Only instead of repeating twice or four times, it was repeating thousands of times.

This sure seems suspect.  An entry repeated over and over in the Quick Launch and the Quick Launch generating an error after timing out the page?  That could be the culprit.  But how do we get rid of these bad entries?  Deleting directly from the database is not a Microsoft supported scenario (at least not without their technicians performing the deletion).  Time was of the essence as the site was down.  So I took a backup of the contentdb and threw caution to the wind and deleted all the rows that matched the repeated entry's criteria.

It was time to load the website.  About five to ten tense seconds later, the site loaded.  Even better, every reload after that took no time at all!  We fixed it!

One strange issue after the page loaded was that some of the repeated entries were still in the Quick Launch, but only about 30 this time.  They were easy to remove from the Quick Launch using the UI.  I suspect they were left over from either a cache or some other table is involved in constructing the Quick Launch (maybe the most recently visited pages list?).

Anyway, now everything seems to be back to normal.  Carry on, SharePoint, you crazy beast.


Enrolling an Out-of-Date Lync Phone Edition Phone With SHA-2 Signed SSL Certificates

Recently there has been an industry-wide push to phase out SHA-1 signed SSL certificates in favor of SHA-2.  This is a good thing from a security perspective, but presents an interesting problem for Microsoft Lync deployments that use Lync Phone Edition (Aries) phones as common area phones.  In order for Lync Phone Edition phones (henceforth, "Lync phones") to connect to Lync servers after transitioning to SHA-2, the firmware must be up-to-date.  Older firmwares did not support SHA-2 and the phone would simply not be able to log into Lync if you tried.  I had trouble finding the exact firmware version where SHA-2 support was added, but I do know that 4.0.7577.4444 and newer work.

Well, this is fine for all your phones already connected to your Lync deployment.  You simply use one of the numerous guides on the internet to push out firmware updates before you switch to a SHA-2 certificate.  But what about if you buy a phone with an older firmware after transitioning to SHA-2?  The phone has to connect to Lync to get a firmware update and it can't connect to Lync until it has the firmware update.  This is a classic chicken/egg problem.  Unfortunately, Microsoft, in their infinite wisdom, provides no means to sideload new firmware to a Lync phone.

Fortunately, there is an easy solution if your phone has a USB connector.  Connect the phone via USB to any computer with the Lync desktop application installed.  Log into your Lync account when prompted.  The Lync desktop application does all the heavy lifting here, so the phone gets all the configuration information it needs without having to connect to the pesky SHA-2 SSL web services on its own.  Now, you just have to wait for the phone (and computer too, I suppose) to be idle for a while.  I suggest doing this towards the end of the day or before a long lunch break.  It will eventually install the firmware update.  You can check that it's installed by looking at the System Information section in the phone's menu.  Now sign out of the phone.  Your phone is ready to be deployed!

I have tested this method successfully with PolyCom CX3000 phones, but the method should work on any Aries-series phone with USB connectivity.  There may be some really old firmwares (such as the phones from the MD5-era certificate devices) where this won't work.  I don't know as I haven't tested.  And, sadly, if you are trying to enroll a PolyCom CX500 or other device without a USB connection, you are still out of luck.  :-(


RD Tabs: Increased Memory Consumption with 2012+ Servers and Error 3334

I've noticed increased reports of error 3334, which RD Tabs shows as an unknown disconnection code due to protocol failure.  I did some research and testing and found that it's actually due to memory pressure.  It's most noticeable on 32-bit version of RD Tabs, because you only have about 1.2GB of memory to work with before you run into .Net Framework overhead limits (actual process limit is 2GB).  Future RD Tabs versions will have a better error code description explaining what is happening.  This is not limited to RD Tabs, but all remote desktop clients.  You can read more about this error code here.

So what's going on here?

I ran some tests and when you connect to a 2012 or newer server, the per-tab memory consumption is about 150MB.  For a 2008 server, it's about 20MB.  That's a significant increase!  Even more interesting, if you let a tab stay idle for a while, the memory consumption eventually starts to decrease.  So, it seems that although when you first connect memory usage may be very high, it also may settle down with time.  If you work actively in the tab, the consumption may not go down, so it's dependent upon usage patterns.

This works out so that if you open about seven 2012+ tabs within a fairly short period using the 32-bit RD Tabs, you hit that limit and get error 3334.  I don't know about you, but for me that's not a lot of tabs!  Based on this alone, if you have a 64-bit computer, use the 64-bit version of RD Tabs.  The 32-bit version is just crippling yourself.

Messing around with the various experience settings I found that it's due to bitmap caching (see this link).  It's pretty easy to see that the graphical system of the Remote Desktop Protocol has changed a lot since 2012.  The most noticeable difference being the compression artifacts that are visible when there is a lot of on-screen animation.  My guess is that the bitmap cache feature has changed, using a lot more RAM presumably to work better over slow network connections.  The side effect is that if you open a lot of simultaneous connections, you quickly run out of RAM on your system.

It is therefore my recommendation to disable bitmap caching for all connections over a speedy LAN, and use it selectively over slow WAN links depending on your situation.  You can use Batch Favorite Editing to quickly do this in RD Tabs for all your saved connections.  :-)  Future versions of RD Tabs will have bitmap caching disabled by default.  If you are using the 32-bit version of RD Tabs, you should disable bitmap caching for all connections, regardless, unless there is a very good reason to turn it on.

And I'm Back

Sorry the site has been offline for most of the last week.  I'm honestly surprised at how many contacted me to ask if Avian Waves was no more.  I know it's been a while since the last RD Tabs release, but it's still on my radar.  :-)  And now the website is back.  So what happened?  Well, I was late to pay my hosting provider (whoops).  Email from them went deep into a sorted folder in my email and, well, let's just say I didn't notice the bill emails until somebody emailed me that my site was down!  I was looking at moving to a new host anyway because my old server was getting long in the tooth.  I could have resolved this quicker, but the bill was for a whole year, so I ended up going to a new host and I'm getting twice the RAM for a little bit less than I was paying before.  Awesome!

For some reason, some of the CSS and images in the website theme aren't working unless you log in first.  I don't know why yet, but this is why you test your backups before you experience an outage.  I should practice what I preach on my personal projects that I do in my day job.


Find All Users in Active Directory in a Specific OU Created on or after a Specific Date with PowerShell

Long title, short post!  Just replace the highlighted portions with your own values.

(get-aduser -SearchBase "ou=Domain Users,dc=mydomain,dc=com" -filter * -properties whencreated) | where {$_.whencreated -ge [datetime]"1/1/2014"} | sort-object whencreated | ft sAMAccountName,Name,WhenCreated -autosize


It's Not That Complicated: PowerShell and Task Scheduler

There is no end to complicated methods on the internet regarding the proper way to schedule a PowerShell task.  You see all manner of command line switches and escape sequences.  It's not that complicated!  Your task is powershell.exe and your argument is the path to the script.  If you need to quote the path because it contains spaces, use a combination of single and double quotes to parameterize the argument and then quote the path within the parameter.  When you do this, you also need to remember to precede the inner quote with an ampersand so PowerShell treats it as a script to execute and not just an output string.  It sounds more complicated than it actually is.


powershell.exe c:\tasks\mycoolscript.ps1

powershell.exe "& 'c:\my path\with spaces\mycoolscript.ps1'"


User Interface Fail: Office 2013

Man, I love the new Office 2013 in every way except the user interface decisions.  It’s so fast and works so well, kudos to the team for continuing to add useful features on such an old product.

I can’t say the same for the user interface.  There are so many usability issues, I don’t know how this passed QA.  Let me vent some, as is what people do on blogs on the internets.

1.  No window borders.  While Windows 8 continues the trend for massively thick window borders on the desktop, Office decided it would ignore the Windows team and their silly user interface guidelines, and just do no borders at all.  Talk about two extremes!  Thick borders are a waste of screen real estate certainly, but no borders are even worse because then the content of the window just blends in with the window underneath (especially since Windows 8 has no drop shadows).  Try working on a spreadsheet in a non-maximized window with other stuff open underneath.  It sucks.

2.  Does not use the standard Windows window frame or chrome.  Okay, I get that they are trying to get rid of as much chrome as possible to go for a flat aesthetic.  But they basically just gave the middle finger to Windows’ own window style in the process.  In basically every version of Windows going back to 3.0, you can set the color of the window frames.  Office says you can have any color you want, as long as it’s light gray.  So even if all your other Windows apps have blue frames, Office will be light gray.  This is particularly awesome because in Windows 8, inactive window frames are also light gray.  So your Office window frames always look inactive.  Awesome job!  Oh, there is one cue you can use to see if the window frame is active.  The title bar text and window widgets are black instead of medium gray.

3.  Everything else is white or light gray.  No really, this is true!  I’m not exaggerating.  From the aforementioned window frames, to the backgrounds behind every list, frame, email, ribbon, or anything else. It’s all exactly the same or just a smidge darker or lighter.  You can change the theme if you want a slightly different look, but it barely changes anything.  They give you three options: white, light gray, and dark gray.  Dark gray is really more light gray and light gray is really more off-white, but ehh… semantics.

6.  No distinguishing feature for folders with unread items except a light blue “count” next to it.  In previous version of Outlook, folders with unread items were bold.  This really stuck out and made it easy to visually see where unread email might be for folders you set up rules to sort.  In 2013, you get light blue folder counts and absolutely no chrome, so it’s quite hard to see at a glance which folder has unread items.

5.  Different appointment colors.  Since forever, Outlook has used the following color schemes for appointments until Outlook 2013: white for “free” appointments, blue for “busy” appointments, and purple for “out of office” appointments.  In Outlook 2013 they use white for “free” appointments, purple for “busy” appointments, and purple for “out of office” appointments.  Wait…what?  Yes.  This is true!  Now, granted, the “busy” and “out of office” purples are a slightly different shade, but it’s not immediately obvious at a glance.  At least it’s not yet for me.  Why change this?  Just more example of the Apple-like attitude of usability be damned, THIS IS ART!

6.  On Windows 8, Outlook 2013 uses the new Windows 8 “toast” notification feature instead of the old school desktop notifications.  While it’s nice there is a toast feature built into Windows now, it’s just not as useful.  You can’t change the position of the notifications – it’s always in the upper right where window widgets are, so you can’t close/minimize a window until you dismiss the notifications or move the window.  And something I personally didn’t use, but there is a lot of angry posts in various Microsoft forums that there are no flag/reply/forward buttons on the notification.

And here’s an honorable mention that’s been around for as long as I can remember:

If you select some cells in Excel – maybe you are trying to highlight something important in a big document – and then you change focus to another window and Excel becomes inactive, the selected cells will stop being highlighted.  Why, Office team, why?  Was there a giant outcry at some point that the spreadsheet must be clean of highlighted cells when it’s not in focus?! 


Targus Customer Support is Terrible

One of my “tech on the side” SMB clients has three Targus USB 3.0 docks and three Dell laptops.  Two docks work on all the laptops.  One dock does not work on any laptop.  So the dock is bad, right?  Congratulations!  You too could be tech support!  Swap it out for a new one and let’s go have a beer.

Targus says no.  They would rather pay somebody to talk me to death on the phone trying completely stupid troubleshooting techniques that in no way possibly would solve this issue BECAUSE THE ISSUE IS HARDWARE rather than just swap me out a new unit.

Sure, I understand they want to make sure I’m on the latest firmware and have my drivers up-to-date.  I also understand running some diagnostic tools.  But when hour number three rolls around for a $150 product, you have to wonder, are we using our time wisely here?  I don’t think uninstalling random applications and reinstalling the same version of drivers (again) is going to help.  Especially since other docks, of the same make and model, work fine.  Call it a hunch!

I would highly recommend avoiding Targus at all costs.


Operations Manager Failed to Access the Windows Event Log After Installing Hyper-V Management Packs

The Windows Server 2008 and 2008 R2 Hyper-V management packs for OpsMgr (aka SCOM) have a bug in them where they discover Windows Server 2012 boxes with the Hyper-V role installed.  Hyper-V has a few logs which have changed between the two versions and when the 2008 MP tries to query the server, it fails because the logs no longer exist.  This generates the alert you see in the title of this post.


There are several monitors that can cause this behavior and numerous other blogs have covered how you can override the MP and exclude your 2012 servers.  This does work most of the time.  However, I had one stubborn server where I simply could not find the object that needed to be overridden. 

Finally, it dawned on me.  The only problem is that the MP can’t find the log.  Why not just create the log and forget about overriding these stupid management packs?

It turns out that it’s actually pretty simple to do this, but the documentation is not the best, especially if you are not a developer, as it’s all buried in MSDN.

The log it’s looking for is one of the new Event Tracing for Windows logs that appear under Applications and Service Logs in Event Viewer.  You can’t create these as easily as it was to create event sources for the application log.  Fortunately, once you figure it out, it’s not too bad.  You need to create a manifest file.  This defines how logging is done for your application (in our case, a non-existent application).  All we need to do is create a “channel” that has the same name as the old Hyper-V logs.  There’s a program that comes with the Windows SDK called ECManGen.exe that does all the heavy lifting.  That’s a big download, though, so if you don’t have it, just use my manifest below.

Once you create the manifest file that defines the target log, use wevtutil.exe (which comes with Windows, thankfully) to import the manifest.  That’s it.  The log is created.  We don’t actually need to populate the log, we just need it to exist.  So this is sufficient.

Here is the manifest file.  For organizational purposes, so that these dummy logs don’t show up under the Microsoft\Windows section (we don’t want to confuse them with the real Hyper-V logs), I place them under a section titled “CompatibilityWithOpsMgrMP.”  Feel free to change this.  It’s just a logical name and doesn’t affect how this works.

<?xml version="1.0"?>
<instrumentationManifest xsi:schemaLocation="http://schemas.microsoft.com/win/2004/08/events eventman.xsd" xmlns="http://schemas.microsoft.com/win/2004/08/events" xmlns:win="http://manifests.microsoft.com/win/2004/08/windows/events" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:trace="http://schemas.microsoft.com/win/2004/08/events/trace">
<provider name="Microsoft-CompatibilityWithOpsMgrMP-Hyper-V-Network" guid="{7103BFE7-D8FC-42B2-A82B-331A24ED2C93}" symbol="Microsoft_Windows_Hyper_V_Network">
<channel name="Microsoft-Windows-Hyper-V-Network-Admin" chid="Admin" symbol="Admin" type="Admin" enabled="true">
<provider name="Microsoft-CompatibilityWithOpsMgrMP-Hyper-V-Image-Management-Service" guid="{E7748442-4EA4-40EB-9A4D-ED4D1AAFF5FC}" symbol="Microsoft_Windows_Hyper_V_Image_Management_Service">
<channel name="Microsoft-Windows-Hyper-V-Image-Management-Service-Admin" chid="Admin" symbol="Admin" type="Admin" enabled="true">

Save the above to a file called hyperv.man then type the following at the command prompt: wevtutil im hyperv.man

To uninstall it later, type: wevtutil um hyperv.man

Note: you need the manifest to uninstall it, so don’t delete that file.

After this, reset the status on the alerts, recalculate health, and you are done!


Response Groups Stop Working After Updating Lync 2010 Certificates

I renewed my Lync certificates recently and after applying them, response groups stopped working correctly.  Basically, a call would come in, the agent would attempt to answer it, and upon doing so the call would immediately disconnect.  The call would continue to ring and bounce between agents until the queue naturally timed out.

The strange thing is that the Lync logs showed nothing remarkable: no errors, no warnings.  I did a SIP trace on the Lync Logging Tool, with no luck.  On a whim, I restarted the Lync Server Response Group service and after doing that, response groups started working again!

Going back through the log, I can see where Lync complained about being unable to connect to the match maker service, with a yellow warning, about the time I changed the certificate.  There were no further errors or warnings.  After I restarted the service, I saw a flood of information messages about connecting to and updating things with the match maker service, so I’m guessing (truly, this is a guess) that might be the culprit.

Moral of the story: update Lync certificates after hours and then reboot (or at least restart all Lync services) to be safe.

Recent Comments
  1. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    kAM aCOSTA: Thanks Edward !!!
  2. Re: 3.0 is coming...
    Dave: Very Cool!
  3. Re: In VB.Net, sending output to the console from a Windows "Forms" application
    clochardM33: Glorious
  4. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Andreas Hagberg: Edward, you are the man. It solved the problem right at the first try. Many thanks for the post.
  5. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Derek: Edward, great find. +1 on the fix... Thanks!!
  6. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    milo: Thanx MIKE - this helped me: DPM 2016 setup will fail if you have SQL Server Management Studio (SSMS...
  7. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Terry: Edward, you are the Man!!!! Looked for a solution for hours, then found your post and BAM!!! it worked...
  8. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Mr. JoeM: Edward! Thank you, saved me hours of work.
  9. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Brian: Thank you so much Edward! :-)
  10. Re: DPM 2016 + SQL 2016 and "An unexpected error occurred during the installation" ID: 4387
    Tom: Thank you Edward! After beating my head against a wall for days, tried your suggestion out and lo and...