Deleting rogue mailbox folder permissions using PowerShell

Yesterday, I wrote a little post about analyzing your hybrid migration logs, using PowerShell. In the case I showed in that post, the large number of BadItems that were causing my move to fail, turned out to be caused by rogue permissions on mailbox folders in the users mailbox. These permissions were given to users that no longer exist in the directory, so they can not be moved to Exchange Online, causing the move to fail.

So… How do we remove these permissions? Well, with PowerShell ofcourse 😉 I wrote up a quick script that checks for rogue permissions on a given mailbox and then removes them. The script is tested only in my environment, so if you want to use or adopt it, please be careful.

First, we need to get a list of al folders in a mailbox so we can check the permissions for those folders. Unfortunately, get-mailboxfolder only works if your querying a mailbox that your the owner of. You can’t use this cmdlet as an administrator to check other people’s mailbox. But, we can use get-mailboxfolderstatistics as a workaround. We just need to make sure we only select the output we need.

This gives us a list of folder that are in the given mailbox. We can then use this list to check al those folders for any rogue permissions. If you investigate the permissions on a mailbox folder, you’ll see that the ‘User’ attribute for these rogue permissions will be the user SID, in stead of the username. As al SID’s start with ‘NT:’, we can use this to filter out the rogue permissions.

We now have a list of folders and the corresponding invalid permissions. It’s fairly easy to delete those with the remove-mailboxfolderpermission cmdlet.

So now for the cool part: putting all those puzzle pieces together to create one script. It’s fairly simple, using two foreach-loops: one to loop through all the folders for a mailbox to get the incorrect permissions, and another one to loop through all the rogue permissions to actually remove them.

The nasty part is in creating a correct list of folders to query the permissions. The list of folders form the get-mailboxfolderstatistics cmdlet contains only folder names, using a forward slash (/) to separate the folders, while the get-mailboxfolderpermission cmdlet expects the folder path to use backslashes (\), and include the name of the mailbox followed by a colon symbol (:). To work around this, I build a $folderpath variable combining the alias, a colon symbol and the folder path from get-mailboxfolderstatistics, combined with the -replace parameter to replace all forward slashes with a backslash.

To top it all off, I do some filtering in the get-mailboxfolderstatistics cmdlet to exclude some folders. These are folders (like ‘top of information store’) that will generate an error if you try to query the permissions.

The entire script then ends up looking like this:

Of course, if you like to run this in your own environment, be careful and make sure to know what your doing. If you are really, really sure it will be okay, remove the -whatif parameter from the last line and have fun.

Happy scripting!

Analyzing hybrid migration logs using PowerShell

While I’m currently working on migrating a customer from an on-prem Exchange environment to Exchange Online, in ran in to some problems with a few mailboxes.

In this case, there were three mailboxes that would fail the first (staging) sync from on-prem to ExO, due to the infamous ‘bad item limit reached’ error. So, I increased the bad item limit for these mailboxes and resubmitted the move request. After some time, the migration failed again, with the same error. The number of bad items had increased to above the limit I had set before. So, time to do some further digging. First, i’ll do a selection on the move requests to see which requests actually did fail.

I get the move requests that have a status of ‘failed’, get the statistics for those requests and load them to the variable $statistics.

Let’s see what the current amount of ‘bad items’ is for these mailboxes

An example from the output for one of the three mailboxes (please note that part of the displayname is hidden in this picture):

As you can see, I previously set the bad item limit to 700, but the migration currently encountered 788 bad items and therefore failed. I always do expect some bad items to occur during these migrations, but this sure is a lot. Where do all these errors come from? To find out, we have to take a look at the actual migration report.

Because I was looking at the third failed mailbox in my list of failed mailboxes, I’ll request the statistics for this mailbox, including the migration report.

This returns a huge wall of text, including all the errors that were encountered moving the messages. One of the last lines is the last failure recorded in the move request.

Of course, you can export this report to a text a file to go through the items to find the root cause. Personally, I find it easier to export the report to an XML-file, so I can use PowerShell to do some further digging.

With this cmdlet, I take the statistics for the given user, including the report, and export it to the given file. Next, I can import this XML-file to an object in PowerShell.

I now have the $report variable, which holds the XML-file with the migration report. I can now navigate through this report as I could with any other XML object within PowerShell. The ‘LastFailure’ entry I mentioned earlier, for example, is in fact an entry in the XML.

So, can we extract some actual info from these bad items from the report? We can. The encountered failures are located in the actual report, in the failures section.

Again, I obfuscated the folder name in this screenshot. This is just a part of the output from the above command, all encountered errors will be listed in the output.

So, let’s see if we can find some common denominator in these errors. I’d like to see all errors, but just a few properties for each error.

Because there is no index number for the entries, I add one manually. That way, I can always look up a specific error by referencing the number. As arrays start to count at zero, I do the same for my index number. For each error in the file, I then select the given index number, the timestamp, failuretype and the error message. At the end, I increase the index number with one, so the next error will have a correct index.

For the mailbox in our  example, this gives the following output:

So there you have it: it seems the mailbox has some items that probably have access rights mapped to an non-existing user. Of course, we can check this from the Exchange Management Shell. In this case, some of the errors referenced items in a subfolder of the ‘verwijderde items’ folder, which is Dutch for ‘Deleted Items’. So, i’ll get the folder permissions for this folder.

And indeed it does show a lot of non-existing, previously deleted, users.

So in this case, I can resolve the issue by removing the legacy permissions and restarting the job. You can also decide, after reviewing the report, to restart the job with the ‘BadItemLimit’ paramater increased to a number high enough the not cause the move request to fail, because these errors indicate that although the permissions will not be migrated, the items itself will be copied to Exchange Online so no data will be lost.

In conclusion, you can see why I prefer to review the errors in an Exchange hybrid migration using the export-clixml cmdlet. It is a much more convenient way to navigate around all errors and get a complete view of the issues.

Exchange Cached mode in RDS/VDI

Today I found myself, again, working with a customer that complained of poor performance in Outlook in combination with Office 365 (Exchange Online) on an environment using RDS. The main complaint is Outlook is slow when switching between folders, opening shared mailbox or using different calendars.

The reason for these issues is simple: when moving from Exchange on-prem to the cloud, the network latency between the clients and the Exchange-server increases and every glitch in the network becomes noticeable. When using ‘fat clients’, you can make sure Outlook is running in ‘cached mode’, so you work from your local .ost-file that syncs with the Exchange servers, thus eliminating most network issues. In RDS (or VDI, for that matter) setups, this isn’t always an option. So how to fix these issues?

First of all, you need to make sure you environment is set up in the best possible way. This TechNet blog can help you optimize and troubleshoot your Office 365 network connectivity. Especially the DNS-settings mentioned in the blogpost can be a culprit. Make sure you use DNS servers in the same geographical location as you RDS/VDI servers, so you get redirected to the right Exhange Online servers in the Azure network. More good pointers on optimizing network performance can be found here.

If all this is okay, you can use cached mode on your RDS/VDI environment. Default, the .ost file generated when using cached mode will be placed in the appdata\local directory, which causes it to not be included in roaming profiles or folder redirection. This way, when you log on to your RDS farm and are being redirected to a new server, your local cache will need to be rebuild. You can avoid this by redirecting the .ost file to a network share using a group policy. Please be aware, that previously this wasn’t a supported environment. However, with modern Outlook clients (specifically Outlook 2010 and above) this is supported. You can find detailed information here, but to summarize: if you use a high-speed, low latency network connection between your Outlook client and the file server, are using Server 2008 R2 or higher for your RDS/VDI and make sure there is ‘single file access’ to the ost-files (in other words: only one process (or user session) will access the ost-file at the same time), you are good to go.

Be aware that redirecting the .ost file does have some impact on search functionality in Outlook. The search is based on indexing by the Windows Search Service, which runs on the client computer. This means that when you are redirected to a different server, search may not be (completely) available within Outlook due the ost-file being re-indexed. Also, you need to plan you deployment and make sure you have enough disk space and disk I/O to serve the OST-files.

As you can see, these performance issues probably need to be fixed within the Outlook client. Until that happens, using cached mode can be a alternative to improve performance, although it does have some side-effects.

Of course, investing in an ExpressRoute connection, and realizing a low-latency connection to Office 365, could also solve (at least some of) these issues. However, if you would like to use ExpressRoute to connect to Office 365 you would need the premium plan which isn’t cheep.

For me, running cached mode in RDS/VDI with a limit of 6 months to cache offline seems to work OK, but you’ll have to learn how to live with the drawbacks.

Completing individual moverequests from a migrationbatch

While working on a hybrid migration from Exchange 2010 to Office 365, I found myself in a place where I had a migration batch of approximately 350 mailboxes that was ready to complete, but I only wanted to complete around five of them so my costumer could do some further testing. This was something that was decided later on in the migration traject, so the migration batch was already set up and had synced.

Now, when you use PowerShell, you can complete the entire migration batch with the complete-migrationbatch cmdlet. But there’s no such thing for completing individual mailboxes. With some workarounds though, you can complete individual moverequests from within a migration batch.

First, get the moverequest you’re completing:

You’ll see the status is ‘synced’, so the mailbox is ready to complete. To do this, we change some properties for the individual moverequest:

In this example, I set the ‘completeafter’ propertiy to 5 minutes. You can also provide a date/time string to already schedule the completion for a few days from now.

After this is set, we need to resume the migrationbatch so it will complete:

After a while, you can do another get-moverequest and see the status will be ‘completed’.

Exchange 2010 EMC issue resolved – so it seems

I blogged earlier about an issue when setting up a connection to Office 365 in the Exchange Management Console in Exchange 2010 (in this case, Service Pack 3, Rollup Update 12).

It seems that this issue was resolved. Despite the fact that I didn’t get any further feedback on the ticket I raised at MS Support or within the support forums thread mentioned in my earlier post, I decided to simply give it another try today… And It Just Worked™.

I didn’t change any thing on the existing configuration, just simply tried adding the connection to EMC again. At another client that encountered the same issue (but for which we didn’t raise a ticket yet), the issue seems to be resolved too, which leads to think there was a configuration change on the Office 365 side. However, we will need to wait for official feedback from Microsoft about this issue to have this confirmed.

Exchange 2010 hybrid connection in EMC fails with ‘ExchangeBuild’ error

I ran in to an issue using Exchange 2010 in a Hybrid setup.

In this case, I am running a fully patched Exchange 2010 SP3 CU12 machine, and I ran the new Hybrid Configuration Wizard. This completed successfully, so I do have a working Hybrid setup and should be able to move mailboxes cross-prem and send and receive mail on both sides.

Next step is to add the Office 365 tenant to my Exchange Management Console. To do this, I right-click the ‘Microsoft Exchange’ tree and click the ‘add new forest’ link. After naming the forest and selecting the ‘Exchange Online’ option for the remote PowerShell instance, I need to enter the credentials for my tenant. When validating, the wizard throws an error.

Exchange Error

‘The format of the Exchange object version is wrong. Parameter name: ExchangeBuild’.

After some searching, I found other people running into the same problem. Some get the error when trying to open an existing instance in EMC, some get it when trying to create a new one, like me. For those who get the error on an existing instance, some users report that deleting and re-adding the instance solves the problem. Others can’t re-add the instance after deleting it, facing the same error as me. A MSFT employee responded in the thread, stating that the problem has been found and the Exchange team is working on resolving it. For now, there isn’t a resolution.

The issue is also mentioned on support.microsoft.com, but aside from two obvious workarounds, there isn’t a solution yet.

For creating the hybrid setup, you can use the new HCW without running in to the issue. For viewing properties for users that are homed on Exchange Online in your hybrid scenario, you can use the Exchange Online console online. However, for the time being, there isn’t a supported way to modify these properties in this scenario.

Edit 3-9-2016: The issue seems to be resolved.

 

Remove calendar processing from a non-resource mailbox

Today I was working on converting an Office 365 room-mailbox to a ‘regular’ shared mailbox. The customer had some legitimate reasons to convert this mailbox, and who am I to question. 😉

The conversion itself is fairly simple, using PowerShell

This sets the mailbox to a ‘regular’ shared mailbox instead of a room mailbox (what it was). After setting some access rights to the mailbox I emailed the customer that his mailbox was changed and that he could access it.

After a few minutes, I received an email stating that he could send an email to the newly converted shared mailbox, but that the email would be moved to the trash can almost instantly. I know this is the default behavior for room mailbox where the calendar autoprocessing is enabled, so I reverted back to my Powershell window.

This shows that the calendar processing for this mailbox is still active. Apparently, this doesn’t get disabled when converting a room mailbox to a different kind of mailbox. No problem, because we can use the set-calendarprocessing to disable this feature.

Unfortunately, this returns an error:

The error is clear: you can only run this command on a resource mailbox. The solution was also clear: convert the mailbox back to a room mailbox, remove the calendar processing and convert the mailbox to a regular shared mailbox again.

Lessons learned: when converting a room mailbox with calendar processing enabled to a different type of mailbox, remove the calendar processing before converting the mailbox.

Tagging someone in your email

One of the previously announced features just showed up in our Office 365 tenant: tagging someone in an email.

It’s fairly simply: by adding someone in the body of your email using the @-sign, you automatically add them to the recipients list of the mail. Check it out:

I start off by creating a new email. I address it to my manager and start typing.

Email tag - image 1

 

 

 

 

Somewhere along the line, I decide I want to include my colleague Robert in the conversation. I add him inline using the @-sign. Outlook for the Web picks this up, and displays some possible people I want to include. I click to select the right one.

Email tag - image 2

 

 

 

 

 

After I select Robert, he gets added in the to-field of the message.

image3

 

 

 

 

 

 

As you can see, the name becomes a clickable link to his email address.

image4

 

 

 

 

 

 

And that’s it! I really like this feature; it makes sure that when you add someone in an email with a ‘call to action’, the get added to the recipient list and you are sure they receive this email, even if they weren’t in the conversation before!

 

Fingerprint readers and corporate email

Fingerprint readers are hot. Many phones now sport this nifty future helping you to unlock your phone even faster. Despite the fact that this Is pretty cool, it does bring some challenges to the admin. Ever since Exchange 2003, and all the way up to Office 365, Exchange has had the option to define a policy to secure the devices connecting through ActiveSync. You can disable camera’s or Bluetooth to protect data leakage from your organization, or define a policy that requires the user to set up a PIN on his device when connecting to his (or her) corporate email account.

There are a few options that can be set here. Simply requiring a PIN, defining the minimum length of the PIN, the time-out before requiring a PIN, and more. What you can’t define, is to allow or disallow certain security options, like fingerprint readers.

If your fingerprint-enabled device plays nice with these settings, varies per device. It turns out that Apple implemented the fingerprint reader on their iPhones to be a substitution for the PIN. When you use the fingerprint reader, the chipset will in the background just use the PIN to unlock the device.

Android phones, or at least the Samsung Galaxy ones, work differently. The fingerprint reader here is just that: a fingerprint reader. The system sees this as a different authentication option and doesn’t use the PIN function as underlying technology. When connecting to Exchange, a PIN will be enforced despite the fact that the fingerprint technology is used, because the Exchange server enforcing this policy isn’t aware of the fact that this reader is there.

Currently, the only way to work around this issue is to not require a password in the mobile device policy, which is a security risk, or to use 3rd party applications to sync your mail. Which is of course an even bigger security risk.