Analyzing hybrid migration logs using PowerShell

While I’m currently working on migrating a customer from an on-prem Exchange environment to Exchange Online, in ran in to some problems with a few mailboxes.

In this case, there were three mailboxes that would fail the first (staging) sync from on-prem to ExO, due to the infamous ‘bad item limit reached’ error. So, I increased the bad item limit for these mailboxes and resubmitted the move request. After some time, the migration failed again, with the same error. The number of bad items had increased to above the limit I had set before. So, time to do some further digging. First, i’ll do a selection on the move requests to see which requests actually did fail.

I get the move requests that have a status of ‘failed’, get the statistics for those requests and load them to the variable $statistics.

Let’s see what the current amount of ‘bad items’ is for these mailboxes

An example from the output for one of the three mailboxes (please note that part of the displayname is hidden in this picture):

As you can see, I previously set the bad item limit to 700, but the migration currently encountered 788 bad items and therefore failed. I always do expect some bad items to occur during these migrations, but this sure is a lot. Where do all these errors come from? To find out, we have to take a look at the actual migration report.

Because I was looking at the third failed mailbox in my list of failed mailboxes, I’ll request the statistics for this mailbox, including the migration report.

This returns a huge wall of text, including all the errors that were encountered moving the messages. One of the last lines is the last failure recorded in the move request.

Of course, you can export this report to a text a file to go through the items to find the root cause. Personally, I find it easier to export the report to an XML-file, so I can use PowerShell to do some further digging.

With this cmdlet, I take the statistics for the given user, including the report, and export it to the given file. Next, I can import this XML-file to an object in PowerShell.

I now have the $report variable, which holds the XML-file with the migration report. I can now navigate through this report as I could with any other XML object within PowerShell. The ‘LastFailure’ entry I mentioned earlier, for example, is in fact an entry in the XML.

So, can we extract some actual info from these bad items from the report? We can. The encountered failures are located in the actual report, in the failures section.

Again, I obfuscated the folder name in this screenshot. This is just a part of the output from the above command, all encountered errors will be listed in the output.

So, let’s see if we can find some common denominator in these errors. I’d like to see all errors, but just a few properties for each error.

Because there is no index number for the entries, I add one manually. That way, I can always look up a specific error by referencing the number. As arrays start to count at zero, I do the same for my index number. For each error in the file, I then select the given index number, the timestamp, failuretype and the error message. At the end, I increase the index number with one, so the next error will have a correct index.

For the mailbox in our  example, this gives the following output:

So there you have it: it seems the mailbox has some items that probably have access rights mapped to an non-existing user. Of course, we can check this from the Exchange Management Shell. In this case, some of the errors referenced items in a subfolder of the ‘verwijderde items’ folder, which is Dutch for ‘Deleted Items’. So, i’ll get the folder permissions for this folder.

And indeed it does show a lot of non-existing, previously deleted, users.

So in this case, I can resolve the issue by removing the legacy permissions and restarting the job. You can also decide, after reviewing the report, to restart the job with the ‘BadItemLimit’ paramater increased to a number high enough the not cause the move request to fail, because these errors indicate that although the permissions will not be migrated, the items itself will be copied to Exchange Online so no data will be lost.

In conclusion, you can see why I prefer to review the errors in an Exchange hybrid migration using the export-clixml cmdlet. It is a much more convenient way to navigate around all errors and get a complete view of the issues.

Completing individual moverequests from a migrationbatch

While working on a hybrid migration from Exchange 2010 to Office 365, I found myself in a place where I had a migration batch of approximately 350 mailboxes that was ready to complete, but I only wanted to complete around five of them so my costumer could do some further testing. This was something that was decided later on in the migration traject, so the migration batch was already set up and had synced.

Now, when you use PowerShell, you can complete the entire migration batch with the complete-migrationbatch cmdlet. But there’s no such thing for completing individual mailboxes. With some workarounds though, you can complete individual moverequests from within a migration batch.

First, get the moverequest you’re completing:

You’ll see the status is ‘synced’, so the mailbox is ready to complete. To do this, we change some properties for the individual moverequest:

In this example, I set the ‘completeafter’ propertiy to 5 minutes. You can also provide a date/time string to already schedule the completion for a few days from now.

After this is set, we need to resume the migrationbatch so it will complete:

After a while, you can do another get-moverequest and see the status will be ‘completed’.

Pass-through authentication and SSO

In an earlier blogpost I wrote about the new ‘pass-through authentication’ feature that is in public preview in the new Azure AD Sync client.

One of the most common reasons to use ADFS in an Office 365 setup, is that it allows you to do Single Sign-On. You only have to authenticate once, when you log on to your domain joined device, and your Kerberos ticket is used to authenticate you against Azure AD and therefore Office 365.

With the pass-through authentication feature, you get the same benefits. Because the PTA agent authenticates you against your on-premises Active Directory, you can use PTA to do single sign-on as well. So lets take this setup for a test-drive!

Basis of this test is the setup from my previous blog: an on-premises AD with PTA enabled in Azure AD Connect combined with the 365Dude Office 365-tenant. I added a Windows 10 machine to the mix, and domain-joined it to the on-premises AD. I use the Mr. Dude account to log on to this machine.

There are a few prerequisites to take into account when doing SSO. Mainly, they regard the OS and browser used. It is required to use a Windows machine, as it uses Kerberos for the underlying authentication. When using Windows 10, 8.1 or 8 clients, Internet Explorer, Chrome and Firefox are supported, where Edge isn’t.  Lower Windows-versions aren’t supported. When installing Azure AD Connect with the PTA / SSO option, a computer account is created in AD to handle your authentication requests.

First, we need to make sure that the Office 365 tenant is configured to allow OAuth. We do this with one single line of PowerShell:

Furthermore, we need to add the Microsoft authentication servers to the Intranet zone of the browser’s security settings. They need to be explicitly added to the machine’s Intranet zone, so that the browser will automatically send the currently logged in user’s credentials in the form of a Kerberos ticket to Azure AD. The easiest way to add the required URLs to the Intranet zone is to simply create a group policy in Active Directory.

There are two URL’s that need to be configured here:

https://autologon.microsoftazuread-sso.com

https://aadg.windows.net.nsatc.net

When done, the GPO should look as follows:

That’s it! After configuring our tenant and setting up the URL’s to be in the Intranet zone, you can ‘single sign-on’ to your Office 365 service.

So, what’s the user experience here? Let’s check it out.

After logging on the machine, we open IE and navigate to the https://outlook.office.com URL.

We are prompted to enter username and password. After we enter the username part, and navigate to the password field, the computer checks for a Kerberos ticket for this username. When it exists, the client is automatically logged on without the need of entering the password.

Well, it’s a bit hard to capture that on a screenshot. But you catch my drift, right? 😉 When the login completes, we just find our Outlook for the Web logged on and ready to use.

So that’s the webmail part. Most of my clients however, prefer to use Outlook as part of the Office suite for their e-mail work. How does SSO work there? Well, almost in the same way.

Prerequisite for SSO to work with client apps, is that the apps support modern authentication. So for our Windows-based clients, that will be Office 2013 and 2016. For this test, I used Office 2016.

When first starting Outlook, we are prompted to connect to Office 365.

So lets do just that. When we click the ‘connect’ button, we get a login screen equal to the one being used in the Outlook for the Web based interface.

And there we are. After checking the username and navigating to the ‘password’ field, the dialog box checks to see if a Kerberos tickets exists for this account and when it does, it uses that to log on to the Office 365 service. No further user interaction is required, It Just Works.

So, it’s a wrap. This single sign-on functionality provides you with all the ease of use you have with ADFS, without the need of a complete ADFS infrastructure. Ofcourse, when you plan on (or need to be) running ADFS in your environment for single sign-on with other applications, such as CRM or ERP software for example, there is no use in using PTA with SSO for Office 365, you just take the ADFS route. If you however want to provide your user with the same experience without the need for the infrastructure, PTA with SSO can be a great alternative!

 

AAD Pass-through authentication checked out

Exactly one week ago, Alex Simons announced what he called ‘the biggest news of the year’: the public preview of Azure AD Pass-through authentication and seamless single sign-on.

I tweeted something about how this would be the start of the end of ADFS for Office 365, but is it? Time to take this new options for a test drive!

First, some side-notes. The new functions are in public preview, which means it’s ‘beta’ and SLA is non-existent. You should therefore not use it in production environments. But I guess you’re smart enough to know that.

So, for the test I created a simpel AD domain 365dude.local, with one user: Mr. Dude. This will be the user I will be syncing to my existing Office 365 tenant that houses 365dude.nl. To make synchronisation easier, I added 365dude.nl to the list of UPN-suffixes in my AD domain and changed the UPN of this user to mrdude@365dude.nl.

I enabled the dirsync functionality on my Office 365 tenant using PowerShell:

After that, I start with installing the Azure AD Connect tool

I guess you’re smart enough to know how to install AADConnect, so I’ll just focus on the new and shiny stuff here.

In the ‘user sign in’ screen, we can select how we want to handle sing in requests for our synced users. Of course we can use password sync or ADFS, but we’ll choose the new ‘pass-through authentication’ option.

So, what does this new pass-through authentication mechanism do? It relies on a connector which runs in local network. When you authenticate through the Office 365 portal (or a supported client app), your authentication request is sent to the connector over a secure channel. The connector will then check your credentials against your on-prem AD. So, no more syncing of password hashes to the cloud and using AAD as your authentication source: you authenticate directly to your on-prem AD. Just like you would when using ADFS, but without the need for the complete ADFS infrastructure.

On the ‘optional features’ tab, you can still choose to do password sync. This way the on-prem passwords are still synced to AAD, so you can easily fall back to authenticating to AAD if you want to, because the passwords are already in place.

 

In this screenshot, I also selected the ‘password write-back’ option. To use this, you need an AD Premium license.

You get some information on what the software will do, and the installation will continue.

Once completed, I can find the user from my on-prem AD in the active users overview of Office 365 and assign a license.

Remember, with pass-through authentication when you sign in to Office 365, your authentication is routed through the connector in your local network and your credentials are verified against your on-prem AD. Thus, you can find eventlog messages for your sign-in request with Office 365.

So, will this be the end of ADFS? In some ways it will, I guess. You can get the same functionality, without the need of the complete ADFS infra. And ofcourse, with the pass-through authentication mechanism in place, you don’t need to sync your (hashed) passwords to the cloud to get the same sign-in experience, which could please your security officers. On the other hand, many (larger) organizations already have the ADFS infrastructure in place to get single sign-in functionality with other LOB applications. If you already went down that road, you can simply use that to connect Office 365, too. Another disadvantage is that when you use directory synchronisation to connect your on-prem AD to AAD, you’ll need an on-prem Exchange Server to perform certain management tasks, such as adding aliases to a mail enabled user. For smaller organizations, this can be reason to not choose the directory sync option all, be it with or withou pass-trough authentication. I expect this will be something that Microsoft will fix somewhere in the next year, but that’s just a guess.

Want more information on this new (preview) functionality in AAD Connect and Office 365? Be sure to check out the official documentation.

Exchange 2010 EMC issue resolved – so it seems

I blogged earlier about an issue when setting up a connection to Office 365 in the Exchange Management Console in Exchange 2010 (in this case, Service Pack 3, Rollup Update 12).

It seems that this issue was resolved. Despite the fact that I didn’t get any further feedback on the ticket I raised at MS Support or within the support forums thread mentioned in my earlier post, I decided to simply give it another try today… And It Just Worked™.

I didn’t change any thing on the existing configuration, just simply tried adding the connection to EMC again. At another client that encountered the same issue (but for which we didn’t raise a ticket yet), the issue seems to be resolved too, which leads to think there was a configuration change on the Office 365 side. However, we will need to wait for official feedback from Microsoft about this issue to have this confirmed.

Exchange 2010 hybrid connection in EMC fails with ‘ExchangeBuild’ error

I ran in to an issue using Exchange 2010 in a Hybrid setup.

In this case, I am running a fully patched Exchange 2010 SP3 CU12 machine, and I ran the new Hybrid Configuration Wizard. This completed successfully, so I do have a working Hybrid setup and should be able to move mailboxes cross-prem and send and receive mail on both sides.

Next step is to add the Office 365 tenant to my Exchange Management Console. To do this, I right-click the ‘Microsoft Exchange’ tree and click the ‘add new forest’ link. After naming the forest and selecting the ‘Exchange Online’ option for the remote PowerShell instance, I need to enter the credentials for my tenant. When validating, the wizard throws an error.

Exchange Error

‘The format of the Exchange object version is wrong. Parameter name: ExchangeBuild’.

After some searching, I found other people running into the same problem. Some get the error when trying to open an existing instance in EMC, some get it when trying to create a new one, like me. For those who get the error on an existing instance, some users report that deleting and re-adding the instance solves the problem. Others can’t re-add the instance after deleting it, facing the same error as me. A MSFT employee responded in the thread, stating that the problem has been found and the Exchange team is working on resolving it. For now, there isn’t a resolution.

The issue is also mentioned on support.microsoft.com, but aside from two obvious workarounds, there isn’t a solution yet.

For creating the hybrid setup, you can use the new HCW without running in to the issue. For viewing properties for users that are homed on Exchange Online in your hybrid scenario, you can use the Exchange Online console online. However, for the time being, there isn’t a supported way to modify these properties in this scenario.

Edit 3-9-2016: The issue seems to be resolved.

 

Change the Office 365 language for synced users

Office 365 is all about letting users set up their own environment and preferences. One of those preferences can be the language in the Office 365 interface. I work for a company in The Netherlands, where everyone speaks Dutch, but I still prefer my computer and software to be in English. In Office 365, I can set this up… Unless I’m a ‘synced user’, a user who originates in an on-prem AD and that is being synced to the cloud.

Even when I ask my administrator, he can’t set the preferred language via the appropriate PowerShell cmdlet

This will result in an error:

So, what causes this error? It’s pretty simple. When you use DirSync (or the newer AADSync), the on-prem directory is leading. You could say the sync is one-way: changes sync from on-prem to the cloud, but not the other way around. Therefore, you can’t make changes to properties of the identity in Office 365. So, if you want to change the preferred language for such users, you should do this in the on-prem AD.

All available language codes are listed here. And keep in mind, if your preferred language is not set, it will default to English.

stopped-extension-dll-exception when running DirSync

Today I ran into a problem at a costumer. They are running an Exchange 2013 server in a hybrid configuration with O365. Part of the users are on the local Exchange server, but the main part of the users use O365 for their mailbox needs.

Some new mailboxes were needed, so I created the users in Active Directory and ran the enable-remotemailbox cmdlet to provision their mailboxes. Next step is to assign a license in O365, so I kicked of DirSync with the start-onlinecoexistencesync cmdlet and reverted to the O365 portal to license my new users…. But they didn’t show up.

I checked the MIIS client to see that the export to Azure AD failed with the ‘stopped-extension-dll-exception’ error.

After some research, I found that this can be related to the execution policy in PowerShell. For DirSync to run successfully, it turns out the PoSH execution policy should be set to ‘unrestricted’. Earlier this week, a GPO was activated at this costumer to enable PowerShell remoting, including the option to set the execution policy to ‘ remote signed’. After creating a deny for this policy for the server running DirSync and manually setting the execution policy back to ‘unrestricted’, DirSync worked again and my newly created users started showing up in the O365 portal.

Wise lesson: for DirSync to run successfully, the execution policy for PowerShell on the server running DirSync needs to be set to ‘unrestricted’. Pretty weird, as in IMHO ‘remote signed’ is the preferred and most secure setting. It is strange, at least, that Microsoft’s own products don’t work in this scenario. I haven’t tried this with the newer AADSync yet, so maybe this flaw was fixed in the newer releases.

There are a few issues on the O365 support pages regarding this issue, some of which stating that even if the execution policy is set to ‘unrestricted’ via GPO DirSync won’t work; the GPO must set the execution policy to undefined and the execution policy on the server needs to be set manually. However, only some of the people responding could confirm this, so your mileage may vary…