Using Exchange 2013 Logs to Troubleshoot Migration Issues

Aug. 15th 2016

During a Quest Migration for Exchange, you may run into problems that can delay or even halt the migration.  Quickly pinpointing the issue can get your migration back on track.  One way of achieving this is to review the logs of the affected systems.  Exchange 2013 has an enhanced set of logging capabilities that can assist you with troubleshooting issues that might arise during your migration.  The three Transport services can each log information over and above the normal event messages they might register in the system’s Windows application event log. Some of these activities are common to all three services, while other logs are maintained by specific services.  Most likely, these logs won’t be used on a daily basis, but can prove useful when troubleshooting connectivity.

There are three types of logs that all of the Transport services can independently maintain:

  • Connectivity logs capture information of all connections to a server. Logging is on by default. The default path Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\Connectivity will log entries for ordinary SMTP delivery and every other significant server-to-server connectivity event. Other components might log different data in the con­nectivity logs, too, although connectivity logs don’t show the details of protocol-level conversations.
  • Receive protocol and send protocol logs show the details of conversations: which party to the conversation said what and what the response was. The default setting for these logs are off. You would enable them only if you suspect problems with a specific protocol’s connectivity because the logs are quite verbose. It’s a good practice to note when they are enabled so they can be turned off after troubleshooting.

In addition, individual services maintain a number of logs. The following table summarizes the log types by component.

Service Log type Notes
Front End Transport Agent log Logs actions and configurations taken by agents.
Mailbox Transport Mailbox delivery agent Records actions taken by the Mailbox delivery agent only.
Mailbox submission agent Records actions taken by the submission agent.
Transport Active user statistics This log records user activity, including the number of messages and bytes sent or received. You can’t disable this log type.
Agent Logs agent actions for the Transport service.
Information Rights Management This log shows activity related to trans­port decryption of information rights management (IRM) messages.
Message tracking These logs are used to power Get-MessageTrackingLog and the rest of the message tracking functionality in EMS and EAC.
Queue These logs record queue actions, such as freezing or resuming queues.  You can’t turn queue logging off.
Routing table The routing table logs are a set of XML files that outline the routing topology Exchange uses; the logs are updated periodically. They used to be viewable with the Routing Log Viewer in Exchange 2010, but that tool was dropped in Exchange 2013.
Server statistics The server statistics log contains detailed information about the server’s activity, including the number and size of mes­sages sent and received, the number of DSNs generated, and the calculated end-to-end latency for message transport.

Logging Control

The Exchange Admin Center (EAC) has a very limited set of controls for logging behavior. Use the Transport Logs tab of the server properties dialog box to enable message tracking and connectivity logging and to change the paths for those logs and the send and receive protocol logs.

The Transport Logs section of the Organization Transport Settings dialog box allows you to control message tracking and connectivity logging.

The Exchange Management Shell (EMS) allows for more control of logging. Each of the services’ log­ging behavior for a service can be changed with the appropriate Set- cmdlet for the target service. For example, if you want to change how FET logs events, you’d use Set-FrontEndTransportService with the parameters to specify the options you want. Each of the logs supports parameters that control whether logging is enabled, how big log files may grow before a new log is created, and how long logs are kept. Each of these parameters has a name that begins with the type of log (AgentLog, ConnectivityLog, IRMLog, et cetera), followed by the parameter name (MaxAge, Path, and so on). When you know this, it is fairly easy to construct commands to do what you want done. For example, you might customize the connectivity logging behavior as follows:

Get-TransportService | Set-TransportService –ConnectivityLogEnabled $true –ConnectivityLogPath c:\logs\Connectivity –IrmLogEnabled $true –IrmLogPath c:\logs\ADRMS

Logs are named with a prefix (CONNECT, RECV, and ACTVUSRSTAT are examples) plus the date; some logging subsystems also include other items. The first log created on a day is named using a convention of YYYYMMDD-1.log where YYYYMMDD represents the year, month, and day. For example, the first active user statistics log created on March 2, 2014 is named ACTVUSRSTAT1.020140302-1.log. By default, Exchange creates a new log after it captures 10 MB of data in that log file. (You can adjust this with the LogMaxSize parameter.)

Each log type has a maximum size which defaults to 250 MB. A circular logging scheme keeps the logs in the directory under this size by removing the old­est logs to free up space for new logs. You can increase the amount of storage assigned to connectivity logs by setting the value like this:

Set-TransportService –Identity <Exchange Server>  –ConnectivityLogMaxDirectorySize 500MB

Assuming the directory storage threshold is not exceeded, logs are normally retained for 30 days. Because only the most recent logs are typically used to debug connectivity problems, you might decide to reduce this period. For example, here’s how you would set the reten­tion period for the connectivity logs on a server to 15 days:

Set-TransportService –Identity <Exchange Server> –IRMLogMaxAge 15.00:00:00

Protocol Logging

Combing through protocol logs can be tedious. It’s easy to miss fine details, and the process of correlating log entries across multiple servers is painful unless you can automate it, which might be easier than you think. Microsoft has a tool called LogParser (available from http://www.microsoft.com/en-us/download/details.aspx?id=24659) that gives you a query engine that works against several flavors of log file. You construct queries using an SQL-like syntax, and LogParser runs them against the log sets you specify.

If you’re familiar with SQL, then LogParser will be easy to understand. If you’re not, there are many examples of various queries and reports of use for Exchange on the Internet, and a few web searches will quickly find samples that you can adapt to get the data you want.

That’s it for this installment of Troubleshooting Exchange 2013.  Utilizing these logs in conjunction with the logs from the Quest Migration Tool should help you to quickly pinpoint and resolve any issues you may encounter during your migration.

Posted by LeadThem Consulting | in Migration Manager for Exchange | No Comments »

Issues when Migrating Exchange

Oct. 8th 2015

Error 2060 When Running Public Folders Sync Job

Environment:

Public folder Sync job is configured and running using legacy agents. During the sync, error 2060 “ImportPF Error 2060 Public store was not found on the current server” is showing up in the logs. Public Folders exist in the target and all the necessary rights are granted and working correctly.

Resolution:

This is a replication issue. Check whether the administrative mailbox specified for the public folder synchronization job is associated with public store on the same Exchange server. In this case, the Public Folder was homed to another server that did not have the Admin mailbox. Once this was changed, the Sync Job was able to find the PF database and import the contents from the source.

 

MaGE Native Move Error “Invalid situation the version of source or target side is not supported”

Environment:

Client is migrating from Exchange 2007 to 2007 with an Exchange 2010 CAS server in the target organization. During configuration of the MaGE agent the following error occurs in the console:

“Invalid situation the version of source or target side is not supported”

Resolution:

Documentation states that as long as there is a 2010 CAS server in the organization, Native Move can be used to migrate mailboxes. However, this is not entirely correct. Because the MaGE agent is using the native move PowerShell cmdlet in Exchange 2010, it expects the target mailboxes to reside on a 2010 database server. Since the mailboxes were being hosted on a 2007 server in the target, this was not possible. Legacy agents were then set up and configured and mailboxes were migrated without incident.

 

Mail Items Missing After Switch to Target

Environment:

Client users are stating that email is missing from their new target mailbox. Checked Mail Agent logs for any sync errors and attempted to perform a full resync of mailbox from Source to Target. Migration Manager Statistics and logged in comparisons of source to target show mailbox sizes to be the same.

Resolution:

While troubleshooting, it was revealed that 5 months earlier there had been a catastrophic failure of the source mail server and the server had been recovered through via recovery group. The OSTs had been preserved during this process and was found to contain the missing mail. Using a 3rd party utility to convert the backup OST to a PST, the client was able to import mail items to the new target mailbox.

 

Posted by LeadThem Consulting | in Migration Manager for Exchange | Comments Off on Issues when Migrating Exchange

Performing an Intra-Forest migration

Oct. 8th 2015

Performing an Intra-Forest migration is different in many aspects than performing an Inter-Forest migrations. The biggest issue that needs to be watched out for is not having two same accounts with same SIDS in both domains. That is why as soon as possible after migrating the objects they need to be deleted from the source domain. This make having a tested backup extremely important in case there is a need to back out the migration. Careful planning needs to be done when performing an Intra-Forest migration. Below are high level steps to help insure that the migration goes smoothly.

  1. Upgrade source Global groups to Universal
  2. Physically migrate all groups to target Domain.
    1. Migrate groups with Sid History and adding source members.
    2. Delete source groups
    3. Change admin point to target for all groups.
    4. Resource Process workgroup data (i.e., file servers, etc.)
    5. Execute ADPW in all domains to update group membership of Source users
    6. Optional but recommended:  Clean up sidHistory on migrated groups.
  3. Create user “stubs” in target. (i.e., Logically Migrate)
    1. Migrate user accounts, skipping sAMAccountName (migration session)
    2. DO NOT copy SID History, Password, Security Descriptor, and Mailbox.
    3. DO NOT Enable user account
  4. Resource process and move all workstations. (delete the source computer accounts during the physical migration – if QMM Directory Sync is running, be sure NOT to sync deletions)
    1. Exclude serviceprincipalname attribute from computer objects
  5. Resource process servers for ACL only
  6. Migrate users (Physical Migration)
    1. Verify RMAD session ran recently (in case of an object restore requirement)
    2. Migrate selected users with Password, SID History, Mailbox, and sAMAccountName
    3. Run ADPW with custom map to update TARGET ‘Update Group Membership”. Verify migration
    4. If SQL servers present, SQL Wizard run with custom map
    5. Delete source users
  7. Migrate Servers (physical migration)
    1. Resource Process (do not double acl, replace the acl).
    2. Join to target domain
  8. Clean-up
    1. If MS SQL present, re-run SQL Wizard with all objects.
    2. Re-run ADPW, clean up legacy memberships.
    3. Verify RMAD run and user ADPW to cleanup SID History.
    4. Remove source domain.

Note that you may choose to “loop” on step three for sets of users at a time.  You may also choose to loop on step 2 for sets of groups at a time

Posted by LeadThem Consulting | in Migration Manager for Active Directory, Migration Manager for Exchange | Comments Off on Performing an Intra-Forest migration

Join a *Nix host to Active Directory without utilizing a clear text password

Aug. 21st 2013

 

Within QAS, the vastool command on *nix hosts is like the Swiss Army Knife of the QAS product.  This command allows an admin to verify authentication, list the QAS cache on that host, verify and remove Kerberos tickets, and join the host to an Active Directory domain just to name a few.  Today, I am going to discuss the join command, more specifically joining a host to Active Directory without using a plain text password.

Most customers wish to automate the process of joining a *nix host to Active Directory by a standalone script, or incorporating code into their kick start/jumpstart provisioning.  This can be accomplished, but the join command requires authentication to Active Directory with an account that possesses the necessary permissions to join a computer to Active Directory.  The most direct way to issue this command is:

Vastool –u ADUser@domain.com –w <password> join domain.com

 

The command above will join the current host to domain.com utilizing the Active Directory credentials for ADUser@domain.com.  The issue most customers have with this is that the password is clear text and would appear as clear text within their script.  The way around this is to create a ‘service’ account that has the necessary permissions to join a computer to active directory, and generate a key tab for this user.  This key tab can then be placed with the kick start files and referenced in the join command instead of passing a clear text password, as below:

Vastool –u ADUser@domain.com –k /etc/opt/quest/vas/sa.keytab join domain.com

 

The command above is the same as before, but now the vastool command is utilizing the key tab to perform the authentication rather than having a clear text password present.

In order to use this method, you will need to create a service account.  This account should be similar to most Windows service accounts in that, it does not force password expiration.  The msDS-KeyVersionNumber will need to be recorded for use in the generation commands of the account, this can be acquired by issuing the following from a QAS connected *nix host:

Vastool –u host/ attrs <ServiceAccountID> msDS-KeyVersionNumber

 

Once the key version is acquired, use the following commands to create a key tab for the service account:

Ktutil –k <Path>/<ServiceAccount>.keytab add –p <AccountUPN> -e arcfour-hmac-md5 –V <KeyVersionNumber> -w ‘password’
Ktutil –k <Path>/<SarviceAccount>.keytab add –p <AccountUPN> -e aes256-cts-hmac-sha1-96 –V <KeyVersionNumber> -w ‘password’
Ktutil –k <Path>/<ServiceAccount>.keytab add –p <AccountUPN> -e aes128-cts-hmac-sha1-96 –V <KeyVersionNumber> -w ‘password’

 

Make sure to include the single quotes around the password at the end to mitigate any issues with miss-interpreting special characters that may exist in the password.

Once the key tab has been completed, it can be placed with the provisioning files and referenced for vastool commands.

 

 

Author: Russ Burden, Technical Architect, LeadThem Consulting

 

 ———————————————————————————————————————————————————————————————-

 

 

Posted by LeadThem Consulting | in Authentication Services | Comments Off on Join a *Nix host to Active Directory without utilizing a clear text password

Updating MAPI CDO

Aug. 21st 2013

 

Have you updated Exchange 2010 SP3? What about your MAPI CDO for you Quest boxes?  Most people forget about updating their MAPI CDO. With the latest installs I have done I have noticed that the latest updates for exchange 2010 SP3 are now requiring that you use the May 2013 update of MAPI CDO in order for MAPI CDO required programs to run.

http://www.microsoft.com/en-us/download/details.aspx?id=39045

To update MAPI CDO:

  1. Download the Latest update
  2. Stop all services that use MAPI CDO
  3. Uninstall old version of MAPI CDO
  4. Install new version of MAPI CDO
  5. Reboot
  6. Start Services that you stopped

Article By: Wayne Thompson, LeadThem Consulting Exchange Architect

 

 ———————————————————————————————————————————————————————————————-

 

 

 

Posted by LeadThem Consulting | in Quest Migration Manager | Comments Off on Updating MAPI CDO