The format of the schedule attribute of the following object is unrecognizable

Symptoms:
Knowledge Consistency Checker logs “The format of the schedule attribute of the following object is unrecognizable. ” error in Directory Service log on a Domain Controller.

Error message:
Log Name: Directory Service
Source: Microsoft-Windows-ActiveDirectory_DomainService
Event ID: 1121
Task Category: Knowledge Consistency Checker
Level: Error
Keywords: Classic
User: ANONYMOUS LOGON
Computer: LON-DC01.contoso.com
Description:
The format of the schedule attribute of the following object is unrecognizable.

Object:
CN=Lon2NYC,CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=contoso,DC=com

A default schedule will be substituted. This event will continue to occur until the schedule attribute on this object has been corrected.

User Action
Modify the schedule attribute.

Resolution:
The error is logged when there is no allowed period when replication can run between the sites connected by the Site Link referenced in the error, so replication fails back to the default 3 hours.
To resolve the issue open Active Directory Sites and Services snap-in, expand to Sites > Inter-site Transports > IP, then locate the Site Link highlighted in the error, and in the Properties of the Site Link on the General tab click on Change Schedule… and make sure there are period(s) when replication is allowed.

replication
The corrected replication schedule will need to replicate to sites that are connected by the Site Link, which can take up to 3 hours.

Disabled user account prevents ActiveSync remote wipe to succeed

It is a common security practice to disable user accounts before a former employee’s device is wiped. But Exchange ActiveSync remote wipe command is received successfully only when the target device can connect and authenticate to Exchange.

Remote wipe command queued.wipe pending

ActiveSync device attempts to connect to Exchange server while the AD account is disabled or password is invalid. The attempt fails with HTTP Error 401: Unauthorized

10:10:54 10.1.1.10 OPTIONS /Microsoft-Server-ActiveSync/default.eas &CorrelationID=<empty>;&ClientId=…&cafeReqId=…; 443 Contoso\jsmith 10.1.1.20 Apple-iPhone4C1/1206.70 – 401 1 1326 437

ActiveSync device successfully connects to the Exchange server

10:42:17 10.1.1.10 OPTIONS /Microsoft-Server-ActiveSync/default.eas &CorrelationID=<empty>;&ClientId=… &cafeReqId=…; 443 Contoso\jsmith 10.1.1.20 Apple-iPhone4C1/1206.70 – 200 0 0 15

ActiveSync device successfully retrives policy update

10:42:18 10.1.1.10 POST /Microsoft-Server-ActiveSync/default.eas User=jsmith&DeviceId=…&DeviceType=iPhone&Cmd=Provision&CorrelationID=<empty>;&ClientId=… &cafeReqId=…; 443 Contoso\jsmith 10.1.1.20 Apple-iPhone4C1/1206.70 – 200 0 0 93

ActiveSync device successfully confirms device wipe started

10:42:18 10.1.1.10 POST /Microsoft-Server-ActiveSync/default.eas User=jsmith&DeviceId=..&DeviceType=iPhone&Cmd=Provision&CorrelationID=<empty>;&ClientId=… &cafeReqId=…; 443 Contoso\jsmith 10.1.1.20 Apple-iPhone4C1/1206.70 – 200 0 0 62

Device wiped successfully .wipe successfull

Now you can disable the user account in Active Directory.

(example logs truncated for readability.)

Summary of how Windows cluster works

Cluster health checks

  • Between every node on all cluster enabled networks
  • By default a heartbeat is sent every 1 second
  • Node is removed from the cluster if 5 heartbeats are missed
  • Not a ping, but a full “Request – Reply” handshake
  • UDP traffic between cluster nodes
  • Sensitive to latency

Intra-Cluster communication

  • Are over single interface
  • Can failover to another interface if there the route fails
    • Networks without network gateway take priority, as it is considered as cluster-dedicated network
  • Includes Resource failover communication:
    • Resource status/state changes (offline/move/online, etc) must be acknowledged by all nodes to the cluster owner between failover steps
  • Sensitive to latency

Microsoft Failover Cluster Virtual Adapter (aka NetFT Virtual adapter)

  • Created and configured when the Cluster service is installed
  • Has APIPA IP address
  • Hidden device in Device Manager
  • “Keep hands off”  – don’t change anything

Microsoft Failover Cluster Virtual Adapter Performance Filter (aka NetFT Virtual adapter Performance Filter)

  • New in Win2012
  • Binds to the physical interface(s)
  • Captures incoming UDP cluster traffic from the network
  • Routes incoming cluster communication directly to the Cluster Service

Cluster service (ClusSvc.exe)

  • Talks to NetFT adapter only
  • Directly via TCP/3343
  • Using IPV6 only

NetFT Virtual adapter

  • Tunnels cluster traffic via the physical interface(s)
  • Communicates on UDP/3343
  • Using IPv4 or IPv6 depending on the physical interface(s) configuration

NetFT Architecture

NetFT-Architecture

Source:

https://blogs.msdn.microsoft.com/clustering/2012/11/21/tuning-failover-cluster-network-thresholds/

The WinRM client cannot process the request

Syptoms:

Error message when you try to start Exchange 2013 Management Shell (EMS):

VERBOSE: Connecting to EX-LON01.contoso.com.
New-PSSession : [EX-LON01.contoso.com] Connecting to remote server
EX-LON01.contoso.com failed with the following error message: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ New-PSSession -ConnectionURI “$connectionUri” -ConfigurationName Microsoft.Excha …
+ CategoryInfo : OpenError: (System.Manageme….RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotingTransportException
+ FullyQualifiedErrorId : -2144108297, PSSessionOpenFailed

Additionally browsing the http://localhost/PowerShell site failed with HTTP 500 error and also in IIS Manager, opening Authentication settings of the PowerShell virtual directory under Default Web Site returns an error about the problem in the virtual directory’s the web.config file:

PowerShell

Resolution:

After comparing the /PowerShell virtual directory’s web.config file to the same file from a working server, it turned out there was an incorrect “<security> </security>” section where it was not allowed.
After removing the section /Powershell virtual directory was accessible and EMS was able to start and connect to the local server.

Get-Mailbox cmdlet returns value of the legacy msExchHomeServerName attribute

Symptoms:

PowerShell cmdlets Get-Mailbox, Get-CASMailbox and Get-Recipient returns the ServerName icorrectly.

Example:

Get-Recipient jsmith | fl Name, ServerName, DataBase

Name       : Smith, John
ServerName : EX-LON1
Database   : LON-DB01

# find out on which server has the active copy of the database

Get-MailboxDatabase LON-DB01 | fl Name, Server

Name   : LON-DB01
Server : EX-LON02

Reason:

Get-Mailbox, Get-CASMailbox and Get-Recipient returns the value of the legacy msExchHomeServerName attribute which is updated when the mailbox is created but not updated later anymore due to a change in introduced in Exchange 2010

More Information:

Exchange 2010: HomeMTA and msExchHomeServerName are not updated on mailboxes.

Exchange AD topology does not discover all DC from all AD sites

Symptoms:

Exchange 2013 AD topology discovery runs every 15 minutes and discovers all the In-site Domain Controllers/Global Catalogs, but Out-of-site Domain Controllers/Global Catalogs are listed only from one another AD site.

First discovery finds all In-Site DC/GC from New York site (NY-DC & NY-DC2) and lists only DC/GC from London site (LN-DC1 & LN-DC2)

Log Name:      Application
Source:        MSExchange ADAccess
Event ID:      2080
Task Category: Topology
Level:         Information
Keywords:      Classic
Computer:      NY-Ex1.contoso.com
Description:
Process Microsoft.Exchange.Directory.TopologyService.exe (PID=1234). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
(Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)

In-site:
NY-DC1.contoso.com CDG 1 7 7 1 0 1 1 7 1
NY-DC2.contoso.com CDG 1 7 7 1 0 1 1 7 1

Out-of-site:
LN-DC1.contoso.com CDG 1 7 7 1 0 1 1 7 1
LN-DC2.contoso.com CDG 1 7 7 1 0 1 1 7 1

Next discovery finds all In-Site DC/GC from New York site (NY-DC & NY-DC2) and lists only DC/GC from Tokyo site (TK-DC1 & TK-DC2)

Log Name:      Application
Source:        MSExchange ADAccess
Event ID:      2080
Task Category: Topology
Level:         Information
Keywords:      Classic
Computer:      NY-Ex1.contoso.com
Description:
Process Microsoft.Exchange.Directory.TopologyService.exe (PID=1234). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
(Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)

In-site:
NY-DC1.contoso.com CDG 1 7 7 1 0 1 1 7 1
NY-DC2.contoso.com CDG 1 7 7 1 0 1 1 7 1

Out-of-site:
TK-DC1.contoso.com CDG 1 7 7 1 0 1 1 7 1
TK-DC2.contoso.com CDG 1 7 7 1 0 1 1 7 1
Resolution:
This is by design in Exchange 2013.
Previous versions of Exchange server discovered all Out-of-site Domain Controllers from every other AD site that had direct AD link defined. This was found to be resource intensive and time consuming therefore starting with Exchange 2013 randomly selects an AD site to use the Domain Controllers in case In-site Domain Controllers became unavailable.

Failure: This mailbox database contains one or more mailboxes

Symptom:

Some Exchange databases cannot be removed because of error message states there are mailboxes in the database.

Get-MailboxDatabase -Server Server1 | Remove-MailboxDatabase

Result:
This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, public folder mailboxes orarbitration mailboxes. To get a list of all mailboxes in this database, run the command Get-Mailbox –Database . To get a list of all mailbox plans in this database, run the command Get-MailboxPlan. To get a list of archive mailboxes in this database, run the command Get-Mailbox -Database -Archive. To get a list of all public folder mailboxes in this database, run the command Get-Mailbox -Database -PublicFolder. To get a list of all arbitration mailboxes in this database, run the command Get-Mailbox -Database -Arbitration.

[Remove-MailboxDatabase], AssociatedUserMailboxExistException    + FullyQualifiedErrorId : [Server=Server1,RequestId=26a04655-6a66-4990-9a07-7599f69850d0]….

Resolution:

To identify the mailboxes that are still in the mailbox database use the –Verbose switch with Remove-MailboxDatabase cmdlet

Get-MailboxDatabase -Server Server1 | Remove-MailboxDatabase -Verbose

Result:
VERBOSE: [12:28:53.599 GMT] Remove-MailboxDatabase : Runspace context: Executing user:
Contoso.com/Admins/Administrator, Executing user organization: , Currentorganization: , RBAC-enabled: Enabled.
VERBOSE: [12:28:53.599 GMT] Remove-MailboxDatabase : Active Directory session settings for ‘Remove-MailboxDatabase’ are: View Entire Forest: ‘False’, Default Scope: ‘Contoso.com’, Configuration Domain Controller: ‘DC1.Contoso.com’, Preferred Global Catalog: ‘DC1.Contoso.com’, Preferred Domain Controllers: ‘{ DC1.Contoso.com }’
VERBOSE: [12:28:53.599 GMT] Remove-MailboxDatabase : Beginning processing Remove-MailboxDatabase
VERBOSE: [12:28:53.599 GMT] Remove-MailboxDatabase : Instantiating handler with index 0 for cmdlet extension agent “Admin Audit Log Agent”.
VERBOSE: [12:28:53.615 GMT] Remove-MailboxDatabase : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s): {}, Exclusive Configuration Scope(s): {} }
VERBOSE: [12:28:53.615 GMT] Remove-MailboxDatabase : Searching objects “DAG-DB1” of type “MailboxDatabase” under the root “$null”.
VERBOSE: [12:28:53.630 GMT] Remove-MailboxDatabase : Previous operation run on domain controller’DC1.Contoso.com’.
VERBOSE: [12:28:53.646 GMT] Remove-MailboxDatabase : Verifying that there is no associated mailbox user or move request on the mailbox database “DAG-DB1”.
VERBOSE: [12:28:53.646 GMT] Remove-MailboxDatabase : Mailbox with DistinguishedName “CN=Doe\,John,OU=Users,DC=Contoso,DC=com” is still present in this database.
VERBOSE: [12:28:53.661 GMT] Remove-MailboxDatabase : Admin Audit Log: Entered Handler:OnComplete.
This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, public folder mailboxes orarbitration mailboxes. To get a list of all …

Note:

The returned DistinguishedName will identify the Active Directory account associated with the mailbox still present in the database.
If the account belowngs to a user mailbox use the Get-Mailbox cmdlet to find out whether it is a primary mailbox or an archive mailbox (or both).