IPv4 addresses are missing from Windows Server 2016 DNS root hints list

Symptoms:
Clients cannot resolve many external DNS names when Windows Server 2016 DNS server is configured to use root hints.

When examining the root hints tab you discover that vast majority of the root servers listed with their IPv6 address without any IPv4 address.

When validating root hints which have IPv6 address only, it results an error.

DNSipv6error

Workaround:
To resolve the issue manually edit each of the roots hint and remove the IPv6 addresses, leaving only IPv4 addresses.
Validate the root hint IPv4 address from the list of root servers published by IANA.org

More details:
This issue occurs because of a recently discovered bug in the DNS service in Windows Server 2016 that is being investigated by Microsoft.

Advertisements

The format of the schedule attribute of the following object is unrecognizable

Symptoms:
Knowledge Consistency Checker logs “The format of the schedule attribute of the following object is unrecognizable. ” error in Directory Service log on a Domain Controller.

Error message:
Log Name: Directory Service
Source: Microsoft-Windows-ActiveDirectory_DomainService
Event ID: 1121
Task Category: Knowledge Consistency Checker
Level: Error
Keywords: Classic
User: ANONYMOUS LOGON
Computer: LON-DC01.contoso.com
Description:
The format of the schedule attribute of the following object is unrecognizable.

Object:
CN=Lon2NYC,CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=contoso,DC=com

A default schedule will be substituted. This event will continue to occur until the schedule attribute on this object has been corrected.

User Action
Modify the schedule attribute.

Resolution:
The error is logged when there is no allowed period when replication can run between the sites connected by the Site Link referenced in the error, so replication fails back to the default 3 hours.
To resolve the issue open Active Directory Sites and Services snap-in, expand to Sites > Inter-site Transports > IP, then locate the Site Link highlighted in the error, and in the Properties of the Site Link on the General tab click on Change Schedule… and make sure there are period(s) when replication is allowed.

replication
The corrected replication schedule will need to replicate to sites that are connected by the Site Link, which can take up to 3 hours.

Disabled user account prevents ActiveSync remote wipe to succeed

It is a common security practice to disable user accounts before a former employee’s device is wiped. But Exchange ActiveSync remote wipe command is received successfully only when the target device can connect and authenticate to Exchange.

Remote wipe command queued.wipe pending

ActiveSync device attempts to connect to Exchange server while the AD account is disabled or password is invalid. The attempt fails with HTTP Error 401: Unauthorized

10:10:54 10.1.1.10 OPTIONS /Microsoft-Server-ActiveSync/default.eas &CorrelationID=<empty>;&ClientId=…&cafeReqId=…; 443 Contoso\jsmith 10.1.1.20 Apple-iPhone4C1/1206.70 – 401 1 1326 437

ActiveSync device successfully connects to the Exchange server

10:42:17 10.1.1.10 OPTIONS /Microsoft-Server-ActiveSync/default.eas &CorrelationID=<empty>;&ClientId=… &cafeReqId=…; 443 Contoso\jsmith 10.1.1.20 Apple-iPhone4C1/1206.70 – 200 0 0 15

ActiveSync device successfully retrives policy update

10:42:18 10.1.1.10 POST /Microsoft-Server-ActiveSync/default.eas User=jsmith&DeviceId=…&DeviceType=iPhone&Cmd=Provision&CorrelationID=<empty>;&ClientId=… &cafeReqId=…; 443 Contoso\jsmith 10.1.1.20 Apple-iPhone4C1/1206.70 – 200 0 0 93

ActiveSync device successfully confirms device wipe started

10:42:18 10.1.1.10 POST /Microsoft-Server-ActiveSync/default.eas User=jsmith&DeviceId=..&DeviceType=iPhone&Cmd=Provision&CorrelationID=<empty>;&ClientId=… &cafeReqId=…; 443 Contoso\jsmith 10.1.1.20 Apple-iPhone4C1/1206.70 – 200 0 0 62

Device wiped successfully .wipe successfull

Now you can disable the user account in Active Directory.

(example logs truncated for readability.)

Summary of how Windows cluster works

Cluster health checks

  • Between every node on all cluster enabled networks
  • By default a heartbeat is sent every 1 second
  • Node is removed from the cluster if 5 heartbeats are missed
  • Not a ping, but a full “Request – Reply” handshake
  • UDP traffic between cluster nodes
  • Sensitive to latency

Intra-Cluster communication

  • Are over single interface
  • Can failover to another interface if there the route fails
    • Networks without network gateway take priority, as it is considered as cluster-dedicated network
  • Includes Resource failover communication:
    • Resource status/state changes (offline/move/online, etc) must be acknowledged by all nodes to the cluster owner between failover steps
  • Sensitive to latency

Microsoft Failover Cluster Virtual Adapter (aka NetFT Virtual adapter)

  • Created and configured when the Cluster service is installed
  • Has APIPA IP address
  • Hidden device in Device Manager
  • “Keep hands off”  – don’t change anything

Microsoft Failover Cluster Virtual Adapter Performance Filter (aka NetFT Virtual adapter Performance Filter)

  • New in Win2012
  • Binds to the physical interface(s)
  • Captures incoming UDP cluster traffic from the network
  • Routes incoming cluster communication directly to the Cluster Service

Cluster service (ClusSvc.exe)

  • Talks to NetFT adapter only
  • Directly via TCP/3343
  • Using IPV6 only

NetFT Virtual adapter

  • Tunnels cluster traffic via the physical interface(s)
  • Communicates on UDP/3343
  • Using IPv4 or IPv6 depending on the physical interface(s) configuration

NetFT Architecture

NetFT-Architecture

Source:

https://blogs.msdn.microsoft.com/clustering/2012/11/21/tuning-failover-cluster-network-thresholds/

The WinRM client cannot process the request

Syptoms:

Error message when you try to start Exchange 2013 Management Shell (EMS):

VERBOSE: Connecting to EX-LON01.contoso.com.
New-PSSession : [EX-LON01.contoso.com] Connecting to remote server
EX-LON01.contoso.com failed with the following error message: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ New-PSSession -ConnectionURI “$connectionUri” -ConfigurationName Microsoft.Excha …
+ CategoryInfo : OpenError: (System.Manageme….RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotingTransportException
+ FullyQualifiedErrorId : -2144108297, PSSessionOpenFailed

Additionally browsing the http://localhost/PowerShell site failed with HTTP 500 error and also in IIS Manager, opening Authentication settings of the PowerShell virtual directory under Default Web Site returns an error about the problem in the virtual directory’s the web.config file:

PowerShell

Resolution:

After comparing the /PowerShell virtual directory’s web.config file to the same file from a working server, it turned out there was an incorrect “<security> </security>” section where it was not allowed.
After removing the section /Powershell virtual directory was accessible and EMS was able to start and connect to the local server.

Get-Mailbox cmdlet returns value of the legacy msExchHomeServerName attribute

Symptoms:

PowerShell cmdlets Get-Mailbox, Get-CASMailbox and Get-Recipient returns the ServerName icorrectly.

Example:

Get-Recipient jsmith | fl Name, ServerName, DataBase

Name       : Smith, John
ServerName : EX-LON1
Database   : LON-DB01

# find out on which server has the active copy of the database

Get-MailboxDatabase LON-DB01 | fl Name, Server

Name   : LON-DB01
Server : EX-LON02

Reason:

Get-Mailbox, Get-CASMailbox and Get-Recipient returns the value of the legacy msExchHomeServerName attribute which is updated when the mailbox is created but not updated later anymore due to a change in introduced in Exchange 2010

More Information:

Exchange 2010: HomeMTA and msExchHomeServerName are not updated on mailboxes.

Exchange AD topology does not discover all DC from all AD sites

Symptoms:

Exchange 2013 AD topology discovery runs every 15 minutes and discovers all the In-site Domain Controllers/Global Catalogs, but Out-of-site Domain Controllers/Global Catalogs are listed only from one another AD site.

First discovery finds all In-Site DC/GC from New York site (NY-DC & NY-DC2) and lists only DC/GC from London site (LN-DC1 & LN-DC2)

Log Name:      Application
Source:        MSExchange ADAccess
Event ID:      2080
Task Category: Topology
Level:         Information
Keywords:      Classic
Computer:      NY-Ex1.contoso.com
Description:
Process Microsoft.Exchange.Directory.TopologyService.exe (PID=1234). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
(Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)

In-site:
NY-DC1.contoso.com CDG 1 7 7 1 0 1 1 7 1
NY-DC2.contoso.com CDG 1 7 7 1 0 1 1 7 1

Out-of-site:
LN-DC1.contoso.com CDG 1 7 7 1 0 1 1 7 1
LN-DC2.contoso.com CDG 1 7 7 1 0 1 1 7 1

Next discovery finds all In-Site DC/GC from New York site (NY-DC & NY-DC2) and lists only DC/GC from Tokyo site (TK-DC1 & TK-DC2)

Log Name:      Application
Source:        MSExchange ADAccess
Event ID:      2080
Task Category: Topology
Level:         Information
Keywords:      Classic
Computer:      NY-Ex1.contoso.com
Description:
Process Microsoft.Exchange.Directory.TopologyService.exe (PID=1234). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
(Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)

In-site:
NY-DC1.contoso.com CDG 1 7 7 1 0 1 1 7 1
NY-DC2.contoso.com CDG 1 7 7 1 0 1 1 7 1

Out-of-site:
TK-DC1.contoso.com CDG 1 7 7 1 0 1 1 7 1
TK-DC2.contoso.com CDG 1 7 7 1 0 1 1 7 1
Resolution:
This is by design in Exchange 2013.
Previous versions of Exchange server discovered all Out-of-site Domain Controllers from every other AD site that had direct AD link defined. This was found to be resource intensive and time consuming therefore starting with Exchange 2013 randomly selects an AD site to use the Domain Controllers in case In-site Domain Controllers became unavailable.