Disclaimer

I am writing this blog  and others to explain how things work and some ways deployment and operational tasks can be handled. In other words, these postings are for demonstration purposes only. Since I am not familiar with your organization or environment I do not know if these steps are applicable to your environment or are even safe to perform in your environment. It is recommended that you contact Microsoft Support prior to making changes in your environment to ensure that these steps are applicable to your environment, and are safe to perform in your environment. By writing this blog I am in no way recommending that you perform these steps in your own environment. If you choose to follow the steps outlined in this or other blog postings on this site, you are assuming the risk for your actions.

1              Troubleshooting Active Directory Replication

1.1                 Repadmin.exe

Repadmin is a tool for checking replication status and troubleshooting replication issue.  Below is a table highlighting commonly used syntax of the repadmin tool.

Syntax	Usage
Repadmin /replsummary	The replsummary operation quickly and concisely summarizes the replication state and relative health of a forest.
Repadmin /replsummary /bysrc /bydest /sort: delta	The replsummary operation quickly and concisely summarizes the replication state and relative health of a forest.
Repadmin /showrepl <DC Name>	Displays the replication partners for each directory partition on the specified domain controller. Helps the administrator build a visual representation of the replication topology and see the role of each domain controller in the replication process.
Repadmin /showutdvec	Displays the highest Update Sequence Number (USN) for the specified domain controller. This information shows how up-to-date a replica is with its replication partners.
Repadmin /showobjmeta <DC> <DN of object>	Displays the replication metadata for a specified object stored in Active Directory, such as attribute ID, version number, originating and local Update Sequence Number (USN), and originating server’s GUID and Date and Time stamp. By comparing the replication metadata for the same object on different domain controllers, an administrator can determine whether replication has taken place.
Repadmin /showconn	Displays the connection objects for a specified domain controller. Default is local site.
Repadmin /replsingleobj <DC List> <Source DSA Name> <Object DN>	Replicates a single object between any two domain controllers that have partitions in common. The two domain controllers do not have a replication agreement. Replication agreements can be shown by using the Repadmin /showrepl command.
Repadmin /replicate <Destination_DC_List> <Naming Context>	Starts a replication event for the specified directory partition between the source and destination domain controllers. The source UUID can be determined when viewing the replication partners by using the Repadmin showrepl operation.
Repadmin /syncall <DC>	Synchronizes a specified domain controller with all replication partners.
Repadmin /queue	Displays tasks waiting in the replication queue.
Repadmin /showmsg <Error>	Displays the error message for a given error number.
Repadmin /viewlist <DC_List>	Displays a list of domain controllers.
Repadmin /showctx <DC_List>	Displays a list of computers that have opened sessions with a specified domain controller.
Repadmin /showcert	Displays the server certificates loaded on a specified domain controller.
Repadmin /removelingeringobjects <Dest_DC_List> <Source DC GUID> <NC> [/ADVISORY_MODE]	Uses an authoritative domain controller to compare the directory of a domain controller (destination) that is suspected of having lingering objects against the directory of a domain controller (source) that is designated as a reference source for up-to-date values for the domain of the destination. When the advisory mode parameter is used, this command provides a list of found lingering objects. When the advisory mode parameter is not used, this command removes lingering objects from the destination domain controller.

Additional information on Repadmin.exe is available here: https://technet.microsoft.com/en-us/library/cc736571(v=ws.10).aspx

1.2                 Repadmin /replsummary

As seen in the screenshot below repadmin /replsummary will give statistics for replication with replication partners.  The output also lists any errors that were encountered with replication. This is useful for getting an overview of any replication issues the DC is having.

You can also sort the output.  In the example below, the output is sorted by the largest delta since last replication.

1.3                 Repadmin /showrepl

As seen below repadmin /showrepl shows the replication status with all of the DCs replication partners and is sorted by the Naming Context that is being replicated.

One trick that can be used to get a more manageable output is to use repadmin to send its output to a CSV and the use PowerShell to convert the CSV to a GridView.  The command to do this is repadmin /showrepl * /csv | ConvertFrom-CSV | Out-GridView

The resulting output is in a manageable GUI.

In GridView you can sort and filter.  Below is an example of filtering on Number of Failures, so that I can easily see what failed.

1.4                 Repadmin /showutdvec

Replications changes are tracked through incrementing numbers called USNs.  There are times where you will want to know what knowledge each DC has about other DCs current state.  The up-to-dateness vector is the knowledge that a DC as about the current state of other DCs.  This information can be useful when trying to troubleshoot replication issues such as USN Rollback.  USN Rollback is when a DC is restored from an unsupported method such as a snapshot.  In that case the up-to-dateness vector would be much larger than the actual USN of the DC. Since, there is going to be some delay in replication you will notice some differences but the numbers should be relatively close.  For example, if you compare the up-to-dateness vector for DC01 across DCs you will notice the following: for itself DC01 has USN of 17347, DC02 has a USN of 17346 for DC01, and DC03 has a USN of 17346 for DC01.  So, we can see the numbers are relatively close and that DC01 potentially has one change that it needs to replicate to DC02 and DC03.

1.5                 Repadmin /showobjmeta

The /showobjmeta switch shows detailed information for attributes of an object.  It is most commonly used when comparing the output of the command from 2 DCs to see if they are in sync and the current status of the attributes.  Differences can be used to identify replication problems.

1.6                 Repadmin /syncall

Repadmin /syncall is used to force replication between domain controllers.  You can easily view options for the /syncall switch with the following command: repadmin /syncall /?

A normal use of repadmin /syncall is with the /AeP switch

1.7                 Repadmin /showmsg

The /showmsg switch is used to convert an error message you may receive as the result of a repadmin command and converts it to human readable text.

1.8                 Repadmin /viewlist

Repadmin /viewlist is used to get a list of domain controllers.

1.9                 PowerShell

PowerShell is an object oriented scripting language that allows enterprises to automate IT tasks.

Below is a conversion table that shows the PowerShell command that can be used in place of the Repadmin command.  So, why would you choose to use PowerShell?  The output of PowerShell commands are objects those objects can be filtered with properties, piped through other PowerShell commands and manipulated to many useful things including great control in how the data is presented to the user.

Command	PowerShell Cmdlet
Repadmin /FailCache	Get-ADReplicationFailure
Repadmin /Queue	Get-ADReplicationQueueOperation
Repadmin /ReplSingleObj	Sync-ADObject
Repadmin /ShowConn	Get-ADReplicationConnection
Repadmin /ShowObjMeta	Get-ADReplicationAttributeMetadata
Repadmin /ReplSummary	Get-ADReplicationPartnerMetadata
Repadmin /Showutdvec	Get-ADReplicationUpToDatenessVectorTable
Repadmin /SiteOptions	Set-ADReplicationSite
Repadmin /ShowAttr	Get-ADObject
Repadmin /Set Attr	Set-ADObject

Get-ADReplicationParnerMetadata is very similar to running repadmin /showrepl.  Without passing the output through another cmdlet the formatting is a bit different then to what you get with repadmin.

However, the advantage is that the output of the command are objects.  You can constrain your views to certain properties.

The other advantage is that you can pass objects through other cmdlets.  As seen here I am passing the output of Get-ADReplicationPartnerMetadata through Output-GridView.

Once in GridView you have the ability to sort and filter the data.

Here is another example of the usefulness of using PowerShell over repadmin.  In this example I take the output of Get-ADReplicationPartnerMetadata then passing it through Select-Object so that we can then limit what objects are presented in GridView.

Here we see the output of that command.

1.10             Replication Errors

Here is a list of replication errors you may come across in either the Directory Services event log or while running repadmin.

Event ID	Replication Error	Issue
1388		Lingering Objects
1988		Lingering Objects
2042		Lingering Objects
1925		DNS Lookup Issues or Connectivity Problems
2087		DNS Lookup Issues
2088		DNS Lookup Issues
1311		Replication Topology Issues
	8614	Tombstone lifetime exceeded
	8524	DNS Lookup failure
	8456	Server is currently rejecting replication requests
	8457	Server is currently rejecting replication requests
	8453	Access was denied
	8452	The naming context is in the process of being removed or is not replicated from the specified server
	5	Access is denied
	-21468930222	The target principal name is incorrect
	1753	There are no more endpoints available from the endpoint mapper
	1722	The RPC server is unavailable
	1396	Logon Failure The Target account name is incorrect
	1256	The remote system is not available
	1127	While accessing the hard disk, a disk operation failed even afer retries
	8451	The replication operation encountered a database error
	8606	Insufficient attributes were given to create an object

2              Troubleshooting Steps for Common Replication Issues

2.1                 Troubleshooting -21468930222  (The target principal name is incorrect)

On the DC that is the cause of the error, perform the following steps:

Step 1: Open Services.msc

Step 2: Configure KDC Service for Manual

Step 3: Stop the Service

Step 4: Restart the Domain Controller

Step 5: Open PowerShell as an Administrator

Step 6: Run: $cred = Get-Credential

Step 7: Enter Credentials and click OK

Step 8: Run, Reset-ComputerMachinePassword –Server <ServerName> -Credential $cred

Step 9: Restart the server

Step 10: Set the KDC service to Automatic, Start the service and click OK.

2.2                 Troubleshoot Replication Error 8606, Event ID 1388, and Event ID 1988

These issues are caused by lingering objects.  Lingering objects can be caused when a domain controller is taken offline for an extended period of time, does not replicate for longer than the tombstone lifetime, or is restored from a backup that is older than the tombstone lifetime. 

When an object is deleted it is put in a tombstone state.  After the tombstone lifetime passes (typically 180 days), DC run garbage collection and those tombstone objects are deleted.  If a DC was offline for the entire TSL and then were brought back online they may have objects that have since been deleted, tombstoned, and garbage collected.  Any objects that were deleted will still exist on that DC.  These objects go unnoticed until a change is made to that object then the DC attempts to replicate that object, and at that point that is where it is either re-introduced into the environment or if strict replication consistency is enabled, blocked. 

2.2.1   How to Determine TSL

Run the following command: dsquery * “cn=directory service,cn=windows nt,cn=services,cn=configuration,<Forest DN>” –scope base –attr tombstonelifetime

2.2.2   How to Remove Lingering Objects

2.2.2.1          Repadmin /removelingeringobjects

One way to remove lingering objects is to user repadmin with the /removelingeringobjects switch.  First you must identify a clean source of the partition.  The syntax of the command is repadmin /removelingeringobjects <Dest DC Name> <Source DC Guid> <Naming Context>.  So, in other words you need to identify the source DCs guid and the Naming Context you want to clean.  The naming context will be available in the Event 1388 or 1988 you receive in the event long.  Once you find a clean source you can obtain the guid by opening DNS Manager and opening up the _msdcs Zone and obtaining the CName record for the DC in question.

Below is an example of running the repadmin /removelingeringobjects command

You will receive an Event 1937 when the removal of lingering objects begins.

You will then receive an Event 1939 when removal completes.

2.2.2.2          Repadmin /rehost

An alternative to using repadmin /removelingeringobjects command is to unhost the partition so that the domain controller no longer has that partition and then rehosting the entire partition with a good source.

The repadmin syntax for unhosting the partition is repadmin /unhost <DC Name> <Partition Name>

You will receive an event an event 1658 when the removal begins.

You will receive an event 1660 when the removal completes

The syntax for rehosting the partition is: repadmin /rehost <Dest DC Name> <Partition> <Source DC Name>

2.3                 Troubleshooting Event ID 2042

Review event log for any 1988 or 1388 errors.  If found use the previous section to remove the lingering objects from the domain controller. 

Option 1: Re-hosting the partition that has not replicated

If the partition is a GC partition consider unhosting and rehosting the partition. Instructions for unhosting and rehosting are in the previous section called Repadmin /rehost

Option 2: Removing and then re-adding the domain controller to Active Directory

Another option is removing the DC from Active Directory and Re-promoting the Domain Controller

Step 1: Run Import-Module ADDSDeployment

Step 2: Run: Uninstall-ADDSDomainController –DemoteOperationMasterRole:$true –Force:$true

Step 3: Enter and confirm the new local password

Step 4:  Next you will need to run the Install –ADDSDomainController cmdlet.  Below is a sample that you can use.  You will need to modify the template to meet the requirements of your environment. 

Install-ADDSDomainController –NoGlobalCatalog:$false –CreateDnsDelegation:$false –CriticalReplicationOnly:$false –DatabasePath “C:\Windows\NTDS” –DomianName “fabrikam.com” –InstallDNS:$true –LogPath “C:\Windows\NTDS” –ReBootOnCompletion:$false –ReplicationSourceDC “DC01.fabrikam.com” –SiteName “Default-First-Site-Name” –SysvolPath “C:\Windows\SYSVOL” –Force:$true

Option 3: Enabling Replication with Divergent and Corrupt Partner

Due to the risk of adding lingering objects to Active Directory the final consideration should be enabling the following setting: Allow Replication With Divergent and Corrupt Partner. 

Step 1: To enable this setting run the following command on the domain controller:

repadmin /regkey <hostname> +allowDivergent

Step 2: Let replication complete

Step 3: Disable the setting with the following command: repadmin /regkey <hostname> -allowDivergent

2.4                 Troubleshooting Event ID 1311

Event 1311 is caused when there is not complete connectivity between domain controllers. There are a number of reasons there may not be complete connectivity. 

2.4.1   ISTG

The Inter-Site Topology Generator (ITSG) is responsible for building the replication topology.  So to determine what the scope of the connectivity issues it is important to identify the ISTGs that are logging 1311. 

To find the ISTGs in your environment you need to use ldp.exe

Below are the steps for locating the ISTGs:

Step 1: Launch ldp.exe

Step 2: When LDP opens, select Connection and then Connect…

Step 3: In the Connect dialog box, enter the name of a Domain Controller for the Server you want to connect to and then click OK

Step 4: Click on Connection and then click Bind…

Step 5: In the Bind dialog box, click OK

Step 6: Select the Browse menu and then select Search

Step 7: In the search enter the following:

Base DN: CN=Sites,CN=Configuration,<DN of Forest Root> (example: CN=Sites,CN=Configuration,DC=fabrikam,DC=com)

Filter: (CN=NTDS Site Settings)

Scope: Subtree

Attributes: Append the following to the attributes that are already listed: ;interSiteTopologyGenerator

Step 8: Click Run

Step 9: For each site you will then need to look for interSiteTopologyGenerator to determine the ITSG for each site.

2.4.2   BASL

By default, Bridge All Site Links (BASL) is enabled in Active Directory.  If your environment is not fully routed, then you will want to disable BASL.  By fully routed we mean each site can contact every other site.  If BASL is configured on a network which is not fully routed, the KCC will generate site bridges that cannot actually be reached. To determine if BASL is enabled launch Active Directory Sites and Services (dssite.msc). 

Expand Sites, then Inter-site Transports.

Right-click on IP and select Properties from the context menu

If Bridge all site links is enabled, there will be a check box next to it.  To disable BASL, uncheck the checkbox and click OK.

2.4.3   Site Link Bridges

If you disable BASL you can still bridge site links.  You would do that if you wanted two spoke sites to communicate directly if they could not communicate with the hub site.  In a hub and spoke configuration the cost of crossing to site links (bridging a site link) will typically be a higher then just connecting directly to the hub site.  So, ordinarily you would not have to worry about the Site Link Bridge being used instead of a direct site link.  That being said, there are not a whole lot of scenarios where you would need to create Site Link bridges.

The following steps will allow you to bridge two site links.

Step 1: Open the Active Directory Sites and Services MMC.

Step 2: Expand Sites and then expand Inter-site Transports

Step 3: Select New Site Link Bridge… from the context menu

Add at least two sites to the Site Link Bridge, give it a Name, and click OK

And the Site Link Bridge has been completed.

2.4.4   Verify that all Sites are in a Site Link

Step 1: Run the following command in a PowerShell Console: Get-ADObject –LDAPFilter ‘(objectClass=site)’ –SearchBase (Get-ADRootDSE).ConfigurationNamingContext –Property Name | Format-Table Name

Step 2: In another PowerShell Console run: Get-ADObject –LDAPFilter ‘(objectClass=sitelink)’ –SearchBase (Get-ADRootDSE).ConfigurationNamingContext –Property Name, Cost, Description, Sitelist | Format-List Name, Sitelist

Step 3: Verify that each site that was listed in Step 1 exists in one of the site lists returned in Step 2

If not all sites are contained in a site link that you need to determine what site link that site needs to be added to or if a new site link needs to be created.

And that is all I have for replication troubleshooting for today.

-Chris

Below is a blog I wrote quite a long time ago, but never posted. Hopefully, this is helpful.

Account Lockout Settings

The first thing that you will want to contemplate is your Account lockout threshold.  If you have this setting set two low this can lead to accounts being locked out.  This is because users may make mistakes when entering their passwords, applications may try to authenticate multiple times, as well as the fact that user may have a multitude of devices that have the credentials stored and will repeatedly attempt to log on. Microsoft’s general guidance on the Account lockout threshold is 10. If your Account lockout threshold is lower than that you should consider raising it.

How Account Lockout Occurs

Everytime a user enters the wrong password when logging on using a resource that authenticates to Active Directory, the badPWDCount attribute has a value that is incremented.  This value is incremented on the domain controller that the user is attempting to authenticate to.  The Domain Controller then retries that authentication attempt with the PDC, since PDC is urgently replicated all password changes.  If the authentication attempt on the PDC fails, the PDC increments its copy of the badPWDCount attribute for that user.  This structure allows the badPWDCount to increment even if different domain controllers are used for authentication.   Once the badPWDCount attribute reaches the Account lockout threshold the account will be locked out.

Troubleshooting Account Lockouts

Domain Controller Auditing

In order to troubleshoot accounts lockouts you will first need to have some auditing in place.

On Domain Controllers, you should have Audit account logon events configured to Audit Success and Failure attempts.

The events that you should be looking for on the Domain Controller are:

Event 4771, which is a Kerberos Pre-Authentication failure.  But be careful this event can be a little misleading since you Windows will intentially fail Pre-Auth the first time to get a list of the DCs encryption capabilities.

Key Information in this event:

Security ID and Accountname tell me which account failed Pre-Authentication.

Under Network Information we see the client address and port, so this can help us identify the source of the failed authentication.

Event 4740, which shows that an account has been locked out.

We see that the Caller Computer Name is listed, which lets us identify the source of the account lockout.

Server and client auditing and logging

On the server or client to which you are trying to authenticate to will need to have Audit logon events enabled for Success and Failure.

Events you will see on the server or client to which you are authenticating to you should look for the following events:

Event 4625, which shows you that an account failed to log on.

Key information in this event include:

Security ID, Account Name, Caller Process, Caller Process Name, Workstation Name, Source Network Address, and Source Port

So how do you find which accounts are locked out?

Step 1: Open the Active Directory Administrative Center

Step 2: Click on Global Search

Step 3: Select the checkbox for Users with enabled but locked accounts

Step 4: Click Add

Step 5: Ensure that the Scope is set to match the location you wish to search

Step 6: Click Search

So how do you determine the DC that the user was logging on to when they get locked out?

Step 1: Download the account lockout tools from here: http://www.microsoft.com/en-us/download/details.aspx?id=18465

Step 2: Launch LockoutStatus.exe

Step 3: Enter the Target User Name and Domain Name and click OK

Step 4: In the lockoutstatus tool view the Orig Lock column.

How do you find the server where the Account Lockout originated?

Option 1: Once you locate the DC where the lock originated you can view the 4740 event and view the Caller Computer Name if it is present.

Option 2: Alternatively you can view the 4771 event and the Client Address should identify the source of the Account Lockout.  It is important that you correlate this to 4740 event to ensure that you are looking at the correct Pre-Authentication failure.

Option 3: If you still cannot locate the source, you can enable Netlogon Diagnostic logging. The following article contains the steps used to enable this logging as well as Fix It scripts that will enable/disable it for you: https://support.microsoft.com/en-us/kb/109626.  All data will be logged to %windir%\debug\netlogon.log and netlogon.bak

In this example, I see that the user lockmeout attempted to authenticate to FCCA01 from the machine FCServer01.  Ultimately an 0xc0000234 was returned, which means the account was locked out.

How do I find out which process caused the account lockout?

On the server or client to which you are trying to authenticate to will need to have Audit logon events enabled for Success and Failure.

As illustrated in the example below the process ID and Name are listed in the event.

If you wish to dig deeper into this you can grab the process ID from the event.

In this example the Caller Process ID is 0x330

We can convert the hex into decimal by using calc.  Simply select Hex, enter 330 and then toggle the radio button to Dec.  In this example, this gives us 816 which is the PID of the process.

Then if you run tasklist /svc, you will get a list of PIDs and what processes are associated with them.

-Chris