Azure Privileged Identity Management (PIM)

Priviledged Identity Management (PIM) is an tool that allows you to securely manage Priviledged Identities in Azure. PIM allows you to assign which users are allowed to elevate to Priviledged Roles. Elevation can be time bound, limiting time that those accounts are elevated. Elevation can require approvals, which provides control over when elevations occur. Additionally, there is strong auditing and reporting on elevations, approvals, and assignments. In this article, I will cover how your organization can leverage PIM to securely Manage Priviledged Identities in Azure Active Directory. However, many of the concepts covered can be utilized for Privileged Identities in Azure. The difference is that these identities are delegated at the Subscription, Management Group, or Resource Group level.


As a side note, there is an aspect of my writing that you might find annoying. And if you do I apologize. I tend to capitalize words that represent aspects of technology. So, you will see me write Resource Group instead of resource group.


Our starting point is the Azure AD Portal. You can of use the Azure Portal. However, I find the AAD Portal less distracting.

Next you will want to Favorite Azure AD Privileged Identity Management, so that it shows up on the left-hand menu.

Next, click on Azure AD Privileged Management

Now, let’s become familiar with the UI. Tasks is where you will perform the majority of your work, once you have PIM configured. Tasks is also where your administrators will spend most of their time interfacing with PIM. Manage will be used primarily for setting up PIM. Activity is used for reviewing your activity in PIM. If you ever feel something strange is happening with your account, you can use this to determine if that is in fact the case.

Configuration
Click on Azure AD Roles to begin the configuration.

Here are your AD Roles that you can now configure. As you can see there are many roles. Some you will become familiar with, some you will not. No need to memorize every role. When you need to delegate access to perform a specific task, you can at that time investigate which role meets your needs.

We will use Global Administrator for an example of how to configure a role. So, locate that role either by scrolling or searching. Next, double click on Global Administrator to begin configuration.

Next, click on Settings to bring up, well the Settings : )

Here, we can see the Settings for the role. Click Edit to configure.

Activation Page
Activation maximum duration (hours)
This setting allows you to configure the duration of time for which a user will be elevated before they are “de-elevated”. Generally, you would want to set this at the average time a change normally takes for a Priviledged User. The maximum you should consider is a “typical” workday. So, generally 8 hours. So, many of us unfortunately do not have an actual workday that short, you have to draw the line somewhere to increase or maintain a certain level of security.


On activation require
This is where you can configure users to have to have MFA enabled to elevate. If the user has not previously configured MFA, they will need to register when they elevate for the first time. I personally think you would be making a huge mistake if you did not require MFA for elevation.


Require justification on activation
This requires the user to enter some text explaining why they are elevating. While it may seem like an inconvenience to users, it is actually in part to their benefit. If there are ever any questions on why they elevated, they can be quickly answered and justified with the information provided here.


Require ticket information on activation
In your environment you should without question require a change control entry prior to making changes to Azure AD. Most if not all change control systems leverage ticket numbers. For reporting and auditing purposes, it is extremely helpful if this is required. Again this protects the user, because if they are ever questioned on if they opened a change control entry for their change, they can reference it here.


Require approval to activate
So, this is a setting that definitely considers additional consideration as there are definitely some tradeoffs. The advantage to requiring approval is that it makes the environment secure. Because a user cannot act on their own to elevate. The disadvantage is what if a user needs to make an emergency change at 3:00 AM in the morning. Is the approver going to be awake to approve? Can you contact the approver? Will the approver be available at that time? If both the person that can make emergency changes and the approver are required to be available 24 x 7, or you otherwise have 24 x 7 coverage, then definitely utilize this setting. As with all of these settings these should be reviewed by your Information Security department, to ensure you meet the security requirements of your organization.

Assignment
Allow permanent eligible assignment
This allows the user to elevate, without their ability to elevate ever expiring. In other words, as you will see later, you can configure a period of time after which a user’s access needs to be removed. I would strongly advise against allowing permanent eligible assignment. As you are aware most organizations do a remarkably poor job of cleaning up accounts that should no longer have access. And this can be sort of a self-remediation for those accounts.


Expire eligible assignments after
As mentioned above this is a restriction of for how long the user has the ability to elevate. Your options at the time of this writing are 15 Days, 1 Month, 3 Month, 6 Months, or 1 year. Unless you have a lot of turnover you may want to go longer rather than shorter.


Allow permanent active assignment
So, based on what I mentioned above you may have been lead to believe that you would not want to Allow permanent active assignment. However, we have not discussed Break Glass accounts. These are Privileged accounts that persist in case no other accounts have access. These type of accounts are typically excluded from Conditional Access Policy. This is done so that if an administrator really misconfigures Conditional Access Policy, you still have a way to recover. So, for Break Glass Accounts you would want to have this configured. Which means you are really going to have two class of accounts Break Glass accounts and non-Break Glass accounts. Break Glass accounts would typically be permanent active assignments and non-Break Glass accounts would be temporary eligible accounts.


Expire active assignments after
If for whatever reason you decide to have temporary active assignments, then you can select how long before those active assignments expire. Your options at the time of this writing are 15 Days, 1 Month, 3 Month, 6 Months, or 1 year.


Require Azure Multi-Factor Authentication on active assignment
This setting does what it says that it does. If you are using active assignments for Break Glass accounts then you should not enable this setting. Any setting that interferes with a Break Glass account from logging in should not be configured. Obviously, extra controls need to be in place for these accounts. Some examples are restricting knowledge of the password, perhaps keeping in a physical or digital safe that is offline. Enabling auditing and especially alerting that sets of all sorts of alarms when someone logs in with the Break Glass account.


Require justification on active assignment. It would obviously be good to have a user have to give a justification when accepting an active assignment.

Notifications
I am not going to cover every option on this page as it is mostly self-explanatory. Basically, you can control whether the email alerts go to the Admin (Global Administrators), Assignee, and Approver. Additionally, you can additional recipients. And you can also configure the alerts so that Critical emails only are sent, which according to the page are: A mode where PIM only sends out emails requiring an immediate action.


And obviously you can complete configuration by selecting Update.

Assigning Roles
Now I am going to assign the Global Administrator role to a user.
Step 1: Starting from the PIM home, click on Azure AD roles

Step 2: Click on Roles

Step 3: Search for or locate the Global Administrator role

Step 4: Click Add assignments

Step 5: Under Select member(s), click on No member selected

Step 6: Search for or scroll to the user you want to assign this role
Step 7: Select the user and click Select

Step 8: Click Next or Setting

Step 9: Select Eligible or Active
Note: Previously I discussed the differences between Eligible and Active. However, for greater clarity I am going to use Microsoft’s verbiage to provide a second explanation.
Eligible assignments require the member of the role to perform an action to use the role. Actions might include performing a multi-factor authentication (MFA) check, providing a business justification, or requesting approval from designated approvers.
Active assignments don’t require the member to perform any action to use the role. Members assigned as active have the privileges assigned to the role at all times.
Assign eligible owners and members for privileged access groups – Azure Active Directory | Microsoft Docs
Step 10: If necessary adjust the assignment start and end date
Step 11: Click Assign

You can now view your assignment for this role

Delegation
So, roles such as Global Administrator are delegated at the Directory level. However, there is another level at which you can delegate. You can delegate at the Administrative Unit level. Administrative Units are basically a container if you will, that you add users or groups to, and then they can be controlled by a user who is delegated “permissions” to the Administrative Units. Some, like to compare Administrative Units to Organizational Units in Active Directory. Others do not prefer that comparison. I personally do not like to make the comparison because these are two different and distinct things.

The following roles can be delegated to an Administrative Unit:
• Authentication Administrator
• Groups Administrator
• Helpdesk Administrator
• License Administrator
• Password Administrator
• SharePoint Administrator
• Teams Administrator
• Teams Devices Administrator
• User Administrators
When assigning roles that can be assigned to Administrative Units, you have the option to select Administrative unit or Directory.

Then in order to select the Administrative unit, under Selected scope click on No scope selected

Then either search for or scroll to the Administrative Unit and click Select

Alternative Method to Assign Roles
Step 1: When in the Privileged Identity Management view, select Assignments

Here you will see all of the Eligible assignments
Step 2: Click Add assignments

Step 3: Select the role you want to assign. In this instance I selected Global Administrator
Step 4: Click on No member selected
Step 5: Choose a member and click Select

Step 6: Select Next

Step 7: Decide on whether to make the assignment Eligible or Active. Remember Active assignments should be reserved for Break Glass accounts.

The account is now eligible to elevate to Global Administrator

Elevating
Below is the process for the user to elevate
Step 1: Starting from the Privileged Identity Management view, click on My roles

You will now see your Eligible assignments
Step 2: For the role you want to elevate to, click Activate

Step 3: Enter whatever information is required to elevate. In this case a Reason is required. Enter a Reason and click Activate. Optionally, configure a Custom activation start time.

Once your elevation is approved, it will show up under Active assignments

Approving requests
Step 1: To approve requests click on Approve requests
Step 2: Locate the request you want to approve
Step 3: Select the elevation you want to approve
Step 4: Click Approve

Step 5: Enter a Justification and click Confirm

Self Audit
To review your own activity in PIM, select My audit history

You can then view and search activity

Privileged Access Groups
Privileged Access Groups is a feature that is currently in Preview, or at least at the time of this writing. This feature allows you to assign Roles to a group. So, for example if you have a position in your organization where a user needs two Azure AD roles. You can create a Privileged access group to have those two roles. Then when you want to assign a user to those two roles for the position, you just add that user as a Member of the Privileged access group.


Assigning roles
First step is when creating a group is to select Yes for Azure AD roles can be assigned to the group

One you create the Group, you can go to Azure AD Portal and select Groups

Next you click on the Group you created. In this example Branch User Administrator

Next you click on Assigned roles

Now to add your roles, click Add assignments

Select the role you want to add and click Next

Determine if you want the role to become Eligible or Active. Also determine if you want to make the role Permanently eligible or Permanently Active. And adjust the assignment dates if necessary.
Finally click Assign

So, I then went and assigned a second role

Provisioning members
To provision members click on Privileged access (Preview)

Choose Member as the role. Then click on No member selected. Select a Member and click on Select.

Click Next

Then choose whether you want to make the Member Eligible or Active. You make the user Eligible if you want them to activate their membership in the group. If you want to just provision the user then click Active.


Finally, click Assign

Activate your Membership in the Privilege access group
If you a user was made eligible instead of active, they will need to accept the assignment.


To do this the user would start from the Privileged Identity Management view


The user would then click on My roles

Then click on Privileged access group (Preview)

Then the user would click Activate to activate their membership

The user would need to enter a Justification and optionally a Custom activation start time
Then the user would click Activate

Once the user is now a member the user can choose to elevate to the roles that are assigned to that group.

Viewing Pending Requests
You can view pending requests

From the Privileged Identity Management view click on Azure AD roles

Then click on Pending requests

-Chris

AGPM Configuration and Use

AGPM Provisioning

There are 4 Roles within AGPM:

AGPM Administrator: The AGPM Administrator has full control of AGPM including the AGPM archive.

Reviewer: The reviewer simply has the ability to view GPOs, they can also run comparisons on GPOs.

Editor: In addition to the reviewer permission, Editors also have the ability to Check In GPOS, Check Out GPOs, Edit GPOS, and request creation of a GPO.

Approver: In addition to Reviewer and Editor capabilities, the Approver can approve the GPO Creation, GPO Deployment.  Approvers can also Create GPOs and Deploy GPOs of their own creation without approval.

Provisioning Decision Point

There really are two options for provisioning access.  The first is to create Active Directory security group and grant those groups access to a role within AGPM.  For example, you could create an AGPM Admins group in AD and then provision that group Full Control within AGPM.  Then to provision access to the AGPM Administrator role you would simply add users to the AGPM Admins group. The alternative approach is that you just provision access to users within AGPM.  From a security perspective, I prefer the second method as it reduces the number of users or groups that will have the permissions necessary to provision access to AGPM. In the examples below I will be demonstrating the second technique for granting access, specifically using AGPM to directly provision access to user.

Creating AGPM Administrator

Temporarily add the user account that you would like to add as the first AGPM Administrator to the group that is the AGPM Archive Owner.  You will need to do remove this user once provisioning of the AGPM Administrator account is complete.

To provision permissions:

  1. Open GPMC.msc
  2. Expand Forest, then Domain, then “Domain Name”. 
  3. Click on Change Control.
  4. In the right-hand pane of AGPM ensure that you have navigated to the Domain Delegation tab.
  5. Click Add…

Enter the name of the user for which you are provisioning access, then click the Check Names button.

Once the name of the user or group has been resolved, click OK.

If creating and Administrator select Full Control as the role.  Otherwise, select Editor, Approver, or Reviewer as appropriate. Then click OK.

Controlling Existing GPOs

Prior to using AGPM with GPOs that already exist in your environment, you will need to give the AGPM Service Account full control of exiting GPOS.  There is a script that can be used to automate much of the configuration of AGPM.  It is located here: https://gallery.technet.microsoft.com/scriptcenter/AGPM-Quick-Configuration-a7f42d46.  To give the AGPM Service Account permissions to existing GPOS, you can use the following portion of the aforementioned script:

import-module activedirectory

$agpmserviceaccountname=”svcAGPM”

get-gpo -all | foreach { Set-GPPermissions -Guid $_.id -PermissionLevel GpoEditDeleteModifySecurity -TargetType User -TargetName $agpmserviceaccountname }

Once you have given the AGPM Service Account access to the GPOs, you can put them under Control of AGPM.  Once the GPOs are under the control of AGPM, all group policy management can be done via AGPM.  The following steps can be followed to take control of existing GPOs:

  1. Open GPMC.msc
  2. Expand Forest, then Domain, then “Domain Name”. 
  3. Click on Change Control.
  4. In the right-hand pane of AGPM ensure that you have navigated to the Contents tab.
  5. Then within the Contents Tab, navigate to the Uncontrolled Tab.
  6. Select or Multi-Select GPOs that you want AGPM to Control.
  7. Right-click and then select Control… from the context menu.

You will then be prompted for a comment.  Type an appropriate comment and click OK.

Then you will see the progress of taking control.  When the actions have completed, click Close.

You will now see the GPOs showed up under the Controlled tab.

Implementing Least Privilege

To ensure that the ability to modify GPOs is restricted to those that are provisioned access through AGPM, new GPOs that are created must have their permissions restricted. You will want to restrict the Production Delegation. To do this you will want to ensure that just the AGPM Service Account and System have the permissions: Edit settings, delete, modify security. All other accounts will need to either be removed or have their permissions changed to Read.

To add the AGPM Service account to the Production Delegation navigate to the Production Delegation tab. Enter the name of the Service Account.  Then click Check Names to validate the account name.  Then click OK.

Then when the Add Group or User dialog box opens, ensure that the Role is configured to Edit Settings, delete, modify security. Then click OK.

For Domain Admins, Enterprise Admins and any other accounts aside from System and the AGPM Service Account that have Edit settings or Edit settings, delete, modify security settings you will need to change the permission to Read.

To change permissions to read, right-click on the security principal and the select Read from the context menu.

When prompted with Do you want to change the permission on this group or user to Read?, click OK.

Below is a screenshot of the Production Delegation tab after it is locked down.

This locked down configuration will apply to existing GPOs once they are controlled by AGPM.

Group Policy Retention

AGPM allows you to control how much of the GPO history is retained.  If you wish to limit how much of that history is retained you can navigate to the AGPM Server tab. Next, Check the option Delete old versions of each GPO from the archive. Retain only the current version and the preceding.  Then enter a number for how many of the preceding GPOs you would like to maintain in the archive and click Apply.

AGPM Client Settings

You can configure what AGPM Server the AGPM Client connects to when you install the AGPM Client.  However, you can also control which AGP Server the client connects. Navigate to \User Configuration\Policies\Administrative Templates\Windows Components\AGPM. 

Open the setting AGPM: Specify default AGPM Server (all domains). Change the configuration Enabled.  For the setting Default AGPM Server for all domains, enter the DNS Alias for the AGPM Server and the port number in the format <DNS Alias>:<Port Number>.

Basic Administrative Tasks

AGPM provides several useful tools for getting additional information about GPOs.

History

If you right-click on a GPO you can you choose History from the context menu.

You can then view the entire history of the GPO, including Creations, Check Outs, Check Ins and Deployments.

On the Unique Version tab you can view a less cluttered view that only displays unique version of the GPO.

Also, you can right-click on any version of the GPO and have the ability to view Settings, perform Comparisons, Deploy, and more.

Viewing Settings

Within AGPM it is simple to view a settings report.  If you right-click on the GPO, you can select the Settings from the context menu.  You then have the option to view:

HTML Report: HTML Report of all settings.  This is the most user-friendly view of the settings.

XML Report: An XML output of the report which may be useful if you somehow wish to convert or manipulate the output of the report.

GPO Links: Lets you view where the GPO is linked.

Below is an example of an HTML report.

Comparing GPOs

One of the more useful features of AGPM is the ability to compare two GPOs. 

To Compare two GPOs, multi-select two GPOs.  Right-click and then from context menu select Differences.  Then you have the option to view an HTML Report or an XML Report.

Below is an example of an HTML Report, highlighting the differences between the two GPOs.

Creating New GPO

The walkthrough below, shows how an Editor can send a request to create a new GPO.

Editor would right-click under Controlled GPOs and select New Controlled GPO… from the context menu.

The Editor would then be prompted to Submit New Controlled GPO Request. The Editor would then fill out the form.  The Editor has the option to Create in archive and production or Create in archive only. Also, the user can create the GPO from a Template. The Editor would click Submit to submit the request to the approver.

The Editor will view the progress of the submission and then click Close when complete.

Below is an example of an email that the approver will receive.

The GPO will then go into the Pending state.

The Approver would then right-click on the GPO and the select Approve… from the context menu.

The approver will then be prompted for a comment, and then choose whether or not to approve the request.

Once the approval is complete the Approver clicks Close.

Editing and Deploying a GPO

An Editor can Edit and Deploy a new GPO as long as the Approver approves the Editors actions.  Below are the steps required by the Editor and Approver to edit an existing GPO and deploy it to production.

Editing

The Editor would right-click on the GPO they wish to edit and select Check Out… from the context menu.

Then the Editor would add a comment and Click OK.

When the Check Out is completed the Editor would click Close.

Then the Editor would right-click on the checked out GPO and select Edit from the context menu.

The Editor can the modify the GPO.  The screenshot below is just an example of an Editor modifying an existing GPO.

Once the Editor has completed the edits, they would then right-click on the GPO and select Check In… from the context menu.

The Editor would then add a comment in the Check in GPO dialog box and click OK.

Finally, when the Check In is complete the Editor would click Close.

To Deploy the GPO, the Editor would then right-click on the GPO and click Deploy…

The Editor would then complete the Submit Deploy GPO form, in part to notify the Approver. Upon completion of the form the Editor would click the Submit button.

Upon completion of the submission, the Editor would click Close.

Approval

Once the Approver receives notification the approver would navigate to the Pending Tab.  The Approver would then right-click on the pending GPO and click Approve… if the approver wishes to approve the GPO.

The Approver would then add a comment to the Approve Pending Operations dialog box, and click Yes.

Upon completion of the approval the Approver would click Close.

Deletion and Recovery of GPOs

AGPM uses a familiar concept called a Recycle Bin.  If a GPO is deleted accidentely or if a deleted GPO needs to be restored it can be un-deleted from the Recycle Bin.

Deleting a GPO

To Delete a GPO simply right-click on the GPO and from the context menu click on Delete…

Then decide whether you want to delete the GPO from the archive or the archive and production.  Based on your decision select either Delete GPO form archive only or Delete GPO from archive and production.  You would then add a comment and click OK.

You will then be asked Are you sure you want to delete the selected GPO?.  Click OK to complete the deletion.

Recovering a GPO

To recover a deleted GPO, navigate to Recycle Bit tab.  Right-click on the GPO you wish to restore and choose Restore… from the context menu.

Enter a Comment and click OK to complete the restore.

Once the restore completes, click Close.

I hope this is userful.

-Chris

PKI Resources

I just came across this website that has a bunch of extremely useful white papers on PKI. I am so excited, that I had to share with those that are not aware that this exists: https://www.sysadmins.lv/blog-en/microsoft-whitepapers-public-key-infrastructure-and-active-directory-certificate-services.aspx

I have visited other pages on that site as Vadims has reposted many blogs that are now hard to find due to Microsoft archiving a lot of the content. Vadims has done a lot of great work here.

-Chris

LDAPS / Domain Controller Certificates

Background and References

So, today I’m going to discuss implementing certificates for Secure LDAP on Active Directory Domain Controllers.

First of all, some helpful links. This article talks about the requirements for secure LDAP as listed below:

  • The LDAPS certificate is located in the Local Computer’s Personal certificate store (programmatically known as the computer’s MY certificate store).
  • A private key that matches the certificate is present in the Local Computer’s store and is correctly associated with the certificate. The private key must not have strong private key protection enabled.
  • The Enhanced Key Usage extension includes the Server Authentication (1.3.6.1.5.5.7.3.1) object identifier (also known as OID).
  • The Active Directory fully qualified domain name of the domain controller (for example, DC01.DOMAIN.COM) must appear in one of the following places:
  • The Common Name (CN) in the Subject field.
  • DNS entry in the Subject Alternative Name extension.
  • The certificate was issued by a CA that the domain controller and the LDAPS clients trust. Trust is established by configuring the clients and the server to trust the root CA to which the issuing CA chains.
  • You must use the Schannel cryptographic service provider (CSP) to generate the key

Enable LDAP over SSL – Windows Server | Microsoft Docs

This article goes into detail and covers many of the topics I will cover in this blog posting:

LDAP over SSL (LDAPS) Certificate – TechNet Articles – United States (English) – TechNet Wiki (microsoft.com)

Deciding on a Certificate Template

So, as seen above the most significant requirement is that the Secure LDAP certificate have Server Authentication as it’s purpose. But, there are other reasons why you may have  a certificate on a Domain Controller such as for supporting services like Smart Card Logon or Windows Hello for Business (WHfB). So, you may want some additional application policies supported in the certificate you are going to issue to Domain Controllers. There are 3 certificate templates designed for use on Domain Controllers. Keep in mind technically you could use a Web Server Certificate Template to support LDAP over TLS. But the section above will provide reasons why to use one of the three templates designed for use on a Domain Controller.

Certificate Templates

Originally, there was a Domain Controller certificate template (Windows Server 2000) that is a version 1 template, then in Windows Server 2003 the Domain Controller Authentication certificate template was released, and finally in Windows Server 2008 the Kerberos Authentication certificate template became available. The latter two are version 2 templates by default.

Application Policies

The table below shows the Application Policies (purposes) for the 3 templates. Of course you can always duplicate these templates and add or remove whatever Application Policies that you want to add or remove.

Certificate TemplatePurpose 1Purpose 2Purpose 3Purpose 4
Domain ControllerServer AuthenticationClient Authentication  
Domain Controller AuthenticationServer AuthenticationClient AuthenticationSmart Card Logon 
Kerberos AuthenticationServer AuthenticationClient AuthenticationSmart Card LogonKDC Authentication

Subject Alternative Names

Additionally, the different templates come with a different Subject and SAN configuration. The table below displays the SANs available in the Certificate Templates. I am not concerned with the subjects, because applications like TLS will ignore the subject if the SAN is present and populated.

Certificate TemplateSAN 1SAN 2SAN 3
Domain ControllerDS Object GuidDC Name (DNS)
Domain Controller AuthenticationDC Name (DNS)
Kerberos AuthenticationAD Domain (DNS)AD Domain (Shortname)DC Name (DNS)

So, the typical SAN for a Domain Controller certificate will look like:

Other Name:

     DS Object Guid=04 10 59 5a 08 29 a7 9a 00 43 a2 75 f3 62 6e aa 62 0b

DNS Name=FCDC01.fourthcoffee.com

The typical SAN for a Domain Controller Authentication certificate will look like:

DNS Name=FCDC01.fourthcoffee.com

And finally, the SAN for a Kerberos Authentication certificate will look like the following:

DNS Name=FCDC01.fourthcoffee.com

DNS Name=fourthcoffee.com

DNS Name=FOURTHCOFFEE

As you see the Kerberos Authentication certificate has the most Application Policies and SANs, and hence it is most likely to support almost any application you need to support, both now and in the future. So, this is the template that you would use in most scenarios.

Deployment Scenarios

There really are 3 deployment scenarios. Depending on your environment it is possible that you could utilize all 3 if some of your domain controllers have other certificates installed that you need to continue to use. These include Autoenrollment using Certificate Template Supplied Names, Using Custom SANs with Automatic Renewal, Manual Deployment of Certificates to the NTDS Store.

Autoenrollment

Most of the configuration options use autoenrollment, so I am going to briefly describe autoenrollment and how to deploy autoenrollment to domain controllers, here. If you would like more information on autoenrollment, I have a video that covers this topic.

Basically, this will be an abbreviated discussion of Autoenrollment. Autoenrollment allows automatic enrollment an automatic renewal of certificates. The autoenrollment itself has some additional functionality, but I most likely won’t discuss that in this posting. To implement autoenrollment there are many requirements, from a certificate template perspective.

The steps below can be used to implement Autoenrollment for Domain Controllers. This walkthrough covers creating a new GPO on the Domain Controllers container.

Step 1: Open the Group Policy Management Console (GPMC.msc) as a user that can create new GPOs and link them to the Domain Controllers container.

Step 2: Right-click on the Domain Controllers OU and from the context menu select Create a GPO in this domain, and Link it here…

Step 3: Give the new GPO a Name and the click OK

Step 4: Right-click on the new GPO and select Edit from the context menu

Step 5: Navigate to Computer Configuration\Windows Settings\Security Settings\Public Key Policies

Step 6: Locate and open the following setting: Certificate Services Client – Auto-Enrollment

Step 7: Change the Configuration Model to Enabled

Step 8: Enable the settings Renew expired certificates, update pending certificates, and remove revoked certificates and Update Certificates that use certificate templates

Autoenrollment using Certificate Template Supplied Names

Configure Autoenrollment

See the Autoenrollment section above.

Configuring the Certificate Template and the CA

So, if you are happy with the SANs that the Kerberos Authentication template provides, and you do not have Server Authentication certificates on any of your domain controllers. Then congratulations, you get to use the easiest option. The easiest option is deploying the Kerberos Authentication certificate template with Autoenrollment.

Step 1: Open the Certification Authority MMC (certsrv.msc)

Step 2: Navigate to Certificate Templates

Step 3: Right-click on Certificate Templates and select Manage from the context menu

Step 4: Right-click on the Kerberos Authentication Certificate Template and select Duplicate Template

Step 5: Navigate to the General Tab and name the Certificate Template and click OK

Step 6: Return to the Certification Authority MMC

Step 7: Right-click on Certificate Templates and from the context menu select New and Certificate Template to Issue

Step 8: Select the Certificate Template that was just created

The template is now available for enrollment

If you want to test enrollment and not wait for the autoenrollment client to run, you can login to the DC and run: certutil -pulse

The certificate should now be installed on the DC

Using Custom SANs with Automatic Renewal

So, the process for using custom SANs requires an initial manual enrollment. However, you can use a PowerShell cmdlet for the initial enrollment allowing you to potentially automate the initial enrollment. The Kerberos Authentication Certificate Template as mentioned above puts the DC FQDN and the Domain DN and NETBIOS name in the certificate. In the example below, we are going to request these and in addition to these SANs we are going to request the DNS name LDAPS.<Domain>. In my example, the domain is FourthCoffee.com, so the custom SAN will be LDAPS.fourthcoffee.com.

Configure Autoenrollment

See the Autoenrollment section above.

Configure the Certificate Template

Configuring the Certificate Template and the CA

Step 1: Launch certtmpl.msc

Step 2: Right-click on the Kerberos Authentication certificate template and select Duplicate Template from the context

Step 3: Give the certificate template a unique name, then click Apply

Step 4: Navigate to the Compatibility tab

Step 5: Change the Certification Authority to Windows Server 2012

Step 6: Acknowledge the resulting changes click OK

Step 7: Change Certificate recipient to: Windows 8 / Windows Server 2012

Step 8: Acknowledge the resulting changes, by clicking OK

Step 9: Click Apply

Step 10: Navigate to the Subject Name tab and change the setting to Supply in the request

Step 11: When prompted about the security concerns, click OK. Note: From a security perspective you really should require Certificate Manager approval when allowing the requester to supply the subject name. The limitation is if we did that in this situation we would be unable to automatically renew the certificates. A mitigation could be to continually review issued certificates and make sure the identities requested make sense and do not violate any security policy.

Step 12: Click OK

Step 13: Go to the Certification Authority MMC, and on the Certificate Templates container right-click and select New and then Certificate Template to Issue

Step 14: Select the certificate template you just created and click OK

The template should now be available on the CA

Manually Requesting the Initial Certificate

So, there are some options here. Of course manually requesting the certificate on each DC is not a scalable solution. However, since this request can be done via PowerShell this enrollment can be initiated by a Script that is initialized by whatever configuration management software you use for Domain Controllers.

Run the following command: Get-Certificate -Template <Certificate Template you created> -DnsName <list of DNS names, separated by a comma> -CertStoreLocation cert:\LocalMachine\My

AN example would be: Get-Certificate -Template “OfflineKerberosAuhentication” -DnsName FCDC01.fourthcoffee.com,FourthCoffee.com,FourthCoffee,LDAP.fourthcoffee.com -CertStoreLocation cert:\LocalMachine\My

You will now see the certificate in the Computer Certificate Store

Optional Testing Steps for Test Environment

If you are setting this up in a pre-production environment and want to verify the autoenrollment works, follow these steps.

Step 1: Just open up the Certificate Template MMC and then right-click on the template and select Reenroll All Certificate Holders and this will cause DCs that have received a certificate to renew the certificate.

Step 2: Run certutil -pulse

Step 3: Log on to one of the Domain Controllers and verify the certificate has been renewed

Manual Deployment of Certificates to the NTDS Store

One issue that can arise is when Domain Controllers have more then one certificate with the Application Policy of Server Authentication. In this case the first certificate that has Server Authentication will be used. This can lead to undesired certificate selection. If there are multiple Server Authentication certificates you can force the selection of the desired certificate by putting the certificate in the NTDS store. The steps below will cover how to deploy certificates to the NTDS store. The disadvantage to putting certificates in this store is that it is a very manual process.

Configuring the Certificate Template and the CA

Step 1: Open the Certificate Template MMC

Step 2: Right-click o the Kerberos Authentication certificate template

Step 3: Select Duplicate Template from the context menu

Step 4: Name the certificate template and the click Apply

Step 5: Remove Autoenroll permissions from Enterprise Read-only Domain Controllers

Step 6: Remove Autoenroll permissions from Domain Controllers

Step 7: Remove Autoenroll permissions from ENTERPRISE DOMAIN CONTROLLERS

Step 8: Navigate to the Request Handling tab and select Allow private key to be exported

Step 9: Open the Certification Authority MMC

Step 10: Navigate to Certificate Templates

Step 11: Right-click on Certificate Templates and from the context menu select New and then Certificate Template to Issue

Step 12: Select the certificate template that you created and click OK

The Certificate Template is now on the CA

Requesting the Certificate

Step 1: Open certlm.msc on the Domain Controller

Step 2: Right-click on Personal or if it exists the Certificate folder underneath Personal

Step 3: From the context menu select All Tasks and the Request New Certificate…

Step 4: This will open the Certificate Enrollment wizard

Step 5: Click Next

Step 6: On the Select Certificate Enrollment Policy page, click Next

Step 7: On the Request Certificates page of the wizard, select the certificate template you created

Step 8: On the Certificate Installation Results page, click Finish

Exporing the Certificate

Step 1: Locate the certificate

Step 2: Right-click on the certificate and from the context menu select All Tasks and then Export…

Step 3: When the Certificate Export Wizard opens click Next

Step 4: On the Export Private Key page of the wizard, select Yes, export the private key

Step 5: Deselect Include all certificates in the certification path if possible and select Delete the private key if the export is successful

Step 6: Click Next

Step 7: Select Password and enter a password

Step 8: Click Next

Step 9: On the File to Export page of the wizard, click Browse…

Step 10: Enter a name for the file and click Save

Step 11: Click Next

Step 12: On the final page of the wizard, click Finish

Step 13: Click OK

Importing the Certificate Into the DS Store

Step 1: Open mmc.exe

Step 2: Click on File and then Add/Remove Snap-in…

Step 3: Select Certificates and then click Add

Step 4: Select Service Account and then click Next

Step 5: Keep Local Computer selected and then click Next

Step 6: Select Active Directory Domain Services, and click Finish

Step 7: Click OK

Importing the Certificate

Step 1: Right-click on NTDS\Personal

Step 2: Select All Tasks and then Import…

Step 3: When the Certificate Import Wizard opens, click Next

Step 4: On the File to import page of the wizard, click Browse…

Step 5: Browse to the PFX file you previously created and click Open

Step 6: Click Next

Step 7: Enter the password and click Next

Step 8: On the Certificate Store accept the default and click Next

Step 9: Click Finish to complete the wizard

Step 10: Click OK

The certificate with now be in the DS Store

Superseding Certificate Templates

One thing I intentionally left out is superseding Certificate Templates, because it may not apply in situations where you have not issues certain types of certificates. But if you have previously issued Domain Controller or Domain Controller Authentication certificates you will want to supersede them. When you do this the previously issued Domain Controller and Domain Controller Authentication certificates will be archived on the Domain Controllers.

To supersede the Domain Controller and Domain Controller Authentication certificates, follow these steps while creating your certificate templates in the previous sections:

Step 1: Navigate to the Superseded Templates tab

Step 2: Select Domain Controller and Domain Controller Authentication certificate templates and click OK

Step 3: Click OK

-Chris

Offline Root CAs / Two Tier Hierarchy

Background

I have come across many questions in regard to whether when deploying a PKI if a two tier PKI hierarchy and whether an offline root should be used.

The answer is yes an offline Root should be used in pretty much all environments. So, why? Some might ask, I have a super small environment, do I really need to this. The answer is, yes. So, again why?

First what is the reason for having an offline Root which pretty much guarantees that you would have to then setup a two or three tier hierarchy?

Root CA

The Root CAs main role is to be the trust anchor for the PKI hierarchy. In other words, the self-signed certificate that the Root CA generates is distributed to all devices that will need to trust the PKI. So, when you have a Root CA and Issuing CAs are subordinate to that CA, you only have one certificate to distribute. So, no matter how many issuing CAs you decide to deploy, you only have one Root certificate to distribute. Some applications may require you to install the subordinate CAs, but many applications can fetch the subordinate CAs from the AIA repository.

Two Tier Hierarchy?

So, let’s say you have a small environment, and you are only planning to install one Issuing CA. Do you still need an offline root and two-tier hierarchy? Yes. The reason you still want to have this, is because it allows you to later expand your PKI if necessary. You can later add additional issuing CAs that terminate in the same trusted root. And while you may only need one CA at this time, it is impossible to anticipate the future needs of your organization, so by deploying a two tier PKI you enable your PKI to be scalable. If you do not install a two tier hierarchy you are limited. Because if you later decide to expand your PKI, you either need to rebuild with an offline Root or manage multiple Issuing CAs that are also Root CAs. So, to avoid these future limitations you still would want to deploy a two tier PKI, or in some edge case scenarios a three tier PKI.

Offline Root

What is an offline root? An offline Root is a Root CA that is disconnected from the network for it’s entire “lifetime”. The OS should be installed without network connectivity, and the machine should never be connected to a network, not even for a minute.

What are the inconveniences of having an offline Root?

The main problem is that you periodically must update the CRL, then copy the CRL to the CRL Distribution Points (CDP) repositories. And if this CRL expires, it can cause outages in your environment.

So, do you need to make your Root CA and offline CA?

 Yes. As mentioned earlier the Root CA is the trust anchor of the PKI. Also, you want to put as much security around CAs as possible since if they are compromised certificates that are trusted in your environment can be issued. And if a malicious actor compromises your CA, they can impersonate identities in your environment. So, the security guidance is to have CAs offline if they infrequently issue certificates. Issuing CAs are online because they issue certificates very frequently. Root CAs and Policy CAs are kept offline because they issue certificates very infrequently.

To explain this further, let’s us a fictional scenario. Let’s say you have an Offline Root CA, and you have an Issuing CA that only issues Code Signing certificates for your organization. And let’s say due to the sensitivity of these types of certificates you want to provide maximum security for the Code Signing CA. And your development group typically only requests Code Signing certificates 5 time a year. In this scenario you would keep the CA offline and only bring it online 5 times a year. And if possible, like the Root CA you would set it up as a standalone CA and keep it disconnected from the network (offline). Sure, this makes administration a bit harder since you have to sneaker-net the requests to the CA and the certificates off the CA. But it dramatically increases the security of the CA, so it is worth the trade off.

So, to answer the question “Why have an offline root?”

  • Because the CA infrequently issues certificates and CRLs
  • Because compromise of the CA can result in a compromise, or assist in the compromise of your environment (Possibly including Active Directory)
  • If it is not connected to the network, it cannot be attacked from the network
  • It is best practice and if your PKI gets audited you will be flagged for not having it offline
  • If you later have to convert the CA to an offline CA, you will have to potentially revoke previously issued certificates and renew the Root CA certificate, if not have to redeploy the Root CA altogether

Why should I not deploy a single tier PKI?

  • Having the Root CA online violates best practice, due to the reasons above
  • The PKI is not scalable (We want to future proof deployments, so we can avoid having to redeploy the PKI in the future)
  • If your PKI is audited, you will be flagged during the audit, since your Root CA is online

-Chris

PKI Tools: Certutil -url

Hello,

There are several tools you can use to troubleshoot certificate validation. The best tool is certutil -verify -urlfetch. That tool is the best because it checks all certificates in the chain and gives us a lot of validation information. You can find information on how to use that tool, at the end of the following blog: Certificate Validation – xdot509.blog.

However, there is a GUI tool that can be used: certutil -url. The disadvantage to this tool is that it just checks one certificate at a time and mostly provides information on the accessibility of CDP, AIA, and OCSP. So, if you just want to check that your repositories are accessible this is a good way to perform that check. It is also easier to understand the output form this tool, which can make it easier for you to communicate your issue to your colleagues when you are troubleshooting.

To run the command, you first need a certificate file. So, you can export the certificate and save it to a file location. If it is a website you are having trouble with you can export the cert from the lock icon in the browser.

The syntax of the command is: certutil -url <Certificate File Name>, where <Certificate File Name> is the certificate file you previously exported. This will launch the GUI.

The tool is pretty self-explanatory, but the graphic below provides some additional information.

If you click Certs (from AIA) and click Retrieve it will verify that it can retrieve the certificates from the AIA repository and that they are the correct certificates.

If you select CRLs (from CDP) and click Retrieve it will verify that it can retrieve the certificates from the AIA repository and that they are the correct certificates.

And finally, you can select OCSP (from AIA) and click Retrieve it will verify that OCSP is working

Hopefully, this helps.

-Chris

Certificates

1              Certificates

1.1                 Background

This blog posting covers certificates. The purpose of this post is for those who would like to better understand certificates, their structure, and their contents. This blog post also covers basic management of certificates.

1.2                 What is a certificate?

A certificate is a digital document that binds an identity to a public key.  The identity contained in a certificate can be associated with any number of things including computers, users, and services such as a web site.  Certificates also contain additional information that makes the certificate more useful in many respects. 

1.3                 Keys

Public Key cryptography makes use of two related keys.  There is a public key and a private key.  Any data that is encrypted with the public key can be decrypted the private key.  Also, any data that encrypted with the private key can be decrypted with the public key. The public key is distributed via certificates and the private key is protected on a computer or on a cryptographic device such as a smart card.

1.4                 Validity Periods

Certificates also have a period of time for which they are valid.  In other words, a certificate is valid for a certain period of time. This built in expiration mechanism helps increase the security of certificates.  This is due to the fact that if a certificate is compromised and that compromise is not detected it will eventually expire and become invalid. 

1.5                 Digital Signature

A certificate is digitally signed by the CA that issued the certificate.  This signature can be validated to verify that the certificate was in fact

1.6                                                                                                                                                             Viewing Certificates in the Windows GUI

There is some basic information you can get from a certificate by viewing it in the Windows UI.  However, for most troubleshooting you will want to look at the certificate fields which will be discussed in the next section.

1.6.1   General Tab

In the General Tab you can look at a description of the purposes of the certificate.  You can also get the identity of the subject and the issuer.  You can view the validity period of the certificate.  If there is a private key associated with the certificate available in the certificate store you will see that as well.

1.6.2   Details Tab

In the Details Tab you will see all of the fields listed in the certificate as well as some additional information supplied by the Operating System.  We will discuss these fields in the next section.  You can also use the Copy to File… button to export a certificate.

1.6.3   Certification Path Tab

The Certification Path Tab shows the chain of the certificate, starting at the current certificate and going up as far as the Root CA.  There is also a Certificate status section that can be used for identifying issues with the certificate in terms of trust or validation.

1.7                 Certificate Fields

To better understand certificates including how they are used and validated we need to understand the various fields that are contained in a certificate.

Certificates contain two types of fields Version 1 Fields and Extensions.

1.7.1   Version 1 Fields

Version 1 fields are those that were specified in the original specification for X.509 certificates defined in RFC 1422. 

1.7.1.1          Version

This field indicates the X.509 version that is being used.  The current version of the X.509 standard is Version 3.

1.7.1.2          Serial Number

Serial Number is a unique identifier for each certificate.  The Serial Number should be unique at least on a per CA basis.  In other words, a Certification Authority should not issue more than one certificate with the same Serial Number.

1.7.1.3          Signature Algorithm

This field contains the name of the Public Key Algorithm and Hashing Algorithm that were used to sign the certificate.

1.7.1.4          Issuer

This field indicates the name of the CA that issued the certificate

1.7.1.5          Valid From (Validity Period)

This field indicates at what date and time the certificate becomes valid.

1.7.1.6          Valid To (Validity Period)

This field indicates at what date and time the certificate expires

1.7.1.7          Subject

This field contains the identity that is bound to the public key of the certificate.  The identity is bound to that public key duet to the fact that the certificate contains both the public key as well as the identity are contained within a certificate that is signed by a trusted certification authority.

1.7.1.8          Public key

This field contains the public key that is bound to the certificate.

1.7.2   Extensions

After the publication of RFC 1422 the idea of extensions in certificates were introduced in a later RFC. Extensions are optional fields that can be included in a certificate.  However, generally the extensions included in RFC 5280 should be understood by PKI enabled applications.

1.7.2.1          Subject Key Identifier

This field contains a hash of the public key in the certificate, this information is one way to identify the certificate.

1.7.2.2          Authority Key Identifier

This field contains a hash of the public key that is contained in the CA certificate, this is one way to identify the certificate of the CA that issued the current certificate.

1.7.2.3          CRL Distribution Points

This field indicates where Certificate Revocation Lists can be downloaded from to determine if the certificate has been revoked.

1.7.2.4          Authority Information Access

This field indicates where the certificate of the CA that issued the certificate can be downloaded. This field will also indicate where the OSCP service can be accessed, if one is available in the environment.

1.7.2.5          Key Usage

This field indicates what purposes the keys in the certificate can be used for and these purposes include Digital Signature, Key Encryption, Key Agreement, Certificate Signing, CRL Signing, Key Agreement Enciphering, Key Agreement Deciphering.

1.7.2.6          Certificate Template Information

This is a Microsoft specific extension which means it is note one included in the RFC 5280, as such it would only be expected to be understood by Microsoft applications. This field contains the name and OID of the Certificate Template.

1.7.2.7          Certificate Template Name

This is a Microsoft specific extension which means it is note one included in the RFC 5280, as such it would only be expected to be understood by Microsoft applications. This field contains the name of the Certificate Template.

1.7.2.8          Enhanced Key Usage

This extension specifies for what purposes the public key may be used.

1.7.2.9          Application Policies

This extension specifies for what applications the certificate can be used.

1.7.2.10      Subject Alternative Name

The Subject Alternative Name or SAN is an optional field that contains the identity or identities that are bound to the certificate.  The SAN extension is typically used when the certificate will contain more than one identity. To elaborate further identity information can be stored in either the subject field, the Subject Alternative Name extension or both.  Some services such as SSL with ignore the subject field if the Subject Alternative Name extension is populated. 

1.7.2.11      Basic Constraints

Typically, this extension is only included in CA certificate and identifies the certificate as a CA certificate.

1.8                 View the Certificate via Command Line

You can also view the raw certificate via the command line if the certificate is saved as a flat file on the file system. 

To view the certificate in the console run: Certutil <Certificate File Name>

It is often more desirable to output the text to a file. 

Below is an example of the output of the certificate file dumped to a text document.

1.9                 Certificate Storage

In Windows certificates can be stored in a number of places, so let’s discuss the various locations where certificates are stored and the benefits of storing certificates in these locations.

1.9.1   Active Directory Certificate Services Database

Certification Authorities by default store a copy of every issued certificate in its database. Storing certificates in the CA Database enables the following capabilities:

  • Retrieval- Having a copy of each certificate provides a central repository from where certificates can be retrieved by enrollees or a certificate manager.
  • Recovery- Certificates can be recovered for a user or computer since a copy of the certificate is stored in the CA Database.  Additionally, if Key Archival is enabled the private key can be recovered for encryption certificates.
  • Forensics and Analysis: The database can be searched to determine what certificates have been issued and provide useful information such as when a certificate expires.
  • Revocation: If a certificate has been compromised or is no longer trusted that certificate can easily be located in the CA database and revoked.

1.9.2   Active Directory

Certificates can be stored on the userCertificate attribute of a user or computer that enrolls for a certificate. Publishing certificates to a computer or user object is typically not necessary for most applications.  However, some applications such as EFS or Email Encryption can query Active Directory for the certificate of the user that the document or email for which it is being encrypted.

As previously mentioned the certificate is published to the userCertificate during enrollment.  The certificate is published by the Certification Authority.  The Cert Publishers security group is the group that has permission to write to the userCertificate attribute.  In order for the CA to be able to publish a certificate to the userCertificate attribute it must be a member of the Cert Publishers group in the corresponding Active Directory domain.

Additionally, the certificate template must be configured to Publish certificate in Active Directory as seen in the screenshot below:

1.9.2.1          Viewing the userCertificate Attribute

There are several ways to view certificates that are stored in the userCertificate attribute.

1.9.2.1.1                Certmgr.msc

If a user opens Certmgr.msc they can view what certificates are stored on their user object by navigating to the Active Directory User Object node.

1.9.2.1.2                Active Directory Users and Computers

To view the userCertificate attribute in Active Directory Users and Computers navigate to the user or computer whose userCertificate attribute you wish to view.  Open the properties for the user or computer account and navigate to the Published Certificates tab.  If the Published Certificates tab is not available, you need to enable Advanced Features in the Active Directory Users and Computers View menu.

1.9.2.1.3                ADSIEDIT

To view the userCertificate attribute connect to the Domain Partition (Default Naming Context).  Go to properties on the corresponding object and navigate to the userCertificate attribute.

1.9.3   Local Storage of Certificates and Private Keys

For certificates that are stored on the local machine for the computer or user these certificates are stored on the local file system or registry.  The only exception to this is if you are storing certificates and/or keys on a cryptographic device such as a TPM or Smart Card. 

The following tables illustrate where certificates are stored on the local computer.

The following table shows where Computer certificates are stored on the computer. 

StoreStorage Location
RootHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\Root\Certificates
MyHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\My\Certificates
RequestHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\Request\Certificates
TrustedPublisherHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\TrustedPublisher\Certificates
TrustedPeopleHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\TrustedPeople\Certificates
DisallowedHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\Disallowed\Certificates
CAHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\CA\Certificates
ACRAHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\ACRS\CTLs

http://msdn.microsoft.com/en-us/library/windows/desktop/bb204781(v=vs.85).aspx

The following table shows where User certificates are stored on the computer.

StoreStorage Location
RootHKEY_CURRENT_USER\SOFTWARE\Microsoft\SystemCertificates\Root\Certificates
Myfile:\\%APPDATA%\Microsoft\SystemCertificates\My\Certificates
Requestfile:\\%APPDATA%\Microsoft\SystemCertificates\Request\Certificates
TrustedPublisherHKEY_CURRENT_USER\SOFTWARE\Microsoft\SystemCertificates\TrustedPublisher\Certificates
TrustedPeopleHKEY_CURRENT_USER\SOFTWARE\Microsoft\SystemCertificates\TrustedPeople\Certificates
DisallowedHKEY_CURRENT_USER\SOFTWARE\Microsoft\SystemCertificates\Disallowed\Certificates
CAHKEY_CURRENT_USER\SOFTWARE\Microsoft\SystemCertificates\CA\Certificates

http://msdn.microsoft.com/en-us/library/windows/desktop/bb204781(v=vs.85).aspx

The following table shows where CNG Private Keys are stored.

Key typeDirectory
User Private%APPDATA%\Microsoft\Crypto\Keys
Local system private%ALLUSERSPROFILE%\Application Data\Microsoft\Crypto\SystemKeys
Local service private%WINDIR%\ServiceProfiles\LocalService
Network service private%WINDIR%\ServiceProfiles\NetworkService
Shared private%ALLUSERSPROFILE%\Application Data\Microsoft\Crypto\Keys

http://msdn.microsoft.com/en-us/library/bb204778%28VS.85%29.aspx

1.9.3.1          Protection of Private Keys that are stored locally

Private keys are protected by the Data Protection Application Programming Interface (DPAPI).  DPAPI protects the private keys with encryption and that encryption is accessed with a key that is a derivation of the user or computer’s password. 

1.9.4   Smart Card

Certificates can also be stored on a cryptographic device called a Smart Card.  Smart cards are typically credit card size and require a smart card reader.  However, there are also USB smart cards that have an integrated smart card reader.  Regardless, these devices have an integrated circuit and cryptographic functions are performed on that hardware.  Certificates and private keys are stored on the smart card. To use the keys on that smart card a user must have knowledge of the PIN associated with that smart card and must enter the PIN when using the smart card. The key advantage is that the private keys are protected by hardware.  A secondary advantage is the smart card provides a mechanism to roam the user’s certificates and keys.  This is due to the fact that the certificates and keys can be used on any system where the user inserts their smart card.

1.9.5   Virtual Smart Card / Trusted Platform Module

Starting in Windows 8.0 the capability to store certificates on the Trusted Platform Module chip was introduced. A Virtual Smart Card is created on the TPM and the certificate is stored on the Virtual Smart Card which is protected by the TPM.  The Virtual Smart Card offers some of the advantages of physical smart card in the sense that the private keys are protected by hardware and cryptographic functions that require use of the private key occur on the TPM. However, since the TPM is physically part of the computer certificates do not roam with the user as they do in the case of physical smart cards.  However, in some cases that is a practical advantage since physical smart cards can be lost or forgotten by the user. 

1.9.6   Hardware Security Modules

A Hardware Security Module (HSM) is a dedicated cryptographic security device for protecting keys.  In PKI we use an HSM to protect the private keys that are linked to a certificate and that are part of a public/private key pair. HSMs are commonly used to protect the Private Keys of   Private Keys are isolated to prevent tampering or exportation of the Private Keys.  Cryptographic functions that require use of the private key occur in the HSM. HSMs typically use K of N security to prevent a single individual from having control of the HSM.  K of N security means that a certain number of credentials have to presented in order to make changes to the HSM.  For example, if 5 credentials are provisioned for administrative actions on HSM, you can configure the HSM so that at least 3 of those 5 credentials must be presented to the HSM for administrative actions to be performed. There are several types of HSMs currently used.  There are USB HSMs that are typically used for an Offline CA or for Code Signing.  There are PCI Express HSMs that can be used with a Physical CA.  However, the most commonly type used of HSM that I have seen in use are Network HSMs. Network HSMs are connected to via Ethernet. 

1.10             Certificate Administration

There are a number of tools that are used for certificate administration.  The most common tool that you may end up using for certificate administration is the Certificates MMC.  The Certificates MMC itself can be targeted at the certificate store of a user, a computer, or a service account. However, you will most commonly use the Certificates MMC for administration of user or computer certificates, and hence we will focus on administration of user and computer certificates.

1.10.1                  Certmgr.msc

Certmgr.msc is the certificates MMC for managing user certificates.  Using this MMC you can perform the following actions for the user:

  • View Certificates
  • Import Certificates
  • Export Certificates
  • Enroll for Certificates
  • View Requests
  • Generate Certificate Request
  • Process Approved Certificate Requests

Below is a screenshot of the Certmgr.msc:

The table below describes the purpose of each container in Certmgr.msc.

Folder NameContents
PersonalCertificates associated with private keys to which you have access. These are the certificates that have been issued to you, or to the computer or service for which you are managing certificates.
Trusted Root Certification AuthoritiesImplicitly trusted certification authorities. Includes all of the certificates in the Third-Party Root Certification Authorities store plus root certificates from your organization and Microsoft.
Enterprise TrustA container for certificate trust lists. A certificate trust list provides a mechanism for trusting self-signed root certificates from other organizations and limiting the purposes for which these certificates are trusted.
Intermediate Certification AuthoritiesCertificates issued to subordinate certification authorities.
Active Directory User ObjectCertificates associated with your user object and published in Active Directory.
Trusted PublishersCertificates from certification authorities that are trusted by Software Restriction policies.
Untrusted CertificatesAn untrusted certificate is a certificate that a certification authority has revoked, or a certificate that for other reasons has been placed in the Untrusted Certificates folder on your computer.
Third-Party Root Certification AuthoritiesTrusted root certificates from certification authorities other than Microsoft and your organization.
Trusted PeopleCertificates issued to people or end entities that are explicitly trusted. Most often these are self-signed certificates or certificates explicitly trusted in an application such as Microsoft Outlook.
Client Authentication IssuersA list of issuers whose certificates are trusted for TLS/SSL Client Authentication.
Certificate Enrollment RequestsPending or rejected certificate requests.
Smart Card Trusted RootsOn smart card insertion, the Certificate Propagation service propagates any root certificates on the card to the Smart Card Trusted Roots certificate stores on the local computer. This process establishes a trust relationship with the organization.

1.10.2                  Certlm.msc

Certlm.msc is the Certificate MMC for managing certificates for the local computer.  Prior to Windows 8.1 the shortcut (certlm.msc) was not available for launching the Certificate MMC targeted to the local computer. So, for pre-Win8 or pre-Windows Server 2012 machines the following steps can be used to launch the Certificates MMC targeted for the local computer.

Steps for Opening Certificates MMC Targeted for Local Computer

Step 1: Click the Start key and then the R key

Step 2: Type mmc.exe and click OK

Step 3: In the new console select the File menu and from the File menu select Add/Remove Snap-in…

Step 4: Select Certificates and then click the Add button

Step 5: Select Computer account and then click Next

Step 6: Select Local computer:  (the computer this console is running on), and the click Finish

Step 7: Click OK

The Certificates MMC targeted for the Local Computer will now be Open

1.10.2.1      Certlm.msc Uses

The following tasks for the local computer can be performed using certlm.msc:

  • View Certificates
  • Import Certificates
  • Export Certificates
  • Enroll for Certificates
  • Generate Certificate Request

Below is a screenshot of the Certmgr.msc:

The table below describes the purpose of each container in Certlm.msc.

Folder NameContents
PersonalCertificates associated with private keys to which you have access. These are the certificates that have been issued to you, or to the computer or service for which you are managing certificates.
Trusted Root Certification AuthoritiesImplicitly trusted certification authorities. Includes all of the certificates in the Third-Party Root Certification Authorities store plus root certificates from your organization and Microsoft.
Enterprise TrustA container for certificate trust lists. A certificate trust list provides a mechanism for trusting self-signed root certificates from other organizations and limiting the purposes for which these certificates are trusted.
Intermediate Certification AuthoritiesCertificates issued to subordinate certification authorities.
Trusted PublishersCertificates from certification authorities that are trusted by Software Restriction policies.
Untrusted CertificatesAn untrusted certificate is a certificate that a certification authority has revoked, or a certificate that for other reasons has been placed in the Untrusted Certificates folder on your computer.
Third-Party Root Certification AuthoritiesTrusted root certificates from certification authorities other than Microsoft and your organization.
Trusted PeopleCertificates issued to people or end entities that are explicitly trusted. Most often these are self-signed certificates or certificates explicitly trusted in an application such as Microsoft Outlook.
Client Authentication IssuersA list of issuers whose certificates are trusted for TLS/SSL Client Authentication.
Remote DesktopCertificates that are used to provide SSL security for RDP connections.
Smart Card Trusted RootsOn smart card insertion, the Certificate Propagation service propagates any root certificates on the card to the Smart Card Trusted Roots certificate stores on the local computer. This process establishes a trust relationship with the organization.
Web HostingHighly scalable certificate store for SSL certificates that can support thousands of certificates to support increased secure site density.

The following video provides additional information on certificates:

-Chris

Certificate Validation

1              Certificate Validation and Revocation

1.1                 Background

One important concept to understand when troubleshooting certificates. This blog covers the basics of understanding certificate validation. Then towards the end, I cover troubleshooting steps.

1.2                 Identity Validation

Certificates bind an identity to a public key.  And one of the reasons that we use certificates is to ensure that we are accessing the resource that we intend to access.  Therefore, we expect the identity in the certificate to match the identity we are trying to access.  For example, if you enter the following URL in your web browser: https://www.microsoft.com, you would expect that the identity www.microsoft.com would be included in the certificate.  There are two possible fields in the certificate where this identity information can be located: Subject and Subject Alternative Name.  The Subject is a Version 1 field that will typically only contain one identity.  The Subject Alternative Name field is an extension and is typically used when there is more than one identity in the certificate.  For some applications such as SSL, the subject will be ignored if the SAN is populated.  Below is an example of an error a user might receive if there is a mismatch between the identity that they are trying to contact and the identity in the certificate.

1.3                 Chain Validation

In order to determine if a certificate is trusted, you need to build a chain to the root certificate.  Once the chain is built to the root certificate you can determine whether or not the root certificate is trusted.  To validate the chain, you must have possession all of the certificates in the chain and then you can build the chain from the end-entity certificate all the way up to the root certificate, which is a self-signed certificate where the chain terminates.  I will discuss more about Chain Validation later on in this document.

1.4                 Validity Period Validation

Certificates also have a period of time for which they are valid.  In other words, a certificate is valid for a certain period of time. This built in expiration mechanism helps increase the security of certificates.  This is due to the fact that if a certificate is compromised and that compromise is not detected it will eventually expire and become invalid. 

The Valid From field is when the certificate begins its validity period and the Valid To field indicates when a certificate ends its validity period.  In order for a certificate to be valid from a time perspective, the current date should fall in between the Valid From and Valid To dates specified in the certificate.

1.5                 Certificate Revocation Validation

When a Windows client checks the revocation status of a certificate it checks revocation for that certificate and every certificate in the chain excluding the Root CA certificate.

  1. The client will first check to see if it has any time valid OCSP Responses or CRLs cached locally.
  2. If an OSCP URI has been deployed via Group Policy the client will send and OCSP response to the OCSP Responder.  The OCSP responder then should send a response back to the client informing the client whether or not the certificate has been revoked. 
  3. If an OCSP URI is included in the AIA extension of the certificate the client will send and OCSP response to the OCSP Responder.  The OCSP responder then should send a response back to the client informing the client whether or not the certificate has been revoked. 
  4. If there is no OCSP URI Specified the client will download the Base CRL.  If the Base CRL has a Freshest CRL extension the client will then download the Delta CRL.  The client will then parse the CRL(s) to determine if the certificate has been revoked.

1.6                 Certificate Chaining

As mentioned earlier a certificate needs to be chained to a root certificate in order to determine if the certificate chains to a trusted root.  There is information included in certificates to assist in this chain building process.  The most common type of information I see used to help the system identify the chain or the Subject Key Identifier (SKI) field and the Authority Key Identifier (AKI) field.  The Authority Key Identifier is a hash of the certificate issuers public key. The Subject Key Identifier is a hash of the public key in the current certificate. The Operating System can chain a certificate by matching the Authority Key Identifier field in a certificate to the Subject Key Identifier in the issuer’s certificate. Matching the Authority Key Identifier field in the certificate to the Subject Key Identifier of the issuer is commonly called a key match.

Keep in mind not only do the fields have to match but the signature needs to verify as well.  A certificate is digitally signed by the issuer.  This means the certificate data was passed through a hashing algorithm and then the resulting hash is encrypted with the private key of the issuer.  The signature can be verified by hashing the same certificate data, decrypting the digital signature with the public key of the issuer, and comparing the hashes.  If the resulting hashes match and the digital signature can be decrypted with the issuer’s public key then the signature is verified.

Below is an example of a key match:

1.6.1   Retrieving CA Certificates / AIA

Remember that the verifier of the certificate must have all certificates in the chain.  The verifier will have access to the certificate it is verifying usually during a handshake during a protocol setup.  Intermediate certificates may also be distributed the same way.  However, if the intermediate certificates are not distributed during the protocol handshake they can be downloaded from the AIA repository.  There is an extension named Authority Information Access that is commonly included with certificates.  The AIA extension usually has a url which indicates from where the issuer’s certificate can be retrieved from. Windows supports two protocols for retrieval of CA certificates via the AIA Repository: LDAP and HTTP.  Also, keep in mind that you have multiple AIA locations specified in the AIA extension of the certificate.  Those AIA locations are processed in the order in which they are present the AIA extension. 

Below is an example of the AIA extension in a certificate:

1.6.1.1          LDAP Repository

When we talk about using an LDAP repository to host CA certificates we typically mean using the LDAP functionality of Active Directory.  Although, other ldap providers can be used for hosting CA certificates it is less common and does not provide the same level of integration as Active Directory.

There are some significant advantages and disadvantages to using LDAP as an AIA repository:

1.6.1.1.1                Advantages

High Availability: When using Active Directory as the LDAP provider, there is built-in fault tolerance.  This is due to the fact that the AIA location is stored in the Configuration container that is replicated to all Doman Controllers in the forest.

Site Awareness: Active Directory has built-in site awareness.  Clients use DCLocator to find the nearest DC, so clients will be able to pull certificates from a local Domain Controller.

1.6.1.1.2                Disadvantages

3rd Party Support: PKI can be leveraged by 3rd party Operating Systems and applications.  Many of these are not integrated into Active Directory and hence cannot access the Active Directory LDAP provider.

Insecure Networks: Providing access to Active Directory LDAP over insecure networks such as a DMZ or Public Network would be a security nightmare so it would not be a good practice to use LDAP if clients on an insecure network need to retrieve the CA Certificates from AIA.

1.6.1.2          HTTP Repository

Using an HTTP or Web Repository is a more common practice.  This is especially true for Public or 3rd Party CAs.  If using an HTTP repository, it is important that the repository can be located by a DNS name and that the DNS name is not tied to a specific machine.  In other words, you would likely use a CNAME or an A record that is not the computer name of the Web Server.

1.6.1.2.1                Advantages

Insecure Networks: Just about every organization is familiar on how to make the http protocol accessible in DMZ’s or over the Internet. 

3rd Party Support: The virtual directory where the CA certificates are made accessible is normally configured to allow anyone to read and hence download the CA certificates. 

1.6.1.2.2                Disadvantages

High Availability: A web server by itself is not highly available.  Additional work and resources must be brought to bear to make a website highly available including having multiple web servers typically behind a load balancer. 

Site Awareness: If you wish to make the HTTP AIA repository available locally to every site it requires additional web servers and a load balancer that will re-direct clients to the closest web server.

1.6.1.3          AIA Design Considerations

When you are in the process of building a PKI you have to decide where the AIA locations are going to be located and how the names used to access those locations are going to be formed.  A common practice for 3rd Party Certification Authorities is just to use a single http URL inside of the AIA extension.  Then that web location is made available from a highly available provider such as a Content Delivery Network (CDN).  For your own internal PKI you have a number of possible ways to make the HTTP AIA location highly available.  You could use for instance Azure CDN to provide a highly available solution.  Alternatively, you could stand up several web servers behind a load balancer to provide this functionality.  Regardless, when you design the name for the AIA location you will want to choose a name that is not tied to a specific machine name.  Even if you do decide to go against best practices and host the HTTP AIA location on the CA itself, you would still want to use some sort of alias instead of the machine name.  This configuration would allow you flexibility in the future if you ever decide to move that location to another machine.  This is due to the fact that aside from reconfiguring where the certificate was published you would simply just need to update where that DNS alias is pointed instead of having to re-configure the AIA location which would require renewing the CA certificate to update the extension in that certificate. Additionally, if you plan on using the PKI to secure internet facing resources you would want to ensure that the DNS name used for the AIA location is not just resolvable internally, but also externally. Aside from the HTTP location some organizations decide to leverage LDAP as a secondary AIA location that would be available if the HTTP location is inaccessible.

1.7                 Root Certification Authority Certificates

The Root CA’s primary function is to be the trust anchor for the PKI.  When a system trusts a Root CA, then that trust extends to subordinate CAs that chain to that same root and the certificates that they issue.

In order for a Root CA to be trusted on a Windows system the Root CA certificate must be present in the Trusted Root Certification Authorities store.  Keep in mind there Trusted Certification Authorities container in both the Computer certificate store and the User certificate store.  Trusted Root CA Certificates present in the Computer store are inherited by the user. There are a number of ways that a Root CA certificate can be populated in that store. 

1.7.1   Microsoft Root Certificate Program

Microsoft has a program where 3rd Party CAs can submit their certificate to be installed in the Trusted Root Certification Authorities store.  These are the Root CA certificates that ship with Windows.  The advantage to this approach is that a user or organization does not need to go out download and install the Trusted Roots for popular 3rd Party CAs. 

1.7.2   Group Policy

Root CA Certificates can be deployed via Group Policy.  This is a common way organizations deploy Root CA Certificates for the PKI of a trusted organization.  Root CA certificates may also be deployed this way during a Merger or Acquisition.

1.7.3   Autoenrollment

Autoenrollment performs a variety of functions.  One of those functions is to download Enterprise Certificates from Active Directory.  Enterprise Certificates includes the Root CA certificates that are published to the Certification Authorities container as well as downloading Subordinate CA certificates from the Authority Information Access container in Active Directory.  All of these containers are located in the Configuration Partition of Active Directory.

1.7.4   Manual

Unless otherwise prevented by Group Policy users can add Root CA certificates in the Trusted Root Certification Authority container in the user store.  A user that has administrator privileges can also add Root CA certificates to the Trusted Root Certification Authorities container in the Computer certificate store.

1.8                 Chain Building Summary

In order to determine whether or not a certificate should be trusted, the certificate chain must be built.  Starting with the certificate that is being validated, the validator locates the next certificate in the chain.  If that certificate is an intermediate CA that cert can be retrieved locally from the Intermediate Certification Authorities container, passed to the validator during the protocol handshake for whatever protocol is using the certificate, or retrieved from the AIA extension in the certificate.  This process continues until it reaches a self-signed certificate or in other words a Root CA certificate.  If that Root CA certificate exists in the Trusted Root Certification Authority store that the certificate being validated is considered trusted.

1.9                 Revocation

Certificates do have a limited time for which they are valid.  This built-in expiration mechanism enhances the security of certificates.  However, there are times where you may no longer trust a certificate before it expires. There are several possible reasons why you may no longer trust a certificate.  Some of those reasons include: the certificate was compromised, the computer was compromised, or the user left the company.  When you no longer trust a certificate you can revoke.  Once the certificate is revoked, applications that perform revocation checking will no longer trust the certificate.  More specifically the CA Manager and Certificate Manager can revoke the certificate.  When an administrator revokes a certificate they must specify a reason for the revocation.  These reason include: Unspecified, Key Compromise, CA Compromise, Change of Affiliation, Superseded, Cease of Operation, and Certificate Hold. If the Certificate Hold reason is selected, the certificate can later be unrevoked.

1.9.1   Revoking a Certificate

The following steps can be taken to revoke a certificate

Step 1: To revoke a certificate the Certificate Manager open the Certification Authority MMC (certsrv.msc)

Step 2: Click on Issued Certificates

Step 3: Right-click on the certificate that needs to be revoked

Step 4: From the context menu click All Tasks and the Revoke Certificate

Step 5: Select a reason for the revocation

Step 6: Click OK

The Revoked Certificate will show up under Revoked Certificates.  The certificate will not be known to be revoked until the CRL is published to the CDP and subsequently downloaded and parsed by clients checking revocation.

1.9.2   Certificate Revocation List (CRL)

A Certificate Revocation List contains a list of certificates that have been revoked.  Specifically, CRLs contain the Serial Number of each certificate that has been revoked, along with the reason for revocation, and the date and time that the certificate was revoked. There are two types of CRLs: Base CRLs and Delta CRLs. Base CRLs contain a list of all non-expired certificates that have been revoked. Delta CRLs contain a list of non-expired certificates that have been revoked since the last Base CRL was published.  The lists from the latest Base CRL and Delta CRL can be combined to get a list of all non-expired certificates that have been revoked. Delta CRLs were implemented so that clients would not have repeatedly download a base CRL which would be larger in size.  So, instead of publishing a Base CRL more frequently, delta CRL can be published instead.  Delta CRLs are optional you can just use a Base CRL if that better fits the requirements for your organization.

1.9.3   CRL Fields

In this section I will discuss the various fields that are included in a CRL.

1.9.3.1          Version

This is the Version of the CRL Profile that is being used.  The current version of the CRL Profiles is Version2.

1.9.3.2          Issuer

The Issuer field indicates which CA issued the CRL.

1.9.3.3          Effective date

This is the date and time at which a CRL becomes effective.

1.9.3.4          Next update

Next update indicates the date and time at which the CRL will be expired.

1.9.3.5          Signature Algorithm

The Signature Algorithm field indicates what Asymmetric Cryptographic Algorithm and Hashing Algorithm were used to sign the CRL.

1.9.3.6          Signature Hash Algorithm

The Signature Hash Algorithm field indicates the Hashing Algorithm used when signing the CRL

1.9.3.7          Authority Key Identifier

The Authority Key Identifier is another way of identifying the issuer of the CRL.  Specifically, the Authority Key Identifier field contains a hash of the issuer’s public key

1.9.3.8          CA Version

The CA Version Field list the version number for the CA Certificate that was used to sign the CRL.

1.9.3.9          CRL Number

The CRL Number indicates the version of the CRL.  The CRL Number increments each time a new CRL is published.

1.9.3.10      Next CRL Publish

The Next CRL Publish field indicates the date and time at which the next CRL will be published.

1.9.3.11      Published CRL Location

Published CRL Location is the location at which the CRL will be published in Active Directory.  This is typically

1.9.3.12      Freshest CRL

If a CA publishes Delta CRLs then the Freshest CRL extension will show the location(s) where Delta CRLs can be retrieved.

1.9.3.13      Revocation List

The Revocation List will list every non-expired certificate that has been revoked.  This list will include the Serial Number and revocation date.  Depending on the reason code selected it may also list the reason for the revocation.

1.9.4   Delta CRL

A Delta CRL contains a list of all non-expired certificates that have been revoked after the publication of the Base CRL.  To determine the location of the Delta CRL the Freshest CRL extension of the Base CRL must be read or known.  To determine revocation, the current Base CRL and latest Delta CRL must be available.

1.9.4.1          Delta CRL Indicator

A Delta CRL has the same extension and formatting as a base CRL.  A Delta CRL also has a field called Delta CRL Indicator.  This field indicates which base CRL the client must have to have complete knowledge of revocations.

1.9.5   CRL Distribution Points (CDP)

The CRL Distribution Points extension is an extension in a certificate that indicates where the CRL can be retrieved from for the purposes of checking revocation. That location where the CRLs can be retrieved is often referred to as a CDP repository. Some of the same concepts that apply to the Authority Information Access extension apply here, but I have listed them below for completeness. 

1.9.5.1          LDAP Repository

When we talk about using an LDAP repository to host CA certificates we typically mean using the LDAP functionality of Active Directory.  Although, other ldap providers can be used for hosting CA certificates it is less common and does not provide the same level of integration as Active Directory.

There are some significant advantages and disadvantages to using LDAP as an AIA repository:

1.9.5.1.1                Advantages

High Availability: When using Active Directory as the LDAP provider, there is built-in fault tolerance.  This is due to the fact that the AIA location is stored in the Configuration container that is replicated to all Doman Controllers in the forest.

Site Awareness: Active Directory has built-in site awareness.  Clients use DCLocator to find the nearest DC, so clients will be able to pull certificates from a local Domain Controller.

1.9.5.1.2                Disadvantages

3rd Party Support: PKI can be leveraged by 3rd party Operating Systems and applications.  Many of these are not integrated into Active Directory and hence cannot access the Active Directory LDAP provider.

Insecure Networks: Providing access to Active Directory LDAP over insecure networks such as a DMZ or Public Network would be a security nightmare so it would not be a good practice to use LDAP if clients on an insecure network need to retrieve the CA Certificates from AIA.

1.9.5.2          HTTP Repository

Using an HTTP or Web Repository is a more common practice.  This is especially true for Public or 3rd Party CAs.  If using an HTTP repository, it is important that the repository can be located by a DNS name and that the DNS name is not tied to a specific machine.  In other words, you would likely use a CNAME or an A record that is not the computer name of the Web Server.

1.9.5.2.1                Advantages

Insecure Networks: Just about every organization is familiar on how to make the http protocol accessible in DMZ’s or over the Internet. 

3rd Party Support: The virtual directory where the CA certificates are made accessible is normally configured to allow anyone to read and hence download the CA certificates. 

1.9.5.2.2                Disadvantages

High Availability: A web server by itself is not highly available.  Additional work and resources must be brought to bear to make a website highly available including having multiple web servers typically behind a load balancer. 

Site Awareness: If you wish to make the HTTP AIA repository available locally to every site it requires additional web servers and a load balancer that will re-direct clients to the closest web server.

1.9.5.3          CDP Design Considerations

When you are in the process of building a PKI you have to decide where the CDP locations are going to be located and how the names used to access those locations are going to be formed.  A common practice for 3rd Party Certification Authorities is just to use a single http URL inside of the CDP extension.  Then that web location is made available from a highly available provider such as a Content Delivery Network (CDN).  For your own internal PKI you have a number of possible ways to make the HTTP CDP location highly available.  You could use for instance Azure CDN to provide a highly available solution.  Alternatively, you could stand up several web servers behind a load balancer to provide this functionality.  Regardless, when you design the name for the CDP location you will want to choose a name that is not tied to a specific machine name.  Even if you do decide to go against best practices and host the CDP AIA location on the CA itself, you would still want to use some sort of alias instead of the machine name.  This configuration would allow you flexibility in the future if you ever decide to move that location to another machine.  This is due to the fact that aside from reconfiguring where the CRL was published you would simply just need to update where that DNS alias is pointed instead of having to re-configure the CDP location which would require renewing the CA certificate to update the extension in that certificate. Additionally, if you plan on using the PKI to secure internet facing resources you would want to ensure that the DNS name used for the CDP location is not just resolvable internally, but also externally. Aside from the HTTP location some organizations decide to leverage LDAP as a secondary CDP location that would be available if the HTTP location is inaccessible.

1.9.5.4          CRL Publishing

So, how do we maximize the amount of time we have for recovery, and at the same time increase the “freshness” of the CRL?  We can do this with the use of Overlap Periods.

So, let’s say we want to implement this solution that maximizes recovery time and increases CRL freshness.  The first consideration is Delta CRLs.  Since Delta CRLs in my view are not necessary in most instances, we will first get rid of Delta CRLs.  So, now that gives us just Base CRLs to work with.  Let us take our Base CRL and make it valid for a period of 3 days.  So, at first you would think great, now I have 3 days to recover the CA if the CA fails since the CRL is valid for 3 days.  Well this is not true.  If the validity period of the CRL and the publishing interval coincide, then you could have as little time as 1 second depending on when the CA failed.  The illustration below exemplifies this scenario.  The problem is without changing the default configuration the Next Update and Next Publish fields will be very close together, providing very little time for recovery.

1.9.5.5          Introducing CRL Overlap Period

As mentioned there is some additional configuration that can be performed that can optimize your CRL publishing intervals so that you have adequate time to perform Emergency CRL Signing or to recover your CA.  What will need to be configured is the CRL Overlap Period.  In order to configure the CRL Overlap Period both the CRLOverlapUnits and CRLOverlapPeriod registry settings need to be configured. 
So, in my previous example my CRL had a validity period of 3 days.  What I can do now is add a CRL Overlap Period of 3 days.  With this configuration, my CRL will be valid for a period of 6 Days.  However, at 3 days a new CRL will be published as well.  This is illustrated in the graphic below:


 
In the example illustrated in the graphic above, CRL 1 will be valid for a period of 6 days.  CRL 2 will be published at Day 3.  So, if my CA fails between Day 1 and Day 3, I would still have 3 Days (Day 3 through Day 6) to perform Emergency CRL signing or to recover my CA in event of failure.  If my CA fails between Day 3 and Day 6, there is a new CRL (CRL 2) that is valid through Day 9.  So, in short if my CA fails between Day 3 and Day 6, I still have at least 3 days to perform Emergency CRL Signing or to recovery my CA, before revocation checking starts to fail.  And the reason that I have the 3 days is the CRL Overlap Period extended out my CRL for 3 days and staggered the Next Publish and Next Update times by 3 days. 

1.10             Troubleshooting Certificate Validation

There are several tools that you can use to troubleshooting Chain Building and Revocation.

1.10.1                  Windows UI

You can use the errors visible in the Windows UI to troubleshoot Certificate Validation.  Below are a sample of some errors you may encounter in the Windows UI.

This CA Root certificate is not trusted.  To enable trust, install this certificate in the Trusted Root Certification Authorities store.

This error simply indicates that the Root CA Certificate is not trusted.

Windows does not have enough information to verify this certificate.

This error indicates that the client verify the certificate does not have access to all of the certificates in the chain.

The issuer of this certificate could not be found.

This error indicates that the client verify the certificate does not have access to all of the certificates in the chain.

This certificate has expired or is not yet valid.

This error typically indicates that the certificate is expired.

1.10.2                  Certutil -verify -urlfetch

Certutil -verify -urlfetch is perhaps the best tool to check the validation of a certificate.  You will want to run this tool on a machine which appears to have validation issues.  For example, if a client is visiting a website that is secured with TLS you would want to copy the certificate to the client and run this tool from the client. This command does all of the validation checks including checking of the AIA, CDP, and OCSP locations are accessible and contain valid configuration.  The normal syntax for running the command is: certutil -verify -urlfetch [File Name of Certificate] > [Name of Output Text File].

Below are screenshots or a sample output of certutil – verify -urlfetch to help you better understand how to read this file.

Here are some of the information or error messages you can receive at the end of the output from the command:

Leaf certificate revocation check passed

Indicates that the certificate passed the verification check.

A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file. 0x800b0101 (-2146762495 CERT_E_EXPIRED)

Typically means the certificate is expired.

ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)

CertUtil: The revocation function was unable to check revocation because the revocation server was offline.

This means that the verifier was unable download a CRL.

The certificate is revoked. 0x80092010 (-2146885616 CRYPT_E_REVOKED)

Indicates that the certificate has been revoked.

Conclusion

That covers certificate validation. For additional information see the following video.

-Chris

The problematic “Publish certificate in Active Directory” option

Disclaimer

I am writing this blog  and others to explain how things work and some ways deployment and operational tasks can be handled. In other words, these postings are for demonstration purposes only. Since I am not familiar with your organization or environment I do not know if these steps are applicable to your environment or are even safe to perform in your environment. It is recommended that you contact Microsoft Support prior to making changes in your environment to ensure that these steps are applicable to your environment, and are safe to perform in your environment. By writing this blog I am in no way recommending that you perform these steps in your own environment. If you choose to follow the steps outlined in this or other blog postings on this site, you are assuming the risk for your actions.

Background

So, a little bit of background. You may or may not have noticed that certificate templates have this Publish certificate to Active Directory option. So, what does this do? Well the answer is that it publishes the resulting certificate to the userCertificate attribute to the user or computer depending on whether the template is a user or computer template. An example of this settings is in the screenshot below.

So, why would you want to do this? Honestly, there is not really many good reasons for doing this. Theoretically, this would be useful when using Email Encryption or Email Signing. So, for email encryption if I wanted to encrypt an email to a user a could pull their certificate from AD and the ecrypt the email with the public key from the certificate. And for email signing if a user is trying to validate a signature they can retrieve the signers certificate from AD and use the public key to decrypt the signature so the hashes can be compared. Theoretically, the access to the users public key could potententially allow a user to give access to another user to an EFS encrypted file. However, this last option I do not have any experience with.

That being said, most organizations do not use email encryption and signing or EFS. Email encryption and signing are a problematic because getting the required certificates to all devices on which a user can check email is extremely difficult if not impossible. Think of all the devices you check email on and imagine trying to get the same exact certificate installed on all of those devices. EFS is problematic for file encryption because the process is very manual and honestly not very enterprise friendly.

So, in short although you can publish certificates to Active Directory there is most likely no need for you to do so.

The problem

So, what is the problem with this setting. For a user certificate the user will in a typical configuration enroll for a new certificate each time they log onto a new machine. If the template is configured to publish a certificate to the userCertificate attribute on the user. So, for a user that such as an IT administrator who logs onto many systems they will publish a large number of certificates to their userCertificate attribute. Another situation where this occurs when there is an MDM solution in place and the MDM uses a service account that requests certificates on behalf of it’s clients. If the certificate template that it leverages is configured to pubish the certificate to Active Direcotry it can bloat the userCertificate attribute for that service account.

So, the main problems are if many users are publishing their certificates to Active Directory it can lead to bloating of the AD database. If it is a single user publishing many certificates to it’s userCertificate attribute it can bloat that attribute in Active Directory.

Once the attribute becomes too large, certificates will no longer be able to publish certificates to the attribute and eventually it can interfere with AD replication because the attribute may be too large to replicate.

Investigating the problem

It is pretty straight forward to determine the severity of the problem using PowerShell. The following script will search AD and display the results:

get-aduser -filter * -property usercertificate | select name, distinguishedname, @{name=’CertCount’;expression={$_.usercertificate.count}}

Alternatively, you export the results to a CSV file with:

get-aduser -filter * -property usercertificate | select name, distinguishedname, @{name=’CertCount’;expression={$_.usercertificate.count}} | Export-Csv -Path .\userCertificate.csv

Clean Up

The following command will clear the userCertiifcate attribute for all users. Before running this command make sure that you do not need the userCertificate attribute populated.

Get-aduser –filter * | set-aduser –certificates $null

Prior I had one account “chdelay” that had 6 certificates in the userCertificate attribute and after running the command that user now has 0.

Thanks for visiting,

-Chris

Troubleshooting Active Directory Replication

Disclaimer

I am writing this blog  and others to explain how things work and some ways deployment and operational tasks can be handled. In other words, these postings are for demonstration purposes only. Since I am not familiar with your organization or environment I do not know if these steps are applicable to your environment or are even safe to perform in your environment. It is recommended that you contact Microsoft Support prior to making changes in your environment to ensure that these steps are applicable to your environment, and are safe to perform in your environment. By writing this blog I am in no way recommending that you perform these steps in your own environment. If you choose to follow the steps outlined in this or other blog postings on this site, you are assuming the risk for your actions.

1              Troubleshooting Active Directory Replication

1.1                 Repadmin.exe

Repadmin is a tool for checking replication status and troubleshooting replication issue.  Below is a table highlighting commonly used syntax of the repadmin tool.

SyntaxUsage
Repadmin /replsummaryThe replsummary operation quickly and concisely summarizes the replication state and relative health of a forest.
Repadmin /replsummary /bysrc /bydest /sort: deltaThe replsummary operation quickly and concisely summarizes the replication state and relative health of a forest.
Repadmin /showrepl <DC Name>Displays the replication partners for each directory partition on the specified domain controller. Helps the administrator build a visual representation of the replication topology and see the role of each domain controller in the replication process.
Repadmin /showutdvecDisplays the highest Update Sequence Number (USN) for the specified domain controller. This information shows how up-to-date a replica is with its replication partners.
Repadmin /showobjmeta <DC> <DN of object>Displays the replication metadata for a specified object stored in Active Directory, such as attribute ID, version number, originating and local Update Sequence Number (USN), and originating server’s GUID and Date and Time stamp. By comparing the replication metadata for the same object on different domain controllers, an administrator can determine whether replication has taken place.
Repadmin /showconnDisplays the connection objects for a specified domain controller. Default is local site.
Repadmin /replsingleobj <DC List> <Source DSA Name> <Object DN>Replicates a single object between any two domain controllers that have partitions in common. The two domain controllers do not have a replication agreement. Replication agreements can be shown by using the Repadmin /showrepl command.
Repadmin /replicate <Destination_DC_List> <Naming Context>Starts a replication event for the specified directory partition between the source and destination domain controllers. The source UUID can be determined when viewing the replication partners by using the Repadmin showrepl operation.
Repadmin /syncall <DC>Synchronizes a specified domain controller with all replication partners.
Repadmin /queueDisplays tasks waiting in the replication queue.
Repadmin /showmsg <Error>Displays the error message for a given error number.
Repadmin /viewlist <DC_List>Displays a list of domain controllers.
Repadmin /showctx <DC_List>Displays a list of computers that have opened sessions with a specified domain controller.
Repadmin /showcertDisplays the server certificates loaded on a specified domain controller.
Repadmin /removelingeringobjects <Dest_DC_List> <Source DC GUID> <NC> [/ADVISORY_MODE]Uses an authoritative domain controller to compare the directory of a domain controller (destination) that is suspected of having lingering objects against the directory of a domain controller (source) that is designated as a reference source for up-to-date values for the domain of the destination. When the advisory mode parameter is used, this command provides a list of found lingering objects. When the advisory mode parameter is not used, this command removes lingering objects from the destination domain controller.

Additional information on Repadmin.exe is available here: https://technet.microsoft.com/en-us/library/cc736571(v=ws.10).aspx

1.2                 Repadmin /replsummary

As seen in the screenshot below repadmin /replsummary will give statistics for replication with replication partners.  The output also lists any errors that were encountered with replication. This is useful for getting an overview of any replication issues the DC is having.

You can also sort the output.  In the example below, the output is sorted by the largest delta since last replication.

1.3                 Repadmin /showrepl

As seen below repadmin /showrepl shows the replication status with all of the DCs replication partners and is sorted by the Naming Context that is being replicated.

One trick that can be used to get a more manageable output is to use repadmin to send its output to a CSV and the use PowerShell to convert the CSV to a GridView.  The command to do this is repadmin /showrepl * /csv | ConvertFrom-CSV | Out-GridView

The resulting output is in a manageable GUI.

In GridView you can sort and filter.  Below is an example of filtering on Number of Failures, so that I can easily see what failed.

1.4                 Repadmin /showutdvec

Replications changes are tracked through incrementing numbers called USNs.  There are times where you will want to know what knowledge each DC has about other DCs current state.  The up-to-dateness vector is the knowledge that a DC as about the current state of other DCs.  This information can be useful when trying to troubleshoot replication issues such as USN Rollback.  USN Rollback is when a DC is restored from an unsupported method such as a snapshot.  In that case the up-to-dateness vector would be much larger than the actual USN of the DC. Since, there is going to be some delay in replication you will notice some differences but the numbers should be relatively close.  For example, if you compare the up-to-dateness vector for DC01 across DCs you will notice the following: for itself DC01 has USN of 17347, DC02 has a USN of 17346 for DC01, and DC03 has a USN of 17346 for DC01.  So, we can see the numbers are relatively close and that DC01 potentially has one change that it needs to replicate to DC02 and DC03.

1.5                 Repadmin /showobjmeta

The /showobjmeta switch shows detailed information for attributes of an object.  It is most commonly used when comparing the output of the command from 2 DCs to see if they are in sync and the current status of the attributes.  Differences can be used to identify replication problems.

1.6                 Repadmin /syncall

Repadmin /syncall is used to force replication between domain controllers.  You can easily view options for the /syncall switch with the following command: repadmin /syncall /?

A normal use of repadmin /syncall is with the /AeP switch

1.7                 Repadmin /showmsg

The /showmsg switch is used to convert an error message you may receive as the result of a repadmin command and converts it to human readable text.

1.8                 Repadmin /viewlist

Repadmin /viewlist is used to get a list of domain controllers.

1.9                 PowerShell

PowerShell is an object oriented scripting language that allows enterprises to automate IT tasks.

Below is a conversion table that shows the PowerShell command that can be used in place of the Repadmin command.  So, why would you choose to use PowerShell?  The output of PowerShell commands are objects those objects can be filtered with properties, piped through other PowerShell commands and manipulated to many useful things including great control in how the data is presented to the user.

CommandPowerShell Cmdlet
Repadmin /FailCacheGet-ADReplicationFailure
Repadmin /QueueGet-ADReplicationQueueOperation
Repadmin /ReplSingleObjSync-ADObject
Repadmin /ShowConnGet-ADReplicationConnection
Repadmin /ShowObjMetaGet-ADReplicationAttributeMetadata
Repadmin /ReplSummaryGet-ADReplicationPartnerMetadata
Repadmin /ShowutdvecGet-ADReplicationUpToDatenessVectorTable
Repadmin /SiteOptionsSet-ADReplicationSite
Repadmin /ShowAttrGet-ADObject
Repadmin /Set AttrSet-ADObject

Get-ADReplicationParnerMetadata is very similar to running repadmin /showrepl.  Without passing the output through another cmdlet the formatting is a bit different then to what you get with repadmin.

However, the advantage is that the output of the command are objects.  You can constrain your views to certain properties.

The other advantage is that you can pass objects through other cmdlets.  As seen here I am passing the output of Get-ADReplicationPartnerMetadata through Output-GridView.

Once in GridView you have the ability to sort and filter the data.

Here is another example of the usefulness of using PowerShell over repadmin.  In this example I take the output of Get-ADReplicationPartnerMetadata then passing it through Select-Object so that we can then limit what objects are presented in GridView.

Here we see the output of that command.

1.10             Replication Errors

Here is a list of replication errors you may come across in either the Directory Services event log or while running repadmin.

Event IDReplication ErrorIssue
1388Lingering Objects
1988Lingering Objects
2042Lingering Objects
1925DNS Lookup Issues or Connectivity Problems
2087DNS Lookup Issues
2088DNS Lookup Issues
1311Replication Topology Issues
8614Tombstone lifetime exceeded
8524DNS Lookup failure
8456Server is currently rejecting replication requests
8457Server is currently rejecting replication requests
8453Access was denied
8452The naming context is in the process of being removed or is not replicated from the specified server
5Access is denied
-21468930222The target principal name is incorrect
1753There are no more endpoints available from the endpoint mapper
1722The RPC server is unavailable
1396Logon Failure The Target account name is incorrect
1256The remote system is not available
1127While accessing the hard disk, a disk operation failed even afer retries
8451The replication operation encountered a database error
8606Insufficient attributes were given to create an object

2              Troubleshooting Steps for Common Replication Issues

2.1                 Troubleshooting -21468930222  (The target principal name is incorrect)

On the DC that is the cause of the error, perform the following steps:

Step 1: Open Services.msc

Step 2: Configure KDC Service for Manual

Step 3: Stop the Service

Step 4: Restart the Domain Controller

Step 5: Open PowerShell as an Administrator

Step 6: Run: $cred = Get-Credential

Step 7: Enter Credentials and click OK

Step 8: Run, Reset-ComputerMachinePassword –Server <ServerName> -Credential $cred

Step 9: Restart the server

Step 10: Set the KDC service to Automatic, Start the service and click OK.

2.2                 Troubleshoot Replication Error 8606, Event ID 1388, and Event ID 1988

These issues are caused by lingering objects.  Lingering objects can be caused when a domain controller is taken offline for an extended period of time, does not replicate for longer than the tombstone lifetime, or is restored from a backup that is older than the tombstone lifetime. 

When an object is deleted it is put in a tombstone state.  After the tombstone lifetime passes (typically 180 days), DC run garbage collection and those tombstone objects are deleted.  If a DC was offline for the entire TSL and then were brought back online they may have objects that have since been deleted, tombstoned, and garbage collected.  Any objects that were deleted will still exist on that DC.  These objects go unnoticed until a change is made to that object then the DC attempts to replicate that object, and at that point that is where it is either re-introduced into the environment or if strict replication consistency is enabled, blocked. 

2.2.1   How to Determine TSL

Run the following command: dsquery * “cn=directory service,cn=windows nt,cn=services,cn=configuration,<Forest DN>” –scope base –attr tombstonelifetime

2.2.2   How to Remove Lingering Objects

2.2.2.1          Repadmin /removelingeringobjects

One way to remove lingering objects is to user repadmin with the /removelingeringobjects switch.  First you must identify a clean source of the partition.  The syntax of the command is repadmin /removelingeringobjects <Dest DC Name> <Source DC Guid> <Naming Context>.  So, in other words you need to identify the source DCs guid and the Naming Context you want to clean.  The naming context will be available in the Event 1388 or 1988 you receive in the event long.  Once you find a clean source you can obtain the guid by opening DNS Manager and opening up the _msdcs Zone and obtaining the CName record for the DC in question.

Below is an example of running the repadmin /removelingeringobjects command

You will receive an Event 1937 when the removal of lingering objects begins.

You will then receive an Event 1939 when removal completes.

2.2.2.2          Repadmin /rehost

An alternative to using repadmin /removelingeringobjects command is to unhost the partition so that the domain controller no longer has that partition and then rehosting the entire partition with a good source.

The repadmin syntax for unhosting the partition is repadmin /unhost <DC Name> <Partition Name>

You will receive an event an event 1658 when the removal begins.

You will receive an event 1660 when the removal completes

The syntax for rehosting the partition is: repadmin /rehost <Dest DC Name> <Partition> <Source DC Name>

2.3                 Troubleshooting Event ID 2042

Review event log for any 1988 or 1388 errors.  If found use the previous section to remove the lingering objects from the domain controller. 

Option 1: Re-hosting the partition that has not replicated

If the partition is a GC partition consider unhosting and rehosting the partition. Instructions for unhosting and rehosting are in the previous section called Repadmin /rehost

Option 2: Removing and then re-adding the domain controller to Active Directory

Another option is removing the DC from Active Directory and Re-promoting the Domain Controller

Step 1: Run Import-Module ADDSDeployment

Step 2: Run: Uninstall-ADDSDomainController –DemoteOperationMasterRole:$true –Force:$true

Step 3: Enter and confirm the new local password

Step 4:  Next you will need to run the Install –ADDSDomainController cmdlet.  Below is a sample that you can use.  You will need to modify the template to meet the requirements of your environment. 

Install-ADDSDomainController –NoGlobalCatalog:$false –CreateDnsDelegation:$false –CriticalReplicationOnly:$false –DatabasePath “C:\Windows\NTDS” –DomianName “fabrikam.com” –InstallDNS:$true –LogPath “C:\Windows\NTDS” –ReBootOnCompletion:$false –ReplicationSourceDC “DC01.fabrikam.com” –SiteName “Default-First-Site-Name” –SysvolPath “C:\Windows\SYSVOL” –Force:$true

Option 3: Enabling Replication with Divergent and Corrupt Partner

Due to the risk of adding lingering objects to Active Directory the final consideration should be enabling the following setting: Allow Replication With Divergent and Corrupt Partner. 

Step 1: To enable this setting run the following command on the domain controller:

repadmin /regkey <hostname> +allowDivergent

Step 2: Let replication complete

Step 3: Disable the setting with the following command: repadmin /regkey <hostname> -allowDivergent

2.4                 Troubleshooting Event ID 1311

Event 1311 is caused when there is not complete connectivity between domain controllers. There are a number of reasons there may not be complete connectivity. 

2.4.1   ISTG

The Inter-Site Topology Generator (ITSG) is responsible for building the replication topology.  So to determine what the scope of the connectivity issues it is important to identify the ISTGs that are logging 1311. 

To find the ISTGs in your environment you need to use ldp.exe

Below are the steps for locating the ISTGs:

Step 1: Launch ldp.exe

Step 2: When LDP opens, select Connection and then Connect…

Step 3: In the Connect dialog box, enter the name of a Domain Controller for the Server you want to connect to and then click OK

Step 4: Click on Connection and then click Bind…

Step 5: In the Bind dialog box, click OK

Step 6: Select the Browse menu and then select Search

Step 7: In the search enter the following:

Base DN: CN=Sites,CN=Configuration,<DN of Forest Root> (example: CN=Sites,CN=Configuration,DC=fabrikam,DC=com)

Filter: (CN=NTDS Site Settings)

Scope: Subtree

Attributes: Append the following to the attributes that are already listed: ;interSiteTopologyGenerator

Step 8: Click Run

Step 9: For each site you will then need to look for interSiteTopologyGenerator to determine the ITSG for each site.

2.4.2   BASL

By default, Bridge All Site Links (BASL) is enabled in Active Directory.  If your environment is not fully routed, then you will want to disable BASL.  By fully routed we mean each site can contact every other site.  If BASL is configured on a network which is not fully routed, the KCC will generate site bridges that cannot actually be reached. To determine if BASL is enabled launch Active Directory Sites and Services (dssite.msc). 

Expand Sites, then Inter-site Transports.

Right-click on IP and select Properties from the context menu

If Bridge all site links is enabled, there will be a check box next to it.  To disable BASL, uncheck the checkbox and click OK.

2.4.3   Site Link Bridges

If you disable BASL you can still bridge site links.  You would do that if you wanted two spoke sites to communicate directly if they could not communicate with the hub site.  In a hub and spoke configuration the cost of crossing to site links (bridging a site link) will typically be a higher then just connecting directly to the hub site.  So, ordinarily you would not have to worry about the Site Link Bridge being used instead of a direct site link.  That being said, there are not a whole lot of scenarios where you would need to create Site Link bridges.

The following steps will allow you to bridge two site links.

Step 1: Open the Active Directory Sites and Services MMC.

Step 2: Expand Sites and then expand Inter-site Transports

Step 3: Select New Site Link Bridge… from the context menu


Add at least two sites to the Site Link Bridge, give it a Name, and click OK

And the Site Link Bridge has been completed.

2.4.4   Verify that all Sites are in a Site Link

Step 1: Run the following command in a PowerShell Console: Get-ADObject –LDAPFilter ‘(objectClass=site)’ –SearchBase (Get-ADRootDSE).ConfigurationNamingContext –Property Name | Format-Table Name

Step 2: In another PowerShell Console run: Get-ADObject –LDAPFilter ‘(objectClass=sitelink)’ –SearchBase (Get-ADRootDSE).ConfigurationNamingContext –Property Name, Cost, Description, Sitelist | Format-List Name, Sitelist

Step 3: Verify that each site that was listed in Step 1 exists in one of the site lists returned in Step 2

If not all sites are contained in a site link that you need to determine what site link that site needs to be added to or if a new site link needs to be created.

And that is all I have for replication troubleshooting for today.

-Chris