Configure and Troubleshoot Kerberos Connection

Kerberos Authentication Support

Overview

Kerberos authentication is enabled by default in domain-joined virtual machines and requires no additional configuration.

Prerequisite

In environments configured for strict Kerberos authentication (with NTLM traffic restricted), the Diagnostic Server logon account must have permission to register a Service Principal Name (SPN). If automatic SPN registration fails, the SPN must be registered manually before configuring or using the Diagnostic Server.

Limitations

Kerberos authentication is unavailable in non-domain-joined virtual machines. In domain-joined environments, some Windows account types, including local machine user accounts, are not supported.

Troubleshooting Kerberos Connection Failures in NTLM-Restricted Environments

Spotlight Enterprise uses Kerberos authentication when available. If Kerberos cannot be used for any reason, Spotlight quietly switches to NTLM instead.

• If NTLM is allowed on your machines, you will not notice anything — the connection continues to function using NTLM authentication.
• If NTLM is blocked by policy in your environment, there is nothing to switch to, and the connection fails.

So if Spotlight has stopped working in an environment where NTLM is restricted, the checks below will usually identify the cause.

Symptoms

• The Spotlight client on one machine cannot connect to a Diagnostic Server on another machine.
• Federated Diagnostic Servers do not connect to each other.

Both use the same Kerberos setup, so the same checks and fixes apply to both.

Confirm the environment can use Kerberos

The following requirements must be satisfied for Kerberos authentication to work correctly.

• The machines are joined to an Active Directory domain. Kerberos does not work on workgroup or personal machines. A machine joined only to Microsoft Entra ID, with no on-premises Active Directory, also does not qualify.
• The machines are in the same domain, or in domains that trust each other.
• A domain controller is reachable from each machine.
• The clocks are in sync. A difference of more than about five minutes between machines will stop Kerberos from working.
• The Diagnostic Server host name resolves correctly in DNS from the machine that is connecting to it.

Use the host name, not an IP address

When entering the address of a Diagnostic Server, use its full host name, for example SERVER01.YOURDOMAIN.COM. The short name usually works too, because Windows expands it automatically.

An IP address only works if your DNS is configured to look up names from addresses. Many environments are not set up this way, so an IP address is the least reliable choice. Using the full host name avoids the problem entirely.

Check the Account the Software Runs As

Kerberos can be used only with certain types of Windows accounts. This applies to both the Diagnostic Server service and the Spotlight client. If either one runs as an unsupported account, Kerberos cannot be used and the connection falls back to NTLM.

The account running the “Spotlight Diagnostic Server” service

Check this in Services → Spotlight Diagnostic Server → Properties → Log On tab.

The Windows account you are signed in with when running the Spotlight client

The Spotlight client uses the account you signed in to Windows with. There is no separate setting for this.

To see which account you are signed in with, open Command Prompt and run:

whoami

A domain account is shown as DOMAIN\username. A local account is shown as COMPUTERNAME\username.

Check the Diagnostic Server log

If everything above is in order and Spotlight still does not connect, open the log file on the machine running the Diagnostic Server:

C:\Program Files\Quest Software\Diagnostic Server\Agent\log\adk.log

Search the log file for SPNManager. Spotlight registers a name in Active Directory when the Diagnostic Server service starts, and these entries record whether that succeeded.

Entries that mean everything is fine

SPN already registered with DS account. No action required: SPOTLIGHT/SERVER01.YOURDOMAIN.COM

SPN registered successfully: SPOTLIGHT/SERVER01.YOURDOMAIN.COM

No further action is needed. If the connection still fails, see the If the problem continues section.

Entries that mean registration was skipped

SPN registration skipped: host is not domain-joined (workgroup/personal machine).

The machine is not joined to a domain. Kerberos cannot be used. See the Confirm the environment can use Kerberos section.

SPN registration skipped: DS is running under local machine account

The service is running as a local Windows account. Change it to a supported account type - see Check the account the software runs as then restart the Spotlight Diagnostic Server service.

Entries that mean registration failed

Any of the following means the name was not registered, and it must be registered manually before Spotlight can use Kerberos.

SPN registration failed for SPOTLIGHT/SERVER01.YOURDOMAIN.COM on ...

The service account was not permitted to register the name, or the domain could not be contacted. This is the most common failure.

SPN registration failed: unable to resolve a fully qualified host name.

The machine’s full name could not be determined. Check the DNS settings on this machine.

SPN registration failed: unable to resolve target DS account for SPN registration.

The account the service runs as could not be identified. Check the Log On tab of the service.

SPN query failed for SPOTLIGHT/SERVER01.YOURDOMAIN.COM

SPN registration failed: unable to determine current SPN ownership state.

Active Directory could not be contacted, or the response could not be read. Check that a domain controller is reachable from this machine.

SPN migration failed: could not remove SPN from current owner before re-registering.

The name is already registered to a different account and could not be moved. Therefore, follow the instructions in the If the name is already registered to another account section.

You may also see this summary line:

DS SPN registration failed. Kerberos authentication may fail and NTLM fallback will be used where supported.

Common reasons for a failure are:

• The service account does not have permission to register the name in Active Directory. Ordinary domain user accounts do not have this permission by default.
• A domain controller was not reachable when the service started, for example immediately after a restart.
• The name is already registered to a different account.
• The setspn command is not available on the machine.

Register the name manually

You have three options. Choose whichever suits your environment.

Option 1 - Give the existing service account permission. In Active Directory Users and Computers, grant the account the Validated write to service principal name permission on its own account object. Then restart the Spotlight Diagnostic Server service; the name will be registered automatically.

Option 2 - Change the service account. Set the Spotlight Diagnostic Server service to run under an account that already has this permission. A domain administrator account has this permission by default. Then restart the service. Make sure the new account is a supported account type, as listed in the table under the Check the Account the Software Runs As section.

Option 3 - Ask an administrator with the necessary permission to register the name. Follow the below steps.

Find the host name and domain name

On the machine running the Diagnostic Server, open Command Prompt and run:

echo %COMPUTERNAME%.%USERDNSDOMAIN%

This prints the full name of the machine on a single line, for example:

SERVER01.YOURDOMAIN.COM

If you prefer to see the two parts separately, hostname prints the host name on its own and echo %USERDNSDOMAIN% prints the domain name on its own.

If the result is blank, or shows %USERDNSDOMAIN% instead of a domain name, you are signed in with a local account. Sign in with a domain account and run the command again.

Work out the name to register

Add the SPOTLIGHT/ prefix to the full name:

SPOTLIGHT/SERVER01.YOURDOMAIN.COM

Identify the account to register it against

This must be the account the Spotlight Diagnostic Server service runs as — not any other account. Check the Log On tab of the service.

• If the service runs as LocalSystem, NetworkService, or a virtual service account, use the computer account, written with a $ on the end: SERVER01$
• If the service runs as a domain user or a managed service account, use that account name: YOURDOMAIN\spotlightsvc

Register the name

The setspn command is used for this. It is available on domain controllers, and on other machines where the Active Directory management tools (RSAT) are installed.

Open Command Prompt as an administrator and run:

setspn -S SPOTLIGHT/SERVER01.YOURDOMAIN.COM SERVER01$

Replace the name and the account with your own values.

To confirm it worked:

setspn -L SERVER01$

The name should appear in the list.

Restart and verify

1. On the Diagnostic Server machine, restart the Spotlight Diagnostic Server service.
2. On the machine running the Spotlight client, open Command Prompt and run klist purge to clear cached Kerberos credentials. Then start the Spotlight client.
3. Open the log file again and confirm you now see:

SPN already registered with DS account. No action required: SPOTLIGHT/SERVER01.YOURDOMAIN.COM

If the name is already registered to another account

This is uncommon, but if it occurs, the name must be registered to only one account.

To see which account currently holds it:

setspn -Q SPOTLIGHT/SERVER01.YOURDOMAIN.COM

The account name is shown in the result. You then have two choices:

• Change the Spotlight Diagnostic Server service to run as that account, or
• Remove the name from that account and register it to the account you intend to use:

setspn -D SPOTLIGHT/SERVER01.YOURDOMAIN.COM <current-account>
setspn -S SPOTLIGHT/SERVER01.YOURDOMAIN.COM <intended-account>

Then follow the steps mentioned in the Restart and Verify section.

If the problem continues

If you have completed all the preceding steps and Spotlight still does not connect, generate a support bundle and open a support case with Quest Support. Include the bundle and a summary of the troubleshooting steps you have already performed.

Amazon RDS SQL Server Kerberos Configuration

To connect to an Amazon RDS for SQL Server instance using Active Directory integrated authentication over Kerberos, use the instance’s private endpoint as the server address instead of the public endpoint.

Use the private AD endpoint for Windows Authentication on NTLM-restricted systems because Kerberos works only when the server name matches the Active Directory identity registered for the RDS instance; the public AWS endpoint does not match that identity.

Connect Using Kerberos

1. Sign in to the client VM using your domain account.
2. Make sure the VM is joined to the same Active Directory domain, or to a trusted domain.
3. Open Spotlight on SQL Server (SoSSE) using your domain user session.
4. In the connection settings, enter the private endpoint for the RDS instance. For example:

   psl-mssql-standard-rds.corp.local,1433
   

5. Select Use Diagnostic Server authentication - Integrated Windows auth.
6. Connect.

| IMPORTANT:

• Do not use the public AWS RDS endpoint for Windows Authentication when NTLM is restricted.
• If SQL Authentication is being used, the public endpoint can still work, but for Kerberos-based Windows Authentication use the private AD endpoint.

  If Connection Fails

  Check the following:

  1. You are logged in with a domain account, not a local machine account.
  2. The VM is domain joined.
  3. The private endpoint resolves from the client VM.
  4. The RDS instance is configured for Active Directory authentication.
  5. Your domain account has permission to log in to the SQL Server instance.