Rundeck Node Authentication with domain account

I have installed Rundeck 4.8.0 on Redhat 9.
I have a Windows 2022 Server node.
I have a local account on the node called rundeck and it’s in the Administrators group.
In Rundeck key storage, I made a password key, with the password for the local rundeck account.
In my project I have a yaml file pointing to the node with the rundeck username.
That works, I can run jobs that call powershell scripts on the node.

However, now I want to use a domain account, [email protected]

I have installed necessary apps:
yum install gcc python-devel krb5-devel krb5-workstation python-devel python3-devel

In My Project config, under Default Node Executor, I first tried to use the built in "WinRM Node Executor Python"

Interpreter - Python3
Authentication - Kerberos
username - [email protected]
Password - path to key store
Protocol - http
shell - powershell
krb5C Config file - /etc/krb5.conf

my /etc/krb5.conf

[libdefaults]
    dns_lookup_realm = false
    dns_lookup_kdc = false
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
    spake_preauth_groups = edwards25519
    dns_canonicalize_hostname = fallback
    qualify_shortname = ""
    default_ccache_name = KEYRING:persistent:%{uid}
udp_preference_limit = 0
default_realm = MANAGEMENT.CORP

[realms]
  MANAGEMENT.CORP = {
     kdc = NYMGMTDC01.management.corp
     admin_server = NYMGMTDC01.management.corp
     default_domain = MANAGEMENT.CORP
}

[domain_realm]
  .management.corp = MANAGEMWNT.CORP
  management.corp = MANAGEMWNT.CORP

On the windows node the winrm config looks like this

winrm get winrm/config
Config
    MaxEnvelopeSizekb = 500
    MaxTimeoutms = 60000
    MaxBatchItems = 32000
    MaxProviderRequests = 4294967295
    Client
        NetworkDelayms = 5000
        URLPrefix = wsman
        AllowUnencrypted = true
        Auth
            Basic = true
            Digest = true
            Kerberos = true
            Negotiate = true
            Certificate = true
            CredSSP = false
        DefaultPorts
            HTTP = 5985
            HTTPS = 5986
        TrustedHosts
    Service
        RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
        MaxConcurrentOperations = 4294967295
        MaxConcurrentOperationsPerUser = 1500
        EnumerationTimeoutms = 240000
        MaxConnections = 300
        MaxPacketRetrievalTimeSeconds = 120
        AllowUnencrypted = true
        Auth
            Basic = true
            Kerberos = true
            Negotiate = true
            Certificate = false
            CredSSP = false
            CbtHardeningLevel = Relaxed
        DefaultPorts
            HTTP = 5985
            HTTPS = 5986
        IPv4Filter = *
        IPv6Filter = *
        EnableCompatibilityHttpListener = false
        EnableCompatibilityHttpsListener = false
        CertificateThumbprint
        AllowRemoteAccess = true
    Winrs
        AllowRemoteShellAccess = true
        IdleTimeout = 7200000
        MaxConcurrentUsers = 2147483647
        MaxShellRunTime = 2147483647
        MaxProcessesPerShell = 2147483647
        MaxMemoryPerShellMB = 1024
        MaxShellsPerUser = 2147483647

When I test I the node I get this error:

[ERROR  ]  generate_request_header(): authGSSClientStep() failed: (kerberos_.py:257)[winrm.vendor.requests_kerberos.kerberos_]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/winrm/vendor/requests_kerberos/kerberos_.py", line 245, in generate_request_header
    result = kerberos.authGSSClientStep(self.context[host],
kerberos.GSSError: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))
[ERROR  ]  (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377)) (kerberos_.py:259)[winrm.vendor.requests_kerberos.kerberos_]

From my googling around, this points to a lack of SPN, but the node’s SPNs look fine.

setspn -L NYMGMTRDNODE01
Registered ServicePrincipalNames for CN=NYMGMTRDNODE01,OU=Servers1,OU=Servers,OU=Management,DC=management,DC=corp:
        WSMAN/NYMGMTRDNODE01.management.corp:5985
        TERMSRV/NYMGMTRDNODE01.management.corp
        WSMAN/NYMGMTRDNODE01.management.corp
        RestrictedKrbHost/NYMGMTRDNODE01.management.corp
        HOST/NYMGMTRDNODE01.management.corp
        TERMSRV/NYMGMTRDNODE01
        WSMAN/NYMGMTRDNODE01
        RestrictedKrbHost/NYMGMTRDNODE01
        HOST/NYMGMTRDNODE01

I even had our admin add "WSMAN/NYMGMTRDNODE01.management.corp:5985" incase the port wasnt being specified.
Also on the node itself I tested the winrm connection.

winrm identify -r:http://NYMGMTRDNODE01.management.corp:5985 -auth:kerberos -u:[email protected] -p:PASSWORD -encoding:utf-8

IdentifyResponse
    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
    ProductVendor = Microsoft Corporation
    ProductVersion = OS: 10.0.20348 SP: 0.0 Stack: 3.0
    SecurityProfiles
        SecurityProfileName = http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/basic, http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/spnego-kerberos

So next I tried the Overthere WinRm plugin, rundeck-winrm-plugin-1.3.8.jar
I created a resources.xml file :

<node name="NYMGMTRDNODE01"
description="Windows node"
tags="Windows"
hostname="NYMGMTRDNODE01.MANAGEMENT.CORP"
username="rundeck"
osFamily="Windows"
osName="Microsoft Windows Server 2022Standard"
osArch="amd64"
node-executor="overthere-winrm"
winrm-auth-type="kerberos"
winrm-protocol="http"
winrm-cmd="Powershell"
winrm-kerberos-debug="true"
winrm-domain="MANAGEMENT.CORP"
winrm-port="5985"
winrm-timeout="PT28800.000S"
winrm-connection-timeout="28800000"
connectionType="WINRM_NATIVE"
winrm-password-storage-path="keys/NYMGMTRDNODE01.password"/>

When I test this node, the debug shows this:

Debug is  true storeKey false useTicketCache false useKeyTab false doNotPrompt false ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is true principal is null tryFirstPass is false useFirstPass is false storePass is false clearPass is false
Refreshing Kerberos configuration
        [Krb5LoginModule] user entered username: [email protected]
principal is [email protected]
Commit Succeeded

and then the error :

[overthere-winrm:NYMGMTRDNODE01.MANAGEMENT.CORP] failed: WinRM Error: Unexpected HTTP response on http://NYMGMTRDNODE01.MANAGEMENT.CORP:5985/wsman:   (401)
Failed: WinRMProtocolError: WinRM Error: Unexpected HTTP response on http://NYMGMTRDNODE01.MANAGEMENT.CORP:5985/wsman:   (401)
Execution failed: 106 in project Staging-Windows: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [NYMGMTRDNODE01: WinRMProtocolError: WinRM Error: Unexpected HTTP response on http://NYMGMTRDNODE01.MANAGEMENT.CORP:5985/wsman:   (401) + {dataContext=MultiDataContextImpl(map={}, base=null)} ]}, Node failures: {NYMGMTRDNODE01=[WinRMProtocolError: WinRM Error: Unexpected HTTP response on http://NYMGMTRDNODE01.MANAGEMENT.CORP:5985/wsman:   (401) + {dataContext=MultiDataContextImpl(map={}, base=null)} ]}, status: failed]

I have found a lot of posts with the "Unexpected HTTP response (401)" issue.
I have tried to follow all the fixes, some people seem to have no resolution and some do.

I’ve on been this for 48 hours straight. So any ideas, any help would be greatly appreciated.

thanks you.