401 Unauthorized on v1.2.0, Works Fine on v1.1.0

[MAbdElRaouf]

A script that utilizes requests-ntlm.HttpNtlmAuth to authenticate with a website had been working fine until I upgraded the package to the latest version 1.2.0 then it started throwing 401 unauthorized exception. Downgrading to v1.1.0 without any changes to the code resolved the issue.

import requests
from requests_ntlm import HttpNtlmAuth


response = requests.get(
    "http://url",
    auth=HttpNtlmAuth("username", "password"),
)

# v1.1.0: 200 OK
# v1.2.0: 401 Unauthorized

Note: the script runs behind an HTTP corporate proxy if it makes any difference.

[doomwomble]

Same here, authenticating to SharePoint 2019 on-prem. Updating to 1.2.0 breaks the authentication code, giving me an HTTP 401 and reverting to 1.1.0 fixes the issue.

[cebaa]

1.2.0 replaced ntlm_auth with spnego:

v1.1.0...v1.2.0#diff-3590900fb531e3b46542fdf34352cfdd58edf42d2fca824e5d518ab3375e6e4aL9

I suspect they are doing things differently under the hood.

[JudahSchwartz]

Finding the same issue in my project

[jborean93]

The pyspnego library should be support all the same features as the old dependency ntlm-auth (I wrote both of them). If the newer version is failing then trying to see what differences in the token will hopefully narrow down what might be happening to them. You can run python -m spnego --token $BASE64_TOKEN. NTLM exchanges 3 tokens across a request so comparing them all might help to identify the differences.

An example of all three messages in an exchange are shown below (I used Wireshark to get these values). A not in that the nonces will have different values per exchange and NTProofStr is a hash of the password + nonce information used so they will differ across the exchanges. Some common examples I've seen in the past

• The Version field might be omitted in the messages and some servers might mandate it in some circumstances
• The MSV_AV_TARGET_NAME AvPair in the authentication message is set to a unknown host which may flag Windows (I'm not sure if there is a policy for this or not)

Ultimately this issue is not something I can replicate, more information will need to be provided to compare the differences between the old and new way to find out why this might be happening for you.

Negotiate Message

$ python -m spnego --format yaml --token TlRMTVNTUAABAAAAN4II4gAAAAAoAAAAAAAAACgAAAAACQEAAAAADw==

MessageType: NEGOTIATE_MESSAGE (1)
Data:
  NegotiateFlags:
    raw: 3792208439
    flags:
    - NTLMSSP_NEGOTIATE_56 (2147483648)
    - NTLMSSP_NEGOTIATE_KEY_EXCH (1073741824)
    - NTLMSSP_NEGOTIATE_128 (536870912)
    - NTLMSSP_NEGOTIATE_VERSION (33554432)
    - NTLMSSP_NEGOTIATE_EXTENDED_SESSIONSECURITY (524288)
    - NTLMSSP_NEGOTIATE_ALWAYS_SIGN (32768)
    - NTLMSSP_NEGOTIATE_NTLM (512)
    - NTLMSSP_NEGOTIATE_SEAL (32)
    - NTLMSSP_NEGOTIATE_SIGN (16)
    - NTLMSSP_REQUEST_TARGET (4)
    - NTLMSSP_NEGOTIATE_OEM (2)
    - NTLMSSP_NEGOTIATE_UNICODE (1)
  DomainNameFields:
    Len: 0
    MaxLen: 0
    BufferOffset: 40
  WorkstationFields:
    Len: 0
    MaxLen: 0
    BufferOffset: 40
  Version:
    Major: 0
    Minor: 9
    Build: 1
    Reserved: '000000'
    NTLMRevision: 15
  Payload:
    DomainName:
    Workstation:
RawData: 4E544C4D5353500001000000378208E200000000280000000000000028000000000901000000000F

Challenge Message

$ python -m spnego --format yaml --token TlRMTVNTUAACAAAADAAMADgAAAA1goniQInKuHLN48QAAAAAAAAAAJwAnABEAAAACgB8TwAAAA9EAE8ATQBBAEkATgACAAwARABPAE0AQQBJAE4AAQAUAFMARQBSAFYARQBSADIAMAAyADIABAAWAGQAbwBtAGEAaQBuAC4AdABlAHMAdAADACwAUwBFAFIAVgBFAFIAMgAwADIAMgAuAGQAbwBtAGEAaQBuAC4AdABlAHMAdAAFABYAZABvAG0AYQBpAG4ALgB0AGUAcwB0AAcACACdH8ukxsTZAQAAAAA=

MessageType: CHALLENGE_MESSAGE (2)
Data:
  TargetNameFields:
    Len: 12
    MaxLen: 12
    BufferOffset: 56
  NegotiateFlags:
    raw: 3800662581
    flags:
    - NTLMSSP_NEGOTIATE_56 (2147483648)
    - NTLMSSP_NEGOTIATE_KEY_EXCH (1073741824)
    - NTLMSSP_NEGOTIATE_128 (536870912)
    - NTLMSSP_NEGOTIATE_VERSION (33554432)
    - NTLMSSP_NEGOTIATE_TARGET_INFO (8388608)
    - NTLMSSP_NEGOTIATE_EXTENDED_SESSIONSECURITY (524288)
    - NTLMSSP_TARGET_TYPE_DOMAIN (65536)
    - NTLMSSP_NEGOTIATE_ALWAYS_SIGN (32768)
    - NTLMSSP_NEGOTIATE_NTLM (512)
    - NTLMSSP_NEGOTIATE_SEAL (32)
    - NTLMSSP_NEGOTIATE_SIGN (16)
    - NTLMSSP_REQUEST_TARGET (4)
    - NTLMSSP_NEGOTIATE_UNICODE (1)
  ServerChallenge: 4089CAB872CDE3C4
  Reserved: '0000000000000000'
  TargetInfoFields:
    Len: 156
    MaxLen: 156
    BufferOffset: 68
  Version:
    Major: 10
    Minor: 0
    Build: 20348
    Reserved: '000000'
    NTLMRevision: 15
  Payload:
    TargetName: DOMAIN
    TargetInfo:
    - AvId: MSV_AV_NB_DOMAIN_NAME (2)
      Value: DOMAIN
    - AvId: MSV_AV_NB_COMPUTER_NAME (1)
      Value: SERVER2022
    - AvId: MSV_AV_DNS_DOMAIN_NAME (4)
      Value: domain.test
    - AvId: MSV_AV_DNS_COMPUTER_NAME (3)
      Value: SERVER2022.domain.test
    - AvId: MSV_AV_DNS_TREE_NAME (5)
      Value: domain.test
    - AvId: MSV_AV_TIMESTAMP (7)
      Value: '2023-08-01T22:22:23.1484317Z'
    - AvId: MSV_AV_EOL (0)
      Value:
RawData:
  4E544C4D53535000020000000C000C0038000000358289E24089CAB872CDE3C400000000000000009C009C00440000000A007C4F0000000F44004F004D00410049004E0002000C0044004F004D00410049004E000100140053004500520056004500520032003000320032000400160064006F006D00610069006E002E00740065007300740003002C0053004500520056004500520032003000320032002E0064006F006D00610069006E002E0074006500730074000500160064006F006D00610069006E002E007400650073007400070008009D1FCBA4C6C4D90100000000

Authenticate Message

$ python -m spnego --format yaml --token TlRMTVNTUAADAAAAGAAYAFgAAAD4APgAcAAAAAAAAABoAQAANAA0AGgBAAAaABoAnAEAABAAEAC2AQAANYKJ4gAJAQAAAAAPLYyl/bMH17nMmNwANMCtpwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMHtVmMbCoV1TZuSGbteLeoBAQAAAAAAAJ0fy6TGxNkBh5sLMRRmtb4AAAAAAgAMAEQATwBNAEEASQBOAAEAFABTAEUAUgBWAEUAUgAyADAAMgAyAAQAFgBkAG8AbQBhAGkAbgAuAHQAZQBzAHQAAwAsAFMARQBSAFYARQBSADIAMAAyADIALgBkAG8AbQBhAGkAbgAuAHQAZQBzAHQABQAWAGQAbwBtAGEAaQBuAC4AdABlAHMAdAAHAAgAnR/LpMbE2QEJACAAaABvAHMAdAAvAHUAbgBzAHAAZQBjAGkAZgBpAGUAZAAGAAQAAgAAAAAAAAAAAAAAdgBhAGcAcgBhAG4AdAAtAGQAbwBtAGEAaQBuAEAARABPAE0AQQBJAE4ALgBUAEUAUwBUAEoAQgBPAFIARQBBAE4ALQBMAEkATgBVAFgA5ucL7LuCRwB+Lnly16t1DA==

MessageType: AUTHENTICATE_MESSAGE (3)
Data:
  LmChallengeResponseFields:
    Len: 24
    MaxLen: 24
    BufferOffset: 88
  NtChallengeResponseFields:
    Len: 248
    MaxLen: 248
    BufferOffset: 112
  DomainNameFields:
    Len: 0
    MaxLen: 0
    BufferOffset: 360
  UserNameFields:
    Len: 52
    MaxLen: 52
    BufferOffset: 360
  WorkstationFields:
    Len: 26
    MaxLen: 26
    BufferOffset: 412
  EncryptedRandomSessionKeyFields:
    Len: 16
    MaxLen: 16
    BufferOffset: 438
  NegotiateFlags:
    raw: 3800662581
    flags:
    - NTLMSSP_NEGOTIATE_56 (2147483648)
    - NTLMSSP_NEGOTIATE_KEY_EXCH (1073741824)
    - NTLMSSP_NEGOTIATE_128 (536870912)
    - NTLMSSP_NEGOTIATE_VERSION (33554432)
    - NTLMSSP_NEGOTIATE_TARGET_INFO (8388608)
    - NTLMSSP_NEGOTIATE_EXTENDED_SESSIONSECURITY (524288)
    - NTLMSSP_TARGET_TYPE_DOMAIN (65536)
    - NTLMSSP_NEGOTIATE_ALWAYS_SIGN (32768)
    - NTLMSSP_NEGOTIATE_NTLM (512)
    - NTLMSSP_NEGOTIATE_SEAL (32)
    - NTLMSSP_NEGOTIATE_SIGN (16)
    - NTLMSSP_REQUEST_TARGET (4)
    - NTLMSSP_NEGOTIATE_UNICODE (1)
  Version:
    Major: 0
    Minor: 9
    Build: 1
    Reserved: '000000'
    NTLMRevision: 15
  MIC: 2D8CA5FDB307D7B9CC98DC0034C0ADA7
  Payload:
    LmChallengeResponse:
      ResponseType: LMv2
      LMProofStr: '00000000000000000000000000000000'
      ChallengeFromClient: '0000000000000000'
    NtChallengeResponse:
      ResponseType: NTLMv2
      NTProofStr: C1ED56631B0A85754D9B9219BB5E2DEA
      ClientChallenge:
        RespType: 1
        HiRespType: 1
        Reserved1: 0
        Reserved2: 0
        TimeStamp: '2023-08-01T22:22:23.1484317Z'
        ChallengeFromClient: 879B0B311466B5BE
        Reserved3: 0
        AvPairs:
        - AvId: MSV_AV_NB_DOMAIN_NAME (2)
          Value: DOMAIN
        - AvId: MSV_AV_NB_COMPUTER_NAME (1)
          Value: SERVER2022
        - AvId: MSV_AV_DNS_DOMAIN_NAME (4)
          Value: domain.test
        - AvId: MSV_AV_DNS_COMPUTER_NAME (3)
          Value: SERVER2022.domain.test
        - AvId: MSV_AV_DNS_TREE_NAME (5)
          Value: domain.test
        - AvId: MSV_AV_TIMESTAMP (7)
          Value: '2023-08-01T22:22:23.1484317Z'
        - AvId: MSV_AV_TARGET_NAME (9)
          Value: host/unspecified
        - AvId: MSV_AV_FLAGS (6)
          Value:
            raw: 2
            flags:
            - MIC_PROVIDED (2)
        - AvId: MSV_AV_EOL (0)
          Value:
        Reserved4: 0
    DomainName:
    UserName: [email protected]
    Workstation: JBOREAN-LINUX
    EncryptedRandomSessionKey: E6E70BECBB8247007E2E7972D7AB750C
  SessionKey: Failed to derive
RawData:
  4E544C4D53535000030000001800180058000000F800F80070000000000000006801000034003400680100001A001A009C01000010001000B6010000358289E2000901000000000F2D8CA5FDB307D7B9CC98DC0034C0ADA7000000000000000000000000000000000000000000000000C1ED56631B0A85754D9B9219BB5E2DEA01010000000000009D1FCBA4C6C4D901879B0B311466B5BE0000000002000C0044004F004D00410049004E000100140053004500520056004500520032003000320032000400160064006F006D00610069006E002E00740065007300740003002C0053004500520056004500520032003000320032002E0064006F006D00610069006E002E0074006500730074000500160064006F006D00610069006E002E007400650073007400070008009D1FCBA4C6C4D9010900200068006F00730074002F0075006E0073007000650063006900660069006500640006000400020000000000000000000000760061006700720061006E0074002D0064006F006D00610069006E00400044004F004D00410049004E002E0054004500530054004A0042004F005200450041004E002D004C0049004E0055005800E6E70BECBB8247007E2E7972D7AB750C

[cebaa]

Changing this line https://github.com/jborean93/pyspnego/blob/main/src/spnego/_ntlm.py#L201

-     return to_text(workstation) if workstation else None
+     return None

fixes it for me. I did a few more tests and apparently this is a length thing - e.g. this works as well (len = 20):

-     return to_text(workstation) if workstation else None
+     return '12345678901234567890'

but this fails:

-     return to_text(workstation) if workstation else None
+     return '123456789012345678901'

What I put there doesn't seem to matter - any string with len <= 20 works.

This:

https://serverfault.com/a/585824/371201

quotes this:

http://technet.microsoft.com/en-us/library/ee617211.aspx

which says:

SamAccountName

Specifies the Security Account Manager (SAM) account name of the user, group, computer, or service account. The maximum length of the description is 256 characters. To be compatible with older operating systems, create a SAM account name that is 20 characters or less. This parameter sets the SAMAccountName for an account object. The LDAP display name (ldapDisplayName) for this property is "sAMAccountName".

Emphasis mine.

Note that I've found services on my end that do work with len > 20, so this issue seems to be server-dependent.

Apparently 1.1.0 is sending empty workstation, so that's why it's working.

If others can apply the same thing (just temporarily update the mentioned line in site-packages/spnego/_ntlm.py), that might shed more light on this. Also, if they have short names, they may want to test putting long strings for a test and that should at least give you some more indication if this is widespread or not.

Enough NTLM for me for tonight. Hopefully this was helpful, enjoy your weekend!

[jborean93]

@cebaa thanks for the investigation, are you able to share the information on the server that is rejecting the NTLM token when the workstation is more than 20 characters? I've found that the Windows hosts I've tested with are just fine but knowing more info about the servers which do reject it might help figure out the next best steps.

Just as an FYI for compatibility with gss-ntlmssp the pyspnego code does use the environment variable NETBIOS_COMPUTER_NAME as a way to override the workstation name used so you can set this environment variable to whateve ris needed instead of modifying the code.

[cebaa]

Thanks @jborean93

are you able to share the information on the server that is rejecting the NTLM token when the workstation is more than 20 characters?

It's a custom java server. I'm not certain, but AFAIK it's using jespa for NTLM authentication.

Just as an FYI for compatibility with gss-ntlmssp the pyspnego code does use the environment variable NETBIOS_COMPUTER_NAME as a way to override the workstation name used so you can set this environment variable to whateve ris needed instead of modifying the code.

Doh, I should have read the code just above. Even easier, but yeah that fixes as well and I feel it's enough of a escape hatch for this particular case.

[jborean93]

Thanks for sharing, considering Windows and all the other platforms I've tested with seem to be fine with a longer workstation name I would be reluctant from trimming it automatically in the pyspnego code. The env var seems like a decent workaround to me for people who are affected by this.

[kiryvl2]

Hi, @jborean93
I followed your instruction to get 3 tokens for authentication with v1.1.0 and v1.2.0.
v1.1.0 works for me while v1.2.0 fails with 401 error.
I'm not using requests-ntlm directly but rather via exchangelib (tried with v5.0.3, v4.7.6 and v4.7.2). But the difference between two cases only in version of requests-ntlm (1.1.0 vs 1.2.0).
I'm trying to authenticate against local Microsoft Exchange Server 2019 (https://local-
ip-address/EWS/Exchange.asmx) and local Active Directory (this is testing environment so everything is installed on the same server).
Could you please have a look at the attached text files with decoded tokens and possibly see what could be wrong?
Thank you!

v110.txt - success
v120.txt - 401 error

Update:
After a bit of debugging I noticed that spnego uses SPPIProxy class. To make it to use NTLMProxy I provided spnego.NegotiateOptions.use_ntlm via options parameter in requests_ntlm.py and this fixed 401 error for me:

client = spnego.client(
    self.username,
    self.password,
    protocol="ntlm",
    channel_bindings=cbt,
    **options=spnego.NegotiateOptions.use_ntlm**
)

I don't know if this is a correct fix but hopefully it might provide more context to you.

Update 2:
We found that requests-ntlm 1.2.0 works as is in case of remote authentication. So this issue might be specific to local authentication flow.
I collected tokens for the remote (successful) scenario and it differs from the local one only by things like timestamps, etc.
v120_remote_success.txt

[kiryvl2]

Update 3:
I found that there is a potentially helpful message generated in the Windows System event log:

The program w3wp.exe, with the assigned process ID 8672, could not authenticate locally by using the target name host/unspecified. The target name used is not valid. A target name should refer to one of the local computer names, for example, the DNS host name. Try a different target name.

This host/unspecified is also mentioned in the AUTHENTICATE_MESSAGE (3) (see v120.txt attached above):

{
    "AvId": "MSV_AV_TARGET_NAME (9)",
    "Value": "host/unspecified"
},

The "unspecified" string is specified as default value for hostname parameter of the client class (spnego/auth.py).
If I pass a correct hostname in requests_ntlm.py file (v1.2.0) then it helps against 401 error.

[jborean93]

This host/unspecified is also mentioned in the AUTHENTICATE_MESSAGE (3) (see v120.txt attached above):

Thanks for the investigation, it should be simple to add the target hostname of the request to the SPN value used there but it'll only be valid if the actual HTTP target host is valid. If you use an IP then that will stay as the IP.

We found that requests-ntlm 1.2.0 works as is in case of remote authentication. So this issue might be specific to local authentication flow.

When using SSPI actual with a localhost target, Windows can now verify the origin of the request as SSPI includes the Single_Host_Data structure in the AV pairs to indicate the origin. With this Windows can do things like verify the origin was elevated or not amongst other things. By switching over to the pure NTLM provider it will no longer include this structure so Windows thinks it is remote.

Overall SSPI is really the true mechanism here and offers more features that the pure NTLM provider does not; cached credentials, etc. While we can do more to provide a more accurate target/SPN this will only be as good as the target hostname used in the original request.

[jborean93]

The PR #137 does a few things to try and make it more like past releases. It will:

• Set the SPN service to http and the hostname to the URL request target hostname
• Favour the pure NTLM implementation in the spnego library if an explicit username/password are specified

This ensures it'll continue to support using the Windows current context if none are specified but solve the loopback authentication problem that previously wasn't an issue.

[mortendaehli]

I'm running into the same issue in my project.

[airpaio]

@jborean93 I have tried your changes from #137 and they still did not resolve the 401 error. I am running this from a python:3.8-slim container image.

The only thing that I could get working was return None as mentioned #136 (comment)

[jborean93]

@airpaio the PR makes it so requests-ntlm will always use the pure Python NTLM code rather than SSPI on Windows. This has connotations around how authentication is done when talking back to localhost but doesn't deal with the hostname problem mentioned elsewhere in this thread.

If you are finding the workstation is a problem with your NTLM service you can set the env var NETBIOS_COMPUTER_NAME to an empty string or a shortened string. As mentioned I don't want to change the default behaviour as most NTLM services can (and should) be able to handle long workstation names here. But in the cases of a problematic NTLM service you can use this env var to set it to whatever you desire.

[airpaio]

Hmm, I couldn't get it to work even with setting NETBIOS_COMPUTER_NAME, and workstation depending on any length < 20 didn't seem to work either as mentioned in other comments above. The only thing that worked in my case was to return None from the _get_workstation in `spnego. Perhaps this is something to do with the container platform that I am running the program from?

For what it's worth, I was able to monkey patch spnego directly from my calling code instead of having to change the spnego code directly in site-packages. Here's what worked for me (using requests-ntlm==1.2.0)...

from requests_ntlm import HttpNtlmAuth

# mokey patch _get_workstation
import spnego._ntlm
spnego._ntlm._get_workstation = lambda: None

... 
# write code that uses HttpNtlmAuth

This will work for my use case because the vendor's software that we are interfacing with is getting rid of NTLM auth in a new release soon.

[MarcusWenzel-Bayer]

The PR #137 does a few things to try and make it more like past releases. It will:

• Set the SPN service to http and the hostname to the URL request target hostname
• Favour the pure NTLM implementation in the spnego library if an explicit username/password are specified

This ensures it'll continue to support using the Windows current context if none are specified but solve the loopback authentication problem that previously wasn't an issue.

@jborean93: This solves the issue in my case. Will this PR be merged to main anytime soon?