Posts

Kerberos fails with CIFS using AOVPN

Hey. Today I want to talk about an interesting case that involves Kerberos, Always On VPN and access to CIFS.
A customer has recently deployed Always on VPN in their infrastructure. Most clients worked well with that but a few had mixed issues with old VPN clients installed on some machines.

Cisco AnyConnect usually worked fine when installed, but there was another VPN client that disabled the IKE/EXT service which prevented the AO VPN IPSEC to work properly. So uninstalling that software solved the issues. The customer still had that installed on some clients as a backup solutions for when IPSEC was blocked at the source (for example hotels, airports etc.)

But hey! I mentioned Kerberos, how does that come into play?

Disclaimer: This post has might have little to do with Always On VPN, but the issue manifested itself when connected through AOVPN.

Well, there was a few clients that actually connected fine. They could ping stuff on the network and everything seemed fine until they tried to access the file server. They got prompted with credentials stating that they had no access to the domain controller even though I could actually get LDAP access with AD-powershell, so LDAP was obviously working. This was interesting. After a few log checks on both sides of the fence, nothing popped out. So I decided to install Wireshark on the domain controller to try and figure things out. This gave me lots of new and critical information.

I could clearly see that Kerberos was not working. UN_SUPPORTED when the client tried to get a Kerberos ticket from the KDC. So I checked the DC logs and found issues with the Kerberos certificate.
Sorry for the lack of screens, this all happened really fast and I was definitely not allowed to screenshot the customers data.

It turned out that the domain controller was using previously issued certificates from and old and retired Certificate Authority. So I deleted all of them and issued the domain controller new certs for domain controller authentication and Kerberos authentication. Now my senses were tingling since I knew that this would fix the problems. And lo and behold, it did! The troubled clients worked right away.

But, one thing was still bothering me and is still bothering me. Since this was a server side fix, why did not all the clients have this issue? Why was only a select few clients using Kerberos auth? The customer is telling me that all computers are equal and installed from the same image and are getting the same policies. So why are only a select few using Kerberos (that failed). At the time of writing, I don’t know. This happened just recently. Maybe you have some ideas? Feel free to contact me on twitter (@mrblackswe) or post a comment below. Something tells me that the clients are not equal at all, despite what the customer is telling me (Usually the case). The clients are Windows 10 Pro 1809, afaik.

Upgrading the lab

Good day to you. Today I’ve done a little write-up about my home lab equipment. I was noticing a few slowdowns once I got around 10 vm’s running on my old “server”, which was an older gen Intel E3 1240 series CPU running 32Gb of DDR3 RAM, SSD cache and spinning HDDs for mass storage. Since the stuff was closing in on 5 years of service I thought it was time to invest in some new hardware.

This time I decided to go AMD and specifically the Threadripper 1920X with 12 cores/24 threads and 64Gb of DDR4 memory. So I doubled the RAM amount and also the RAM speed and the core count tripled with higher clocks as well. All flash this time as well did it’s thing for sure as there are now 5 SSD in a combination of SATA and M2 drives in RAID-0 hosting the VM’s through storage spaces. As far as I know, the only limitation of running Hyper-V on Threadripper is that it can’t do nested virtualization, but I haven’t verified that myself yet as it is a feature I don’t specifically need.

I did not invest in networking at all since I don’t really need more that 1 Gbit externally from the host. Everything else in my network runs a singe NIC except for the NAS, which cannot get to line-speeds anyway despite having 4 ports. I could always get a 10Gbit addon-card later if needed.

So, once the new work horse was built and Hyper-V installed, it was only a matter of settings the constrained kerberos delegations correctly and start migrating machines. Live-migration was out the window due to CPU differences so I had to make the VMs “Migration enabled”. This is done for example with Powershell:

Set-VMProcessor -VMName NameOfVM -CompatibilityForMigrationEnabled 1

Note that the machine has to be turned off for this to run correctly. Once I’ve run the Move-VM command, I just run the above command again with -CompatibilityForMigrationEnabled 0 and the move was completed.

The new machine feels very much faster than the old one and a new install of WS2019 w/ desktop from the MDT took just 4 minutes to finish. I may do some iops testing further down the road but I expect that the numbers are pretty good for consumer/workstation grade hardware.