Kerberos fails with CIFS using AOVPN

Hey. Today I want to talk about an interesting case that involves Kerberos, Always On VPN and access to CIFS.
A customer has recently deployed Always on VPN in their infrastructure. Most clients worked well with that but a few had mixed issues with old VPN clients installed on some machines.

Cisco AnyConnect usually worked fine when installed, but there was another VPN client that disabled the IKE/EXT service which prevented the AO VPN IPSEC to work properly. So uninstalling that software solved the issues. The customer still had that installed on some clients as a backup solutions for when IPSEC was blocked at the source (for example hotels, airports etc.)

But hey! I mentioned Kerberos, how does that come into play?

Disclaimer: This post has might have little to do with Always On VPN, but the issue manifested itself when connected through AOVPN.

Well, there was a few clients that actually connected fine. They could ping stuff on the network and everything seemed fine until they tried to access the file server. They got prompted with credentials stating that they had no access to the domain controller even though I could actually get LDAP access with AD-powershell, so LDAP was obviously working. This was interesting. After a few log checks on both sides of the fence, nothing popped out. So I decided to install Wireshark on the domain controller to try and figure things out. This gave me lots of new and critical information.

I could clearly see that Kerberos was not working. UN_SUPPORTED when the client tried to get a Kerberos ticket from the KDC. So I checked the DC logs and found issues with the Kerberos certificate.
Sorry for the lack of screens, this all happened really fast and I was definitely not allowed to screenshot the customers data.

It turned out that the domain controller was using previously issued certificates from and old and retired Certificate Authority. So I deleted all of them and issued the domain controller new certs for domain controller authentication and Kerberos authentication. Now my senses were tingling since I knew that this would fix the problems. And lo and behold, it did! The troubled clients worked right away.

But, one thing was still bothering me and is still bothering me. Since this was a server side fix, why did not all the clients have this issue? Why was only a select few clients using Kerberos auth? The customer is telling me that all computers are equal and installed from the same image and are getting the same policies. So why are only a select few using Kerberos (that failed). At the time of writing, I don’t know. This happened just recently. Maybe you have some ideas? Feel free to contact me on twitter (@mrblackswe) or post a comment below. Something tells me that the clients are not equal at all, despite what the customer is telling me (Usually the case). The clients are Windows 10 Pro 1809, afaik.

Upgrading the lab

Good day to you. Today I’ve done a little write-up about my home lab equipment. I was noticing a few slowdowns once I got around 10 vm’s running on my old “server”, which was an older gen Intel E3 1240 series CPU running 32Gb of DDR3 RAM, SSD cache and spinning HDDs for mass storage. Since the stuff was closing in on 5 years of service I thought it was time to invest in some new hardware.

This time I decided to go AMD and specifically the Threadripper 1920X with 12 cores/24 threads and 64Gb of DDR4 memory. So I doubled the RAM amount and also the RAM speed and the core count tripled with higher clocks as well. All flash this time as well did it’s thing for sure as there are now 5 SSD in a combination of SATA and M2 drives in RAID-0 hosting the VM’s through storage spaces. As far as I know, the only limitation of running Hyper-V on Threadripper is that it can’t do nested virtualization, but I haven’t verified that myself yet as it is a feature I don’t specifically need.

I did not invest in networking at all since I don’t really need more that 1 Gbit externally from the host. Everything else in my network runs a singe NIC except for the NAS, which cannot get to line-speeds anyway despite having 4 ports. I could always get a 10Gbit addon-card later if needed.

So, once the new work horse was built and Hyper-V installed, it was only a matter of settings the constrained kerberos delegations correctly and start migrating machines. Live-migration was out the window due to CPU differences so I had to make the VMs “Migration enabled”. This is done for example with Powershell:

Set-VMProcessor -VMName NameOfVM -CompatibilityForMigrationEnabled 1

Note that the machine has to be turned off for this to run correctly. Once I’ve run the Move-VM command, I just run the above command again with -CompatibilityForMigrationEnabled 0 and the move was completed.

The new machine feels very much faster than the old one and a new install of WS2019 w/ desktop from the MDT took just 4 minutes to finish. I may do some iops testing further down the road but I expect that the numbers are pretty good for consumer/workstation grade hardware.

Powershell for printer creation, with bonus feature.

Hey, the Finn here again with another random post.
A client reached out to me since they are in the process of switching print servers. They have a lot of printers, to say the least. They are all managed manually and they felt that they would like avoid the hassle of creating all the print queues manually on the new server. Powershell to the rescue!

They gave me a list with desired things that the script should do. I might have over delivered on that, but I like Powershell. It’s freaking awesome for so many things in Windows and beyond.

So, let’s have a look at what I did. Script full of comments for explanation ofc and split in regions for easy reading.

Since we will be dealing with installing printer ports, printers and DNS-records, I figured it might be a good idea to check that the script is “run as admin” or just end it right there.

Once that is out of the way we need to establish some customer specific variables such as the input file (CSV), the DNS server name, the zone in which to create the record and lastly the template printer that we want to use for custom settings.

After the initial variables now defined and checked by the script, we can move on to creating the actual functions that do stuff. So first we create the DNS-record.

This requires the DNSServer RSAT module to be installed on the machine running the script and you will not be able to run at all if that is missing since the script actually checks for that as well. However, this is really easy with Windows 10. Just run this powershell command: Add-WindowsCapability -Name Rsat.Dns.Tools~~~~ -Online and you are all set.

For the next portion of the script, we’ll need to create the network port on which the printer resides. We’ll get som in-data from the CSV that we utilize here. We’ll do a try/catch approach. Also, we will try to create the printer itself right after. Log to screen and file if something goes wrong.

Now, we have the printer created and ready to go. Now we will set the permissions on it from the template printer and also any other settings that are presented in XML from the originating printer.

That would be the last step in this excercise of the functions. Now, onto the trigger of the whole thing.

We do some basic checks before kicking things off. Making sure that there is a template printer, that we have the necessary powershell modules we need and also that the input-csv actually exists, before doing anything else.

Lastly we store the variables as arguments to send to the different functions and voilá, the script is done.

Feel free to comment all of my mistakes. Download file here.

Windows Admin Center certificate

Hey there. The Finn here with another little quick update. This time on Windows Admin Center and certificates.
You are using Windows Admin Center, right? It’s good stuff. You like good stuff, right?

Self-signed is the default option, but man is that an ugly way to go.
Here is how you update your Windows Admin Center certificate without any third party tool.
Open up Powershell, run dir cert:\localmachine\my to get a list of installed certificates.
Copy the thumbprint for the certificate you want to use.

Then we will check for the application ID that WAC uses and the port it is bound to (Default is 6515) with netsh http show sslcert

Once we have the new thumbprint and the appid, we can go ahead and delete the existing certificate binding, again with netsh http delete sslcert= (or the port you are using)

An finally we can bind the new certificate to the WAC port.

And that is all there is to it. However… I saw a retweet from my MVP friend Andy Syrewicze who linked to a colleague of his who wrote a little handy tool that does this switch-a-roo as well.
You can find that tool here: