How to find user directories from unauthorized Users

We wanted to clean up our Terminal Servers a little. So we wanted to delete existing user directories from unauthorized users. As a side effect we also got all authorized users never logged in on the server.

# Find User directories on a server (eg Terminal Server) from unauthorized Users (eg deleted Users) and delete them.
# Find authorized Users not logged in yet (authorized users without user directory)
import-module activedirectory
# The directory name has to be equal to the Sam Account Name, the path has to end with ""
$path = "C:Users"
# Subgroups are not supported
$adgroup = <AD group with all authorized Users>
$users = get-adgroupmember $adgroup |select samaccountname
$Users = $Users | add-member -membertype Aliasproperty -Name name -value samaccountname –passthru
$dirs=Get-ChildItem $path | where {$_.Attributes -eq 'Directory'} | select Name
$delme=Compare-object $dirs $users -property name | ? {$_.sideindicator -eq "<="}
$nie = Compare-object $dirs $users -property name | ? {$_.sideindicator -eq "=>"}
#delete them all
foreach ($d in $delme) { $dd = $path + $; rm -r $dd }