Microsoft Aims For More Federal Government Business With New Secured Hybrid Cloud And Productivity Tools

The salvo of new offerings will bolster Microsoft's challenge to AWS for lucrative contracts hosting sensitive government workloads and powering infrastructure deployed in global hot spots.

https://www.crn.com/news/cloud/300100316/microsoft-aims-for-more-federal-government-business-with-new-secured-hybrid-cloud-and-productivity-tools.htm

This was an interesting issue.  There were hundreds of moving parts that hit us at once.  Essentially the issue was that after Installing SharePoint, all our Server 2012 R2 servers would lock up saying “Applying User Settings” or “Applying Computer Settings”.  It would take considerable time (sometime days) for a VM to start.  Eventually the error would return, and there didn’t seem to be any rhyme or reason for the failure.  We opened an express MS ticket, and found the following information.  This seemed to be a 2008 server issue, but it appears it leaked into 2012!  So here is the fix!  At first we looked for NIC Driver problems, and Permissions.  But here is the fix, hope it saves someone some time.

Information on HTTPS Registry Settings: https://support.microsoft.com/en-us/kb/820129
Hotfix https://support.microsoft.com/en-us/kb/2379016 also includes manual fix.

Computer stops responding and appears to hang at the "Applying User Settings" or "Applying Computer Settings" stage of the logon process.

Note The following method can be used in Windows Safe Mode when you are not able to log on successfully to install the hotfix or fixit.

To work around the issue without installing the hotfix, create a DependOnService registry key to modify the behavior of HTTP.sys. This makes HTTP.sys depend on crytosvc service to be started first. To do this, follow these steps:

1.      Click StartStart button, type regedit in the Start Search box, and then press ENTER.  If you are prompted for an administrator password or for confirmation, type the password, or provide confirmation.
2.      Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP
3.      On the Edit menu, point to New, and then click Multi-string Value.
4.      Type DependOnService, and then press ENTER.
5.      Right-click DependOnService, and then click Modify.
6.      In the Value data box, type CRYPTSVC, and then click OK. 
7.      Exit Registry Editor.
8.      Restart the computer.

While it's not perfect, it does provide official documentation that issues will arise.

Below is the whole context of the ticket we submitted for documentation on this matter.

Subject: RE: [REG:115061612850630] PREM B Business hours|SharePoint Server 2013|Want to have documentation/approval from Microsoft not use [$] dollar sign on the service account name

Hello Ed,

After researching internal documentations and Collaborating with my colleagues, it was determined that the most specific things that we can provide on this is the following:

SharePoint leverages PowerShell heavily in its functionality and PowerShell views the '$' character as setting a variable to be used in storing information. Once it sees the '$' character, it views the text following it as the Variable name that it is to store the data in or retrieve data from, that may have been stored in it previously. Based on this, and how SharePoint does not utilize quotations when it is passing in an account name to the PowerShell command, it would have issues in some scenarios where it is passing in an account name that has '$' in it as it will have issues since it will most likely see the name provided as a Null variable and not as the String type of data that would occur if the '$' were not used in the account Name.

Since SharePoint uses PowerShell so heavily and is running PowerShell commands in the background when using the UI or manually running command, the use of the '$' character in account names or service account names does not fall within the Microsoft Best Practices for SharePoint 2010 or higher.

We will assist with troubleshooting issues that arise from the use of the '$' character in the Account names, however, there are some scenarios that may arise where there is nothing that can be done to correct the issue that it can cause. We would support each separate issue in a 1-on-1 basis where each specific error or issue is handled in a separate service request.

You can go through the following links to read up more on PowerShell variables:

https://technet.microsoft.com/en-us/library/ee692790.aspx
https://technet.microsoft.com/en-us/magazine/2007.03.powershell.aspx

Outside of this, and the fact that the use of the '$' character is against Microsoft Best Practices for SharePoint, we would not be able to provide any more in-depth details as to why not to use it in the Service Account names.

Based on all of the testing that has been performed on this issue, that would seem like some of the specific scenarios that you would run into with the use of the '$' character in the service account name. However, the most that we can provide is to have a separate service request to troubleshoot why it is causing those issues to see about resolving them, without having to change the service account being used.

Please let me know if you have any additional questions or concerns regarding the above information

Some time ago, I was asked to prove what errors would occur when you installed/Configured SharePoint 2013 with the $ in the service account name.  I know, weird suggestion, but our network Gods have decided that all service accounts must start with a $, and when we asked for an exception, we were told to prove it.  We asked Microsoft what the problem was since you only see it reference like below: 

The Microsoft representative we were presented with said "Hey, no problem, we'll support deployments with the $ in the name", and we were told to pound sand.  Today, we finally found proof, and I'm sharing it with you.

I had a fully functional test farm in a Hyper-V lab we used to test this.

How mine looked before

Note that this is a Hyper-V server test machine, with no GPO’s at all in our test environment at SERCO on a Stand-alone network. 

Here are the set of commands I used to show that it was currently fully operational before doing anything...

Get-SPServiceInstance | ? {($_.service.tostring()) -eq "SPDistributedCacheService Name=AppFabricCachingService"} | select Server, Status

Server                                                                  Status
------                                                                  ------
SPServer Name=SPED                                                      Online
SPServer Name=SPEDAPP1                                             Online

Get-SPServiceInstance | ? {($_.service.tostring()) -eq "SPDistributedCacheService Name=AppFabricCachingService"}

TypeName                        Status  Id
--------                        ------  --
Distributed Cache                Online  cda90885-4197-4bfe-b6d9-5f31fcd4e711
Distributed Cache                Online  d2f3d755-1538-4a20-972e-9eed188d5bc1 

CORP\sp_service was the account used.

Picture below

Again, I grab more details, before I break something...

PS C:\Users\sp_admin> Get-CacheHost
HostName : CachePort Service Name Service Status Version Info
-------------------- ------------ -------------- --------
SPEd.corp.contoso.com:22233 AppFabricCachingService UP 3 [3,3][1,3]
SPEDAPP1.corp.contoso.com:22233 AppFabricCachingService UP 3 [3,3][1,3]
Picture below: 

So now, I added our new account $SP_Test to the domain, made it a domain admin, and also added him to the ShellAdmin group (Add-SPShellAdmin), and we're ready to test.  Logged in with the new account and ran the following:  Note that NOWHERE Do I tell it to use the account name with the $ in it, it is configured when you add it.

$svc = Get-SPServiceInstance -server SPED | ? {($_.service.tostring()) -eq "SPDistributedCacheService Name=AppFabricCachingService"}
$svc.Unprovision()
$svc.delete()
$svc = Get-SPServiceInstance -server SPEDAPP1 | ? {($_.service.tostring()) -eq "SPDistributedCacheService Name=AppFabricCachingService"}
$svc.Unprovision()
$svc.delete()
Get-SPServiceInstance | ? {($_.service.tostring()) -eq "SPDistributedCacheService Name=AppFabricCachingService"} | select Server, Status
PS C:\Users\$SP_Test> Add-SPDistributedCacheServiceInstance
Add-SPDistributedCacheServiceInstance : Could not find a part of the path 'C:\U
sers\9376SP_Test\AppData\Local\Temp\DCacheAdministration2015-06-15-102722[9376]
.log'.
At line:1 char:1
+ Add-SPDistributedCacheServiceInstance
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (Microsoft.Share...ServiceInstance:
   SPCmdletAddDist...ServiceInstance) [Add-SPDistributedCacheServiceInstance]
  , CmdletInvocationException
    + FullyQualifiedErrorId : Microsoft.SharePoint.PowerShell.SPCmdletAddDistr
   ibutedCacheServiceInstance

Picture below:

Note how the $ was replaced with 9376 in the error line

Could not find a part of the path 'C:\Users\9376SP_Test\AppData\Local\Temp\DCacheAdministration2015-06-15-102722[9376].log'.

It is important to note that the file it is looking for was actually created, however it is EMPTY (0 Bytes),  and of course exists only at the location 'C:\Users\$SP_Test\AppData\Local\Temp\DCacheAdministration2015-06-15-102722[9376].log'.

To further explain to the Network gods that this was indeed an error, I reran the same exact commands using the SP_Admin account, Picture below, ignore the first three lines, that was a copy of when I added the account to the shelladmin prior to the test:

So we've documented the whole thing, and are preparing to argue with the Network gods.  I'll update this when/if we succeed.

Make sure to always check your farm for failed jobs.  These are often your first alert that something has gone wrong in your farm, and SP2013 has nice ways to troubleshoot right from Central Admin. To do so, in Central Admin under the Monitoring Section, click "Check Job Status".  You will get a brief listing of the scheduled, running (probably blank) and a brief history, mostly within the last few minutes.  Top Left, on the links section under the SharePoint banner, select "Job History".

 On the top right of the Job History page, there is a View dropdown, select "Failed Jobs"

You will then get a list of all the failed jobs like below:

When you click on the Failed link on the far right side, it will place detailed information about the failure at the top of your page as seen below.

If you click on the job title, you have the option to run it again, where you can check the history to see what happened when you reran it.  You can also disable the timer job in cases where it is filling up your error logs too much while you do research. 

In my next post I will detail how to fix this particular (and common) error by flushing the Configuration Cache on each of your Servers.

There is a ServerID GUID for SharePoint (all versions) that sits at the Registry location

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\15.0 named ServerId 

It's not easy to find the GUID in Central Admin.  But the powershell script below, will give you a list of the Servers, and their ServerID Guids:

get-spserver | format-list -Property Name,Id

Example:

PS C:\Users\sp_admin> get-spserver | format-list -Property Name,Id

Name : SPED
Id   : 0f31d37c-ab52-4e36-b02c-8706e9f60f3f

Name : SQLED
Id   : 8e869d81-58a7-46f9-83b6-a7ce46ad0e67

I work in an environment where they repeatedly fill in every single group policy entry available, and they often change ACL's on our server drives without warning/permission.  Attached is a working in progress list I've created over the years that contains everything from what kind of permissions accounts need, what registry entries are required, and what file permissions are required, along with hints and warnings about what happens when you've deleted them.  I've compiled this list over the last 4 years and keep finding more to tuck into it.  It's a fabulous reference.  Updated 24 Jun 2014