Index index everywhere but not a result in sight.

Send to Kindle

I have been doing a bit more tech work than normal lately – SP2010 popularity I guess, and was asked to remediate a few issues on a problematic server that I hadn’t set up. The server in question had a number of issues (over and above the usual “lets all run it as one account” type stuff) that had a single root cause, so I thought I’d quickly document the symptoms and the cause here.

Symptom 1: SQL Server Event ID 28005:

“An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user ‘DOMAIN\someuser’, error code 0x5”

This error would be reported in the Application log around 70 times per minute. As it happened, I had removed the account in question from running any of the SharePoint web, application or windows services, but this still persisted. I suspect SharePoint was installed as this account because it was the db owner of many of the databases on the SQL Server. Whatever the case, SQL was whinging about it despite its lack of actual need to be there.

Symptom 2: Event ID 4625:

At a similar rate of knots as the SQL error, was the rate of 4625 errors in the security log. These logs were not complaining about the account that the SQL event complained about, but instead it complained about ANY account running the SQL Server instance. I tried network service, a domain account and a local account and saw similar errors (although the local one had a different code).

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Event ID:      4625
Task Category: Logon
Keywords:      Audit Failure
Description:  An account failed to log on. 

Subject:
    Security ID:        NETWORK SERVICE
    Account Name:        COMPUTER$
    Account Domain:        DOMAIN
    Logon ID:        0x3e4 

Failure Information:
    Failure Reason:        Unknown user name or bad password.
    Status:            0xc000006d
    Sub Status:        0xc0000064 

Process Information:
    Caller Process ID:    0x7e0
    Caller Process Name:    E:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe 

Detailed Authentication Information:
    Logon Process:        Authz   
    Authentication Package:    Kerberos

When using a local, rather than a domain user account the code was:

Failure Information:
    Failure Reason:        An Error occured during Logon.
    Status:            0xc000005e
    Sub Status:        0x0 

Symptom 3: Search only working for domain administrators

On top of the logs being filled by endless entries of the previous two, I had another error (the original reason why I was called in actually). A SharePoint search would yield zip, nada, zero, no results for a regular user, but a domain administrator could happily search SP2010 and get results. (well actually regular users did get some results – people searched actually worked fine).

The crawler was fine and dandy and the default content source had not been messed with. There were no errors or logs to suggest anything untoward.

The resolution:

It was the second symptom that threw me because I thought that the problem must have been kerberos config. But I quickly discounted that after checking SPN’s and the like (notwithstanding the fact this was a single server install anyway!)

On a hunch (helped by the fact that I had dealt with the issue of registering managed accounts not so long ago), I concentrated on the user account that was causing SQL Server all the trouble (Event ID 28005). I loaded up Active Directory and temporarily changed the security of this user account so that “Authenticated Users” had “READ” access to it.

image

As soon as I did this, both event ID 28005 and 4625 stopped.

I then checked the search (symptom 3) and it was still barfing. In this case I decided to turn up the debug juice on the “Query” and “Query Processor” functions of “SharePoint Server Search”.

image

After upping the level of verbosity, I found what I was looking for.

08/12/2010 22:13:41.00     w3wp.exe (0x2228)                           0x1F0C    SharePoint Server Search          Query Processor                   g2j3    High        AuthzInitializeContextFromSid failed with ERROR_ACCESS_DENIED. This error indicates that the account under which this process is executing may not have read access to the tokenGroupsGlobalAndUniversal attribute on the querying user’s Active Directory object. Query results which require non-Claims Windows authorization will not be returned to this querying user.    debd2c54-d6a5-41b8-bf26-c4697b36f4d4

I knew immediately that this was very likely the same issue as the first two symptoms and when I googled this result, my Perth compatriot, Jeremy Thake had hit the same issue in July. The fix is to add your search service account to a group called “Pre-Windows 2000 Compatibility Access” group. This group happens to have the right required to read this attribute. Whether it is the same attribute I needed for my SQL issue, or the registering managed accounts issue I don’t know, but what I do know is that this group loosens up permissions enough to cure all four of these issues.

The little security guy in me keeps telling me I should confirm the least privilege in each of these scenarios, but hey if Microsoft are saying to put the accounts into this group, then who am I to argue?

Finally, it turns out that SP2010 re-introduced something that was in SP2003. A call to a function called AuthzInitializeContextFromSid which seems to be the root of it all. Apparently it was not used in SP2007, but its sure there now. I assume that one of the many stored procedures that SharePoint would call in SQL may have been the cause of Symptom 1. When you look at Symptom 2, it now makes sense because AD was faithfully reporting an access denied when the call to AuthzInitializeContextFromSid was made. It reported SQL Server as the culprit, I assume, because of a stored proc doing the work perhaps? It just sucks that the security event logged is a fairly stock message that doesn’t give you enough specifics to really work out what is going on.

Anyway, I hope that helps someone else – as googling the event ID details is not overly helpful

 

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle
Bookmark the permalink.

7 Responses to Index index everywhere but not a result in sight.

  1. Scott says:

    Very timely information. By searching just the event id combo I came across your post and by adding the appropriate domain account to the Pre-Windows 2000 Compatibility Access Group the noise was silenced. Although, the errors did not have an adverse effect on anything other than generate a high volume of errors.

    Many thanks!

  2. Dave says:

    Helped me out too. Well done!

  3. Kedric says:

    Thank you very much!

  4. Philip says:

    Great job! Thanks very much for saving me the hair pulling out session that I am sure you went through!

  5. David says:

    worked for me after hours of pain

  6. James says:

    Hi Paul, thanks so much for posting this! Doubtlessly saved me a chunk of time – i can now go straight back and try to “wow” my test users again, before they wander off 🙂

  7. rainabba says:

    Over many months and far too many hours I’ve searched for a solution to this issue and in less than 5 minutes it was resolved using your advice. Thank you so much for taking the time to share your solution.

Leave a Reply

Your email address will not be published. Required fields are marked *