Darn IIS and Service Packs!

Send to Kindle

I thought that I was getting pretty good at deciphering odd SharePoint errors. After all, most of the time they are not very friendly! After the subsite/site collection issue of yesterday, you would think I would be in for an easier day today!

But it was not meant to be…


I was out at a client site today, basically going through a SharePoint installation with the very nice infrastructure people. They had a single Win2003 Standard server all ready and waiting. I sat back while they took the driver’s seat. They installed while I passed on lots of insightful, brilliant and invaluable advice as all high class consultants like me do :-P. All was going swimmingly, IIS and SQL was installed easily, security was locked down, service accounts created and the SharePoint installation went through without a hitch.

(By the way, if you don’t want to read my tale of woe then skip to the end, but writing this is cathartic, so please indulge me 🙂 )

So, as I was saying, the install of SQL, IIS and SharePoint went fine. Ran the 9 step SharePoint configuration wizard and that was all good!

Then…

Then I started up "SharePoint Central Administration" on the SharePoint server and received an authentication prompt.

"Aha!", I confidently pronounced to said client, "I have not uninstalled the ‘Internet explorer enhanced security configuration’ option, nor have I added central admin to the trusted sites zone in MSIE".

So I did those things and  was *still* prompted for authentication.

"Aha", I exclaimed, "maybe it’s a DCOM issue. I blogged about this". I then confidently explained that sometimes when attempting a secure by default install, you do have to loosen up DCOM permissions here and there, especially when the server performs multiple roles. So I checked the event logs, and sure enough saw a few DCOM complaints. To satisfied nods all round, I corrected them with DCOMCNFG.

I was *still* prompted for authentication.

"Hmm, okay", I said, thinking about some of the other zillion stupid authentication issues you can have with IE/Zones/IIS/Impersonation and the likes. "It might be a Kerberos thing…" Even though I can’t see how, and the fact that I chose NTLM when I ran the configuration wizard, I thought that IIS might be trying to authenticate with Kerberos before NTLM as per KB215383. After all, the article says:

"By default, the NTAuthenticationProviders metabase property is not defined when you install IIS 6.0. IIS 6.0 uses the Negotiate, NTLM parameter when the NTAuthenticationProviders metabase property is not defined. Therefore, you do not have to configure IIS to use the Negotiate,NTLM property value unless the default value has been overwritten".

So, using adsutil.vbs I forced it to be NTLM only and retried…

I was *still* prompted for authentication.

At this point I said to the client, "Let’s go have lunch!". All agreed and I continued to ponder over a large juicy burger and a coke.

So, upon my return, the clients left me for half an hour to see if I could nail it. I noticed that when I entered valid credentials at the authentication prompt, even the domain admin account, it popped up again for credentials.

If I clicked cancel, after 3 cancel attempts, it would let me into Central Administration. If I entered credentials and clicked ok, it would also let me in eventually, but it would take more attempts than three.

"Aha, I have seen this before", I said to the empty room. "This sort of thing happened to me when the SharePoint configuration account didn’t have appropriate permissions to the ‘Temporary ASP.NET Files’ folder under the ‘C:\WINDOWS\Microsoft.NET\Framework\<version>\’ folder."

So, I checked the permissions here and it looked good. The config account was a member of all local IIS/WSS worker process groups that it needed to be. But to be sure, I enabled failure auditing on the C:\WINDOWS\Microsoft.NET\Framework and set the local security policy to log pretty much any audit event that was a failure.

Tried again, reproduced the problem and the security event logs showed no files access/create failure.

Decided to see if there was anywhere else on the filesystem causing it, I extended auditing to the whole filesystem.

Still no-go, this was not a filesystem permission issue. It was also not Kerberos/NTLM I reasoned, because if it were, why was I able to eventually get into central administration?

So, I decided to look at the IIS logs for the central admin web application. The first part is pasted below. (I’ve snipped some of the superfluous log into for readability)

#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2007-11-15 06:07:07
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
2007-11-15 06:07:07 W3SVC1117623206 192.168.6.11 GET / 401 2 2148074254
2007-11-15 06:07:07 W3SVC1117623206 192.168.6.11 GET / 401 1 0
2007-11-15 06:07:13 W3SVC1117623206 192.168.6.11 GET / 200 0 0

Now, the sequence above (the 401 2 error followed by a 401 1 and 200) is the normal integrated authentication sequence. As described in this KB article 264921, when the browser makes a request, it always considers the first request to be Anonymous. Therefore, it does not send any credentials. If the server does not accept Anonymous, the IIS server responds with an "Access Denied" error message (401 messages) and sends a list of the authentication types that are supported. If the browser supports the authentication type, it will use it and successfully retrieve the page (200 message).

But now, let’s look at the next sequence…

2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /default.aspx SOMESITE\Administrator 200 0 0
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /_themes/CENTRLAD/CENT1011-65001.css 401 2 2148074254
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /_themes/CENTRLAD/CENT1011-65001.css 401 1 0
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /_themes/CENTRLAD/CENT1011-65001.css 200 0 0

Now, what we see above is that /default.aspx was retrieved successfully. Then the browser tried to retrieve a CSS file from /_themes and we are back into the integrated auth sequence. This is probably because _themes is another web application (I’ve never checked). The point is, things are still going to plan.

Then, a little further down, we see this…

2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 2 2148074254
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 0
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 2148074248
2007-11-15 06:07:34 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 0
2007-11-15 06:07:34 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 2148074248
2007-11-15 06:07:34 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 0
2007-11-15 06:07:34 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 2148074248

[snip]

2007-11-15 06:07:36 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 0
2007-11-15 06:07:36 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 2148074248

So, what’s going on here? Instead of the 402 2, 401 1 and 200 sequence as it is supposed to be, we now have 401 2, 401 1 and then another 401 1 with a win32 code of 2148074248. And, what is this WebResource.axd anyway? That seems to be the source of this issue.

So, what did uncle google have to say about WebResource.axd and 2148074248?

Lot’s actually, but not quite what I was looking for. But, then I had a lucky break and found a newsgroup posting from Jan 07 to the microsoft.public.sharepoint.portalserver.development group.

David Schneider wrote this in answer to the original request:

"I have exactly the same problem on a customer site. The Central Admin of SharePoint use the httphandler webresource.axd to publish some javascript code. But when the browser try to load this javascript code via the webresource.axd, the current user isn’t unable to authenticate.

I did a lot of debugging (IIS log, file security, policies, fiddler, etc.) but I didn’t find a solution yet. Finally I opened a case at Microsoft Partner Support. And now I’m waiting for a response of these guys. "

Fortunately for me, he followed up…

"I had to re apply Windows 2003 SP1 and then the problem was gone. I had reinstalled IIS and the reinstallation has overwritten the SP1 DLLs with the ‘original’ DLLs"

Now, I checked with the client. The server was Windows 2003 SP2 and they used the Windows 2003 SP1 media to install IIS prior to the SQL Server install. My hopes jumped in nerdy anticipation! Could it be the same issue?

As it turned out it *was*, and I kicked myself for not including this requirement in my "insightful, brilliant and invaluable advice" to my client. After re-applying service pack 2, the problem was solved, the client was happy.

So, another day ends and another weird error sorted! One day I will actually not encounter this sort of issue and instead have a day where I can blog on governance topic (my actual area of interest!!).

Wouldn’t that be nice!

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle
Bookmark the permalink.

19 Responses to Darn IIS and Service Packs!

  1. Wellick says:

    Would you mind if I translate the gist of your excellent article and and post it to my blog (with a link to you, of course)?
    I’ve never encountered this issue, but who knows what could happen tomorrow =)

  2. admin says:

    no probs at all mate..

  3. rich says:

    I seem to have the exact same problem including steps I attempted prior to SP2. However, reinstalling Win2K3 SP2 on both WFE’s did not seem to resolve the problem.

    I have set the NTAuthentication provider to “NTLM” using adsutil.vbs.

    I have set launch permissions for DCOM IIS WAMREG admin service to have launch permissions for application pool users.

    No proxy server is enabled in IE.

    When I open a new browser I usually get the DNS error as an error and the logs will have the 401.2 errors as you did. Cntrl-R one or more times will prompt me for credentials and the page will render although some of the page (CSS, images) may be missing.

    I really thought the SP2 fix would put this to rest.

    Any thoughts would be appreciated!

  4. rich says:

    Turns out that the infrastructure team had NLB multi-cast turned on but Cisco router multicast disabled by default. DOH!

  5. admin says:

    hehe, nice work sorting that one

  6. jeff says:

    Thank you! This post basically saved my life!

  7. Pingback: XinSpace » weird sharepoint installation

  8. Paul Lynch says:

    There must be something broken with the service pack installation routine because the ‘received wisdom’ is that since Windows 2000 it has not been necessary to re-apply service packs following a system change, unless you apply a hotfix. This is documented here :
    http://support.microsoft.com/default.aspx?scid=kb;en-us;274215

    Full marks for perseverance though 😉

    Regards,

    Paul

  9. Jay Galego says:

    This was the exact problem I was having. The solution worked PERFECTLY!!!! Thanks

  10. Pingback: CleverWorkarounds » Not Good Enough Stories…

  11. Graham says:

    Ohhhhh this blog post ends my 2 very long days of agony trying to circumvent this stupid problem. How can one simple error code have so many possible fixes? Thanks for posting your solution.

  12. Pingback: .. of ones and zeros .. » Blog Archive » IIS and Windows 2003 Mismatched Service Packs

  13. TROC says:

    Thanks Paul – This Fixed me up right!

  14. Eugene Tsimberg says:

    Thank you for your post. I spent hours banging my head against this, and trying all the other common issues until I read this. Great fix.

  15. Beats says:

    YEEEEEES…. You’re the man! I have been googling for about 2 days looking for:

    WebForm_PostBackOptions is undefined
    webresource.axd
    401 1 2148074248
    NTLM MOSS2007 WSS3.0

    and a their combinations…. I don´t know really how i end here, but now i’m happy as after w2k3SP2 reapply the issue is gone….

    THANKS THANKS THANKS THANKS

  16. TheDarkSide says:

    Yo sé de un tal Beats que le debe unas cervecitas a un tal admin por salvarle su precioso culo…. 😉

  17. Ravi Patel says:

    Thanks a ton. We thought this issue was invincible. It was because of some Windows SP 2 problem and we were breaking our head with MOSS. How did you figure it out!!??

  18. talc says:

    Thanks, something else to try….

  19. Praveen Behara says:

    I would say your article is a lucky find for me. It was bang on target and my issue got resolved. After installing SP2 again, I extracted the I386 folder from the SP2 exe file. But that itself missed some files (???).. IIS installation couldn’t find files. I used another extract after my collegue (don’t know where she got that).. but when I completed my install, ran the ASPNET_REGIIS -r -enable, I was able to deploy application, look up my BizTalk BAM portal and browse them.. Great work with the deciphering this issue.

Leave a Reply

Your email address will not be published. Required fields are marked *