Just a quick note to mention that I’m off travelling again, this time swapping 39 degree Celsius summer weather of Perth for somewhere between –6 to 5 degrees of Canada. I’ll be spending a week in Canada running two classes – one public and one private. The first class is a public SharePoint Governance and Information Architecture class running in Vancouver. MVP Michal Pisarek of SharePointAnalystHQ fame will be there and it should be a terrific two days of learning how to think a little differently to govern SharePoint strategy and deployment. You will learn a bunch of new skills, techniques and perspectives. Best of all, the skills learnt are applicable for many other types of complex projects.

The class flyer is here: http://www.sevensigma.com.au/wp-content/uploads/downloads/2011/02/SPIA.pdf

The registration site is here: http://spiavancouver.eventbrite.com/

In terms of course coverage and content it is worth noting the research performed by the Eventful group (who run the Share conferences). According to them, the hot topic areas for SharePoint are governance, user adoption, change management, information architecture and user empowerment. These sort of topics are the sort where plenty of people tell you what the issues are, but are typically lighter on what to do about them. This class covers why this is, as well as dealing with all of these areas and presents detailed strategies, tools and methods to address them. Furthermore, aside from the 500+ page manual of meaty governance goodness, as a take home, we supply a CD for attendees with a sample performance framework, governance plan, SharePoint ROI calculator and sample mind maps of Information Architecture.

At last count there were 5 places left for the Vancouver class, so if you have been pondering if it is a worthwhile class, check out some of the feedback from the class web site. Also, if you know anybody who might be interested in attending, please pass the course flyer and registration site details to them. We always end up with people who tell us “Ah – if only I knew about the class!!”

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

www.hereticsguidebooks.com

                      

No Tags

Each student lab setup is two virtual machines. The first being a fairly stock standard AD domain controller and the second being a SQL/SharePoint 2010 box. Someone else set up the machines, and I came in to make some changes for the labs next week. But as soon as I fired up the first student VM’s I hit a snag. I loaded SQL Server Management Studio, only to find that I was unable to connect to it as the domain administrator, despite being fairly certain that the account had been granted the sysadmin role in SQL.

Checking the event logs, showed an error that I had not seen before. “Token-based server access validation failed with an infrastructure error”. hmmm

Log Name: Application

Source: MSSQLSERVER

Date: 31/07/2010 6:45:37 p.m.

Event ID: 18456

Task Category: Logon

Level: Information

Keywords: Classic,Audit Failure

User: TRAINSBYDAVE\administrator

Computer: SP01.trainsbydave.com

Description:

Login failed for user ‘TRAINSBYDAVE\administrator’. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors.

[CLIENT: <local machine>]

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="MSSQLSERVER" />

<EventID Qualifiers="49152">18456</EventID>

<Level>0</Level>

<Task>4</Task>

<Keywords>0×90000000000000</Keywords>

<TimeCreated SystemTime="2010-07-31T06:45:37.000000000Z" />

<EventRecordID>8281</EventRecordID>

<Channel>Application</Channel>

<Computer>SP01.trainsbydave.com</Computer>

<Security UserID="S-1-5-21-3713613819-1395520312-4192346095-500" />

</System>

<EventData>

<Data>TRAINSBYDAVE\administrator</Data>

<Data> Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors.</Data>

<Data> [CLIENT: &lt;local machine&gt;]</Data>

As it happened, whoever set it up had SQL Server in mixed mode authentication, so I was able to sign in as the sa account and have a poke around. All things considered, it should have worked. The user account in question was definitely in the logins and set with sysadmin server rights as shown below.

Uncle google showed a few people with the error but not as many as I expected to see since half the world gets nailed on authentication issues. I also took the event log suggestion and looked for a previous error. A big nope on that suggestion. In fact in all respects, everything looked sweet. The machine was valid on the domain and I was able to perform any other administrative task.

Finally, I removed the TRAINSBYDAVE\administrator account from the list of Logins in SQL Server. It gave the unsurprising whinge about orphaned database users, but luckily for AD accounts when you re-add the same account back it is smart enough to re-establish the link.

As soon as I re-added the account, all was good again. If I was actually interested enough I’d delve into why this happened, but not tonight – I have another 6 student machines to configure

Goodnight!

                      

No Tags

Note: This post and content is really going make utterly no sense unless you have watched Zoolander. Even if you have seen the movie, before you launch into the webcasts, some scene setting is required.

The business need

Some time ago, Peter and I were contracted by the Derek Zoolander School for the Really, Really Good Looking after Derek saw Microsoft’s new SharePoint diagram when he accidentally picked up a “Computerworld” magazine. Apart from matching Derek’s suit colour rather nicely, the diagram captivated his imagination with the notion of “Insights”.

Zoolander thought that “Insight”, sounded like the perfect look to follow up from the highly successful “Magnum”, which he used to save the Malaysian prime ministers life. He took the diagram to his wife, and demanded that he must have “Insights” at all costs.

Zoolander’s wife saw the business problem that “Insights” would help to address. You see, the Derek Zoolander School for the Really, Really Good Looking, at great expense, custom developed an ERP system to manage everything you needed to know about male models. The system was called the “Computerised Records for Attractive People”…

The CRAP system stored all sorts of interesting information about male models, such as tracking their “hotness”, as well as important detail such as stated age versus actual age, and any cosmetic procedures that they have undertaken. After a long and expensive consultation, Peter and I concluded that SharePoint 2007, integrated with SQL Reporting Services, was the perfect solution to create the all important “Insights” that Zoolander so desperately needed.

As a result, we conducted a project kickoff meeting with Hansel and Peter tried to explain the architecture of reporting services using a nice diagram.

… but we worked out pretty quickly that this was not the way to explain how it all worked to poor old Hansel…

So instead, we went the live demo route. Being male models, custom development was totally out of the question. This solution had to be done using all out of the box methods in a quick and easy manner. Below are the four live demos that were recorded and now you can use them as inspiration for your own male modelling school.

• Our first webcast illustrates how we were able to create a meaningful report from the CRAP system within five minutes.
• The second webcast expanded on this idea, by illustrating how reports can be parameterised and linked together for drilldown reporting.
• The third demo modifies the user profile store to allow for recording of each users unique ID in the CRAP system
• The last webcast strings this all together for the final demonstration where we pimp the report to make it dynamic with no custom code.

The 5 minute report	Drilling down with Derek
User Profiles for the really really good looking	Pimp my report

We hope you find some value from these webcasts and we look forward to hearing about your hot new look as a result!

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

In the words of Doctor Smith from lost in space, while everyone else was in Vegas having a grand old time, I was at a client site, having to come to grips with the beast known as Windows Small Business Server 2008. I rarely work with SBS2003 and had never used SBS2008 until now.

This was one of those engagements that is somewhat similar to those awful dreams that you have when you are trying to get to some place, but you never quite get there and your subconscious puts all sorts of strange and surreal obstacles in your path. In my case, the surreal obstacles were very real, yet some of them were really really dumb. Whatsmore, it is a very sad indictment on IT at several levels and a testament to how complexity will never be tamed with yet more complexity.

As a result, I really fear the direction that IT in general is heading.

So where to begin? This project was easy enough in theory. A former colleague called me up because he knew of my dim, dark past in the world of Cisco, Active Directory and SharePoint. He asked me to help put in SBS2008 for him, configuring Exchange/AD/SharePoint and migrating his environment over to it.

“Sure”, I say, “it’ll be a snap” (famous last words)

I haven’t use the coffee or tequila ratings for a while, so I thought that this post was apt for dusting them off. If you check the Why do SharePoint Projects Fail series, you will see that I use tequila shots or coffee at times. In this case, I will use the tequila shots to demonstrate my stress levels.

Attempt 1

We start our sorry tale a few weeks ago, where my client had ordered Small Business Server 2008 and the media/key had not arrived by the time I was due to start. The supplier came to the rescue by sending them a copy of the media and promised to send the license key in a couple of days.

The server was a HP Proliant DL360 G6, a seemingly nice box with some good features at a reasonable price. HP/Compaq people will be familiar with the SmartStart software and process, where instead of using the windows media, you pop in the supplied SmartStart CD and it will perform some admin tasks, before asking for the windows media, auto-magically slipstreaming drivers and semi-automating the install.

On client machines, I never use the CD from the vendor because there is always too much bloatware crap. However on servers I generally do use the CD, because it tends to come with all the tools necessary to manage disk storage, firmware and the like. I dutifully popped in SmartStart CD, answered a few basic questions, and it asked for the windows media CD.

Cleverworkarounds stress rating: Good so far

Next it asked me for the Windows SBS2008 licence key. Of course, I was using media that had been lent to us from the supplier because my client’s media (and keys) had not arrived. Thus, since I did not have a license key I was unable to proceed with the install using this preferred manner. HP, in their infinite wisdom, have assumed that you always have the license key when you install via their SmartStart CD, despite Microsoft giving you 30 days to activate the product. To be fair on HP, they are hooking into Microsoft’s unattended installation framework, so perhaps the blame should be shared.

All was not lost however. The SmartStart CD can be run after windows has been installed. It then will install all the necessary “HP bits” like graphics and system board drivers. So I booted off the Windows CD and fortunately, windows installer detected the HP storage controller and the disk array,  and proceeded to let me partition it and install.

Cleverworkarounds stress rating: Minor annoyance, but good so far

Small Business Server 2008 did its thing and then loaded up a post install wizard that sets the timezone, active directory domain name and the like. At a certain step when running this wizard, SBS2008 informed me that there was no network card with a driver loaded, so it could not continue. As it turns out, Microsoft’s initial SBS2008 configuration wizard simply will not proceed unless it finds a valid network card. But since this is the pre-install wizard, we are not yet at the point in the installation where we have a proper windows desktop with start menu and windows explorer. All is not lost (apparently), because SBS allows you to start device manager from within the wizard and search for the driver.

Fair enough, I think to myself, so I pop in the HP SmartStart CD and tell device manager to search the media.

Cleverworkarounds stress rating: Spider senses tingle that today might not turn out well

Windows device manager comes back and tells me that it cannot find any drivers.

Why?

Well after some examination, the SmartStart drivers are all self extracting executables and therefore device manager could not find them when I told it to. Of course, the self extracting zip files have hugely meaningful names like C453453.EXE making it really obvious to work out what driver set is the one required… not!

Luckily, Ctrl+Alt+Del gave me task manager which allowed me to start a windows explorer session, and I was able to browse to the CD and run the autorun of the SmartStart CD manually. This loaded up HP’s fancy schamsy driver install software that produces a nice friendly report on what system software is missing and proceeds to install it all for you.

SmartStart did its thing, finding all of the driverless hardware and installed the various drivers. A few minutes and a reboot later and SBS2008 reruns its configuration wizard and this time finds the network and allows me to complete the wizard. This triggers another thirty minutes of configuration and another reboot and we have ourselves a small business server!

Cleverworkarounds stress rating: Spider senses subside – back on track?

Next I did something that is a habit that has served me well over the years (until now). I reran the Smartstart CD, now with a network and internet access. This time I told the driver management utility to connect to HP.COM. It scanned HP and reported to me that most drivers on the SmartStart CD were out of date. This is unsurprising because most of the time I do server builds for any vendor, I find that about half of the drivers, BIOS and various firmwares have been replaced by newer versions since the CD was pressed.

Since this is a brand new server build, it is a habit of mine to upgrade to the latest drivers, BIOS and firmware before going any further.

Among the things found to be out of date was the BIOS, the firmware on the RAID Storage controller as well as the network card. The SmartStart software downloaded all of these updates, and another reboot later, all are installed happily. Another hour of patching via Windows update, and we have a ready to go SBS2008 server with WSS3, Exchange, SQL Express and WSUS all configured automatically for you.

Cleverworkarounds stress rating: This SBS2008 stuff isn’t so bad right?

Okay, so things were good so far, but now here is where the fun really begins.

Windows 2008 SBS comes with a pre-installed WSS3 site called http://Companyweb. As we all know, search completely sucks in WSS3. It has a bunch of limitations and isn’t a patch on what you get with MOSS. But no problem – we have Microsoft Search Server Express now, a free upgrade which turns WSS search from complete horribleness to niceness fairly quickly.

For those of you reading this who run WSS3 and have not installed Search Server Express, I suggest you investigate it as it does offer a significant upgrade of functionality. Search server express pretty much makes WSS3 have the same search capabilities of MOSS 2007.

So, I proceeded to install Search Server 2008 express onto this Small Business Server 2008 box. I have installed Search Server Express quite a few times before and I have to admit, it is a tricky install at times. But given that this was a fresh Small Business Server 2008 install and not in production, as well as having successfully installed it on Small Business Server 2003 previously, I felt that I should be safe.

I commenced the Search Server 2008 express install, and the first warning sign that my day was about to turn bad showed itself. The install of search server express only allowed me to choose the “Basic” option. The option that I wanted to use, “Advanced” was greyed out and therefore unavailable.

Cleverworkarounds stress rating: Spider senses tingling again

Knowing this server was not in production, I went ahead and allowed Search Server Express to install as per the forced basic setting. The install itself appeared to work, but it died during the SharePoint configuration wizard. It specifically crapped out on step 9, with the error message

“Failed to create sample data. An exception of type Microsoft.SharePoint.SPException was thrown.  Additional exception information: User cannot be found”.

“Curses!” I say, “another trip to logs folder in the 12 hive”. For the nerds, the log is pasted below.

[SPManager] [INFO] [10/21/2009 3:38:47 PM]: Finished upgrading SPContentDatabase Name=ShareWebDb Parent=SPDatabaseServiceInstance Name=Microsoft##SSEE.
[SPManager] [DEBUG] [10/21/2009 3:38:47 PM]: Using cached [SPContentDatabase Name=ShareWebDb Parent=SPDatabaseServiceInstance Name=Microsoft##SSEE] NeedsUpgrade value: False.
[SharedResourceProviderSequence] [DEBUG] [10/21/2009 3:38:47 PM]: Unable to locate SearchDatabase. Exception thrown was: System.Data.SqlClient.SqlException: Cannot open database "SharedServices_DB_ed3872ca-06b1-44c5-8ede-5a81b52265f9" requested by the login. The login failed.
Login failed for user ‘NT AUTHORITY\NETWORK SERVICE’.
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)

The long and short of this error takes a little while to explain. First we need to explain the historical difference between SQL Server Express edition and SQL Server Embedded edition (also known as the Windows internal database). From wikipedia:

SQL Server 2005 Embedded Edition (SSEE): SQL Server 2005 Embedded Edition is a specially configured named instance of the SQL Server Express database engine which can be accessed only by certain Windows Services.

SQL Server Express Edition: SQL Server Express Edition is a scaled down, free edition of SQL Server, which includes the core database engine. While there are no limitations on the number of databases or users supported, it is limited to using one processor, 1 GB memory and 4 GB database files.

Why does this matter? Well Microsoft, being the wise chaps that they are, decided that when you perform a SharePoint installation using the “basic” option, different editions of SharePoint use different editions of SQL Server! Mark Walsh explains it here:

• When you use the "Basic" install option during MOSS 2007 installation it will install and use SQL Server 2005 Express Edition and you have a 4GB database size limit.
• When you use the "Basic" install option during WSS 3.0 installation it DOES NOT use SQL Express, it uses SQL Server 2005 Embedded edition and it DOES NOT have a 4GB size limit.

It happens that Small Business Server 2008 comes with WSS3 preinstalled. Annoyingly, but unsurprisingly, the Small Business Server team opted to use the BASIC installation mode. As described above, SQL Server Embedded Edition (known on Win2008 as the Windows Internal Database) is used. For reference, WSUS on Small Business Server 2008 also uses this database instance.

BUT BUT BUT…

Search Server 2008 Express, uses SQL Server Express edition when performing a basic install. As a result, an additional SQL Server Express instance (SERVERNAME\OFFICESERVERS) gets installed onto the Small Business 2008 server. Then, to make matters worse, the installer gets mixed up and installs some Search Server express databases into the new instance (a Shared Service Provider), but then uses the SQL Embedded Edition instance to install other databases (like the searchDB). Then later during the configuration wizard, it cannot find the databases that it needs because it searches the wrong instance!

The net result is the error shown in the log above. I tried all sorts of things like copying the Express databases into the embedded edition, but I couldn’t disentangle this dependency issue. Some parts of SharePoint (the search server express bits no doubt) looked in the SQL Express instance and the WSS bits looked in the Embedded SQL instance. Eventually, conscious of time, I proceeded to uninstall Search Server Express.

Cleverworkarounds stress rating: Some swear words now uttered

Uninstalling Search Server Express was attempted and tells me that it has successfully completed and wants a reboot. Unfortunately SharePoint is now even more hosed than it was before and I tried a few things to get things back on track (psconfig to create a new farm and the like). After more frustration, and conscious of time I decided to uninstall WSS3 altogether and then reinstall it according to the SBS Repair guide for WSS3.

This had the effect of stuffing up WSUS as well! (I assume because it shares the same Windows Internal Database instance), and after a couple of hours of trying all sorts of increasingly hacky ways of getting all of this working, I was forced to give up.

Note: Whatever you do, do not attempt this method. At one point I tried to trick WSS3 into temporarily thinking it was not a basic mode install so get Search Server Express to prompt for Advanced mode, but it made things worse because the configuration database got confused.

Cleverworkarounds stress rating: Installing SharePoint in basic mode is committing a crime against humanity.

Attempt 2

At this point I hear Doctor Smith abusing me like the poor old robot. “You bubble-headed booby, you ludicrous lump, adlepated amateur, dottering dunderhead”.

Since WSUS got completely screwed, as well as the Windows Internal Database through relentless uninstallation and repair attempts, I started to get nervous. Small Business Server 2008 is a very fussy beast. Essentially it can get very upset at seemingly benign changes. I felt that I had messed about so much that I could no longer guarantee the integrity of this server so I wiped clean and reinstalled.

I installed Windows as per the previous method, and once again the install wizard stopped, asking me for a network driver. Once again I popped in the HP SmartStart CD and proceeded to run the driver install program from the SBS configuration wizard.

This time, no network card was detected!

What the? During attempt 1, I happily installed the necessary HP drivers using the same %$^#% SmartStart CD! Why is it not detecting now??

Cleverworkarounds stress rating: Now thinking about how much fun everyone is having in Vegas while I am fighting this server

After some teeth gnashing, the cause of this problem hit me as I was driving home from the site. I had upgraded the firmware of the network card during attempt 1, as well as the storage controller and system BIOS. I realised that the stupid, brain-dead HP network card drivers likely could no longer recognise its own network card with a newer firmware.

The next day I came back refreshed, and found that indeed, there were newer network drivers at the HP site. I downloaded them and extracted them and sure enough, suddenly the network card was found and I was back in business. How dumb is that! Surely if you are going to write a driver, at the very least make it recognise the hardware irrespective of firmware!  

Once I got past this stupid annoyance, it did not take too long for SBS2008 to be installed and ready to go. Remember that WSS3 is installed for you in basic mode, so to change it requires an uninstall of WSS and then to reinstall it in a different mode. But the problem here is SBS2008 and its fussiness about messing with configuration. In going down this path, you risk future service packs and updates breaking because things are not as expected. Additionally you would have to create companyweb manually and it raises the risk of a misconfiguration or mistake along the way.

I logged a call with Microsoft and got a pretty good engineer (hey Ai Wa) and she was able to consistently reproduce all of my issues in the lab, but was unable to work out a supportable fix. In the meantime, I tried to force Search Server 2008 to install into advanced mode using a scripted install, using an article written by my old mate Ben Curry. Alas, I could not bend it to my will and at the time of writing this article, I had to give up on Search Server 2008 with SBS2008 for now.

For what its worth, I know that I can make this work by installing WSS3 differently, but I planned to properly nail this issue in my lab using the out of the box installs and then publish this article, but I didn’t have the sufficient hardware to run Small Business Server 2008. It requires a lot of grunt to run! So I will revisit this issue once my new VM server arrives and post an update.

But there is more…

If the whole Smartstart, network driver, SBS2008 with its dodgy scripted WSS3 install, with Search Server Express dumb installation assumptions were not enough, I hit more dumb things that resulted in showing how ill equipped HP’s support is able to deal with these sorts of issues.

The server, not surprisingly, supplied with a nice hardware RAID configuration and my client opted to buy some additional disk. When the disks arrived and were installed, we found that the RAID controller could see the disks, but we were unable to add the disks to the existing array using HP’s management tools. HP’s “friendly” support was unable to work out the issue and asked me to do things that were never going to work and insulted my intelligence. Eventually I worked out what was going on myself, via HP’s own forums. It turns out that the HP Server requires a write cache module to be able to grow an array. We had one of these installed. Upon further examination by opening up the chassis, we were missing a battery to go with the write cache module. HP were unable to determine the part number that we needed and we ended up working it out ourselves and telling our supplier.

Then HP stuffed up the order and after following up for two weeks, it turned out they accidentally forgot to put the order through to Singapore to get the part. It seemed that once we went outside of the normal supply chain system at HP, it all broke down. After two weeks and numerous calls, they suddenly realised that the part was available in Sydney all along and it was shipped over next day!. The irony was that next day the part from Singapore arrived!

So now we had two batteries.

Grrrr. It shouldn’t be this hard! Why supply a write cache module and not supply the battery! Dumb Dumb Dumb!

Cleverworkarounds stress rating: Really hating HP and Microsoft at this point.

… and piece de resistance!

Okay so we had a few frustrating struggles, but we more or less got there. But here is the absolute showstopper – the final issue that for me, really made me question this IT discipline that I have worked in for twenty years now.

My client rang me on Monday morning to tell me the server was powered off when we arrived on site. This was unusual because the rack was UPS protected and no other devices were off. We ran HP’s diagnostic tools and no faults were reported. The UPS was a fairly recent APC model, and we installed a serial cable to the server and loaded the APC UPS Management software. The software (which looks scarily like what 16 bit apps used to look like in the early 90’s), found the UPS and showed that was all hunky dory.

We decided to perform a battery test using the software. No sooner than I clicked OK, the server powered off with no warning or shutdown. Whoa!!

I put in a call to APC, and was told that the UPS was not compatible with HP’s new fancy shmansy green star rating power supplies. We had to buy a new UPS because of some “sine wave” mumbo jumbo (UPS engineer talk that I really wasn’t interested in). If the UPS switched to battery, this server would think power was dodgy and do a pre-emptive shutdown. The reason that the server was powered off on the Monday morning was that the UPS has a built in self test that runs every 7 days that cannot be disabled!

Cleverworkarounds stress rating: For %%$$ sake!.

Conclusion

Now I don’t know about you, but once a UPS cannot be “compatible” with a server, that’s it – things have gone too far. For crying out loud, a UPS supplies power. How $%#^ hard can that be? How is an organization supposed to get value out of their IT investments with this sort of crap to deal with.

Then add in the added complexity of Blade servers, Citrix, Virtualisation, shared storage, and I truly feel that some sites are sitting on time bombs. The supposed benefits in terms of efficiency, resiliency and scalability that these technologies bring, come at an often intangible and insidious cost – sheer risk from incredible complexity. If you look at this case study of a small organisation putting in a basic server, most of the issues I have encountered are the side effects of this complexity and the lack of ability for the vendors to be able to help with it.

As the global financial crisis has aptly demonstrated, when things are complex and no-one person can understand everything, when something bad does happen, it tends to do it in a spectacularly painful and costly way.

Finally, before you reply to this likely immature rant and tell me I am a whiner, remember this all you Vegas people. You got to have fun and marvel at all the new (complex) SP2010 toys, while I sat on the other side of the planet in a small computer room, all bitter and twisted, sprouting obscenities to HP, Microsoft and APC dealing with this crap. When you put that into perspective, I think this article is quite balanced!

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

I now find myself saying stuff to my kids that my parents said to me, I think today’s music is crap, I have taken a liking to drinking quality scotch. Essentially all I need now to complete the metamorphosis to being my father is for all my hair to fall out!

So when I write an article whining about an assertion that IT has a credibility issue and has gone backwards in its ability to cope with various challenges, I fear that I have now officially become my parents. I’ll sound like grandpa who always tells you that life was so much simpler back in the 1940′s.

Consequences of complexity…

Before I go and dump on IT as a discipline, how about we dump on finance as a discipline, just so you can be assured that my cynicism extends far beyond nerds.

I previously wrote about how Sarbanes Oxley legislation was designed to, yet ultimately failed to, provide assurance to investors and regulators that public companies had adequate controls over their financial risk. As I write this, we are in the midst of a once in a generation-or-two credit crisis where some seven hundred billion dollars ($700,000,000,000) of US taxpayers’ money will be used to take ownership of crap assets (foreclosed or unpaid mortgages).

Part of the problem with the credit crisis was through the use of "collateralized debt obligations". This is a fancy, yet complex, way of taking a bunch of mortgages, and turning them into an "asset" that someone else who has some spare cash invests in. If you are wondering why the hell someone would invest in such a thing, then consider people with home loans, supposedly happily paying interest on those mortgages. It is that interest that finds its way to the holder (investor) of the CDO. So a CDO is supposedly an income stream.

Now if that explanation makes your eyes glaze over then I have bad news for you: that’s supposed to be the easy part. The reality is that the CDO’s are actually extremely complex things. They can be backed by residential property, commercial property, something called mortgage backed securities, corporate loans – essentially anything that someone is paying interest on can find its way into a CDO that someone else buys into, to get the income stream from the interest paid.

To provide "assurance" that these CDO’s are "safe", ratings agencies give them a mark that investors rely upon when making their investment. So a "AAA" CDO is supposed to have been given the tick of approval by experts in debt instrument style finance.

Here’s the rub about rating agencies. Below is a news article from earlier in the year with some great quotes

http://www.nytimes.com/2008/03/23/business/23how.html?pagewanted=print

Credit rating agencies, paid by banks to grade some of the new products, slapped high ratings on many of them, despite having only a loose familiarity with the quality of the assets behind these instruments.

Even the people running Wall Street firms didn’t really understand what they were buying and selling, says Byron Wien, a 40-year veteran of the stock market who is now the chief investment strategist of Pequot Capital, a hedge fund. “These are ordinary folks who know a spreadsheet, but they are not steeped in the sophistication of these kind of models,” Mr. Wien says. “You put a lot of equations in front of them with little Greek letters on their sides, and they won’t know what they’re looking at.”

Mr. Blinder, the former Fed vice chairman, holds a doctorate in economics from M.I.T. but says he has only a “modest understanding” of complex derivatives. “I know the basic understanding of how they work,” he said, “but if you presented me with one and asked me to put a market value on it, I’d be guessing.”

What do we see here? How many people really *understand* what’s going on underneath the complexity?

Of course, we now know that many of the mortgages backing these CDO’s were made to people with poor credit history, or with a high risk of not being able to pay the loans back. Jack up the interest rate or the cost of living and people foreclose or do not pay the mortgage. When that happens en masse, we have a glut of houses for sale, forcing down prices, lowering the value of the assets, eliminating the "income stream" that CDO investors relied upon, making them pretty much worthless.

My point is that the complexity of the CDO’s were such that even a guy with a doctorate in economics only had a ‘modest understanding’ of them. Holy crap! If he doesn’t understand it then who the hell does?

Thus, the current financial crisis is a great case study in the relationship between complexity and risk.

Consequences of complexity (IT version)…

One thing about doing what I do, is that you spent a lot of time on-site. You get to see the IT infrastructure  and development at many levels. But more importantly, you also spend a lot of time talking to IT staff and organisation stakeholders with a very wide range of skills and experience. Finally and most important of all, you get to see first hand organisational maturity at work.

My conclusion? IT is completely f$%#ed up across all disciplines and many will have their own mini equivalent of the US $700 billion dollar haemorrhage. Not only that, it is far worse today than it previously was – and getting worse! IT staff are struggling with ever accelerating complexity and the "disconnect" between IT and the business is getting worse as well. To many businesses, the IT department has a credibility problem, but to IT the feeling is completely mutual

You can find a nice thread about this topic on slashdot. My personal favourite quote from that thread is this one

Let me just say, after 26 years in this business, of hearing this every year, the systems just keep getting more complex and harder to maintain, rather than less and easier.

Windows NT was supposed to make it so anyone who could use Windows could manage a server.

How many MILLION MSCEs do we have in the world now?

Storage systems with Petabytes of data are complex things. Cloud computing is a complex thing. Supercomputing clusters are complex things. World-spanning networks are complex things.

No offense intended, but the only people who think things are getting easier are people who don’t know how they work in the first place

Also there is this…

There are more software tools, programming languages, databases, report writers, operating systems, networking protocols, etc than ever before. And all these tools have a lot more features than they used to. It’s getting increasingly harder to know "some" of them well. Gone are the days when just knowing DOS, UNIX, MVS, VMS, and OS/400 would basically give you knowledge of 90% of the hardware running. Or knowing just Assembly/C/Cobol/C++ would allow you to read and maintain most of the source code being used. So I would argue that the need for IT staff is going to continue to increase.

I think the "disconnect" between IT and Business has a lot more to do with the fact that business "knows" they depend on IT, but they are frustrated that IT can’t seem to deliver what they want when they want it. On the other side, IT has to deal with more and more tools and IT staff has to learn more and more skills. And to increase frustration in IT, business users frequently don’t deliver clear requirements or they "change" their mind in the middle of projects….

So it seems that I am not alone

I mentioned previously that more often than not, SQL Server is poorly maintained – I see it all the time. Yet today I was speaking to a colleague who is a storage (SAN) and VMware virtualisation god. I asked him what the average VMware setup was like and his answer was similar to my SQL Server and SharePoint experience. In his experience, most of them were sub-optimally configured, poorly maintained, poorly documented and he could not provide any assurance as to the stability of the platform.

These sorts of quality assurance issues are rampant in application development too. I see the same thing most definitely in the security realm too.

As the above quote sates, "it’s increasingly harder to know *some* of them well". These days I am working with specialists who live and breathe their particular discipline (such as storage, virtualisation, security or comms). Those disciplines over time grow more complex and sub-disciplines appear.

Pity then, the poor developer/sysadmin/IT manager who is trying to keep a handle on all of this and try to provide a decent service to their organisation!

Okay, so what? IT has always been complex – I sound like a Gartner cliche. What’s this got to do with SharePoint?

Consequences of SharePoint complexity…

SharePoint, for a number of reasons, is one of those products that has a way of really laying bare any gaps that the organisation has in terms of their overall maturity around technology and strategy.

Why?

Because it is so freakin’ complex! That complexity transcends IT disciplines and goes right to the heart organisational/people issues as well.

It’s bad enough getting nerds to agree on something, let alone organisation-wide stakeholders!

Put simply, if you do a half-arsed job of putting SharePoint in, you will be punished in so many ways! The simple fact is that the odds are against you before you even start because it only takes a mistake in one particular part of the complex layers of hardware, systems, training, methodology, information architecture and governance, to devalue the whole project.

When I first started out, I was helping organizations get SharePoint installed. However lately I am visiting a lot of sites where SharePoint has already been installed, but it has not been a success. There are various reasons;I have cited them in detail in the project failure series, so I won’t rehash all that here. (I’d suggest reading parts three, four and five in particular).

I am firmly of the conclusion that much of SharePoint is more art than science, and what’s more, the organisation has to be ready to come with you. Due to differing learning styles and poor communication of strategy, this is actually pretty rare. Unfortunately, IT are not the people who are well suited to "getting the organisation ready for SharePoint."

If that wasn’t enough, then there is this question. If IT already struggle to manage the underlying infrastructure and systems that underpin SharePoint, then how can you have any assurance that IT will have a "governance epiphany" and start doing things the right way?

This translates to risk, people! I will be writing all about risk in a similar style to the CFO Return on Investment series very soon. I am very interested in methods to quantify the risk brought about by the complexity of SharePoint and the IT services it relies on. For me, I see a massive parallel from the complexity factor in the current financial crisis and I think that a lot can be learned from it. SOX was supposed to provide assurance and yet did nothing to prevent the current crisis. Therefore, SOX represents a great example of mis-focused governance where a lot of effort can be put in for no tangible gain.

A quick test of "assurance"…

Governance is like learning to play the guitar. It takes practice, and it does not give up its secrets easily and despite good intent, you will be crap at it for a while. It is easy to talk about, but putting it into practice is another thing.

Just remember this. The whole point of the exercise is to provide *assurance* to stakeholders. When you set any rule, policy, procedure, standard (or similar), just ask yourself: Does this provide me the assurance I need that gives me confidence to vouch for the service I am providing? Just because you may be adopting ITIL principles, does *not* mean that you are necessarily providing the right sort assurance that is required.

I’ll leave you with a somewhat biased, yet relatively easy litmus test that you can use to test your current level of assurance.

It might be simplistic, but if you are currently scared to apply a service pack to SharePoint, then you might have an assurance issue.

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

Microsoft bashing is a favourite pastime of many a nerd. Whether it is justified or not in many cases is debatable since M$ will never please everyone. But the point is, it is cathartic and in actual fact, good therapy because venting your frustrations at Bill Gates is much healthier than at your colleagues or family.

To my Microsoft employee friends reading this. Don’t feel all defensive – some of the very best Microsoft bashing I have ever heard comes from you guys anyway

So although sometimes the M$ bashing is completely unjustified, long shall it continue to preserve the sanity of IT professionals around the globe.

Having said that, on occasion you will hit some Microsoft induced pain that is legitimately and frustratingly dumb. By "legitimately", I mean that you cannot say "although in hindsight it was dumb, I can actually understand why they decided to do that". Instead you get caught out and experience pain and frustration simply because of a silly Microsoft oversight.

In this case, the oversight is with the SharePoint Configuration Wizard

Domain Controllers and SharePoint

I was unfortunate enough to inherit a MOSS 2007 install that had been performed on a Domain Controller and the farm was running with Domain Admin privileges. (Now this alone is not Microsoft bashworthy, as it is an administrator decision to make). Although you install SharePoint onto a domain controller, most people will tell you that it is not a best practice. I’d go as far as to say it is a bloody terrible idea unless you a really small organisation.

One of the main reasons for this is because a Domain Controller has no local security groups. There is no real concept of "local admin" access. Instead, if you are an administrator of the domain controller, by definition you are a Domain Administrator. If you are granted privileges on the domain controller, you are granted those privileges for the entire domain.

Why is this a problem? A few reasons actually, but for the purposes of this article, we only have to talk about one.

When installed onto a member server (i.e not a Domain Controller), SharePoint creates a bunch of local security groups. This is not unusual at all, as SQL Server does exactly the same. But on a domain controller, there are no local security groups – just domain groups. Thus, the SharePoint installer figures out that it is running on a Domain Controller and changes the behaviour of the creation of those groups.

Normally on a member server, a SharePoint Installation would create three local groups as shown below

In a domain controller situation, these local groups are not created because there is no concept of local security accounts. Accounts get created in Active Directory instead.

I mentioned earlier that SharePoint was running as domain administrator as well as installed onto a DC (talk about a double whammy!). My main motivation was to change this back to use low privileged accounts. I had created a new account to run the SharePoint farm, granted it db_creator and securityadmin privileges on the SQL Server, and added the account to the farm administrators group in SharePoint. However when I tried to change the application pool identity from the domain administrator to this account, I received the dreaded "Cannot connect to configuration database" error.

A quick scan of the logs showed that the farm account was still missing some required permissions on the server.

887x    Unexpected    Can not access configuration database registry key: System.Security.SecurityException: Requested registry access is not allowed.     at Microsoft.Win32.RegistryKey.OpenSubKey(String name, Boolean writable)     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.get_RegistryConnectionString()  The Zone of the assembly that failed was:  MyComputer     
88bl    Monitorable    An exception occured while trying to acquire the local farm: System.Security.SecurityException: Requested registry access is not allowed.     at Microsoft.Win32.RegistryKey.OpenSubKey(String name, Boolean writable)     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.get_RegistryConnectionString()     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.get_Local()     at Microsoft.SharePoint.Administration.SPFarm.FindLocal(SPFarm& farm, Boolean& isJoined)  The Zone of the assembly that failed was:  MyComputer     
887x    Unexpected    Can not access configuration database registry key: System.Security.SecurityException: Requested registry access is not allowed.     at Microsoft.Win32.RegistryKey.OpenSubKey(String name, Boolean writable)     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.get_RegistryConnectionString()  The Zone of the assembly that failed was:  MyComputer     
88bl    Monitorable    An exception occured while trying to acquire the local farm: System.Security.SecurityException: Requested registry access is not allowed.     at Microsoft.Win32.RegistryKey.OpenSubKey(String name, Boolean writable)     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.get_RegistryConnectionString()     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.get_Local()     at Microsoft.SharePoint.Administration.SPFarm.FindLocal(SPFarm& farm, Boolean& isJoined)  The Zone of the assembly that failed was:  MyComputer     
8yr8    Exception    An unhandled exception has occurred. Access to the port is denied. System.UnauthorizedAccessException: Access to the port is denied.     at System.IO.Ports.InternalResources.WinIOError(Int32 errorCode, String str)     at System.Threading.Semaphore..ctor(Int32 initialCount, Int32 maximumCount, String name)     at Microsoft.SharePoint.Publishing.BlobCache.GetCacheTokenThreadStart()     at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)     at System.Threading.ThreadHelper.ThreadStart()    

"Screw this", I said to myself at this point, "I am better off *demoting* the DC back to a member server, and rerunning the SharePoint Configuration Wizard so it can reapply all of the necessary permissions required".

Demoted DC – Not so wizardry…

So in this site, there was actually no reason for the SharePoint server to be a domain controller. So I decided to demote the server back to a member server, and then rerun the configuration wizard.

The domain demotion worked fine, a reboot later and I was back as a member server.

I reran the SharePoint Configuration Wizard, choosing *not* to disconnect from the farm and to continue using this server as the Central Administration server. Pretty quickly the whole thing died at step 3 with the following error in the logs.

09/15/2008 09:20:35  9  INF                          The securityGroup value is WSS_WPG
09/15/2008 09:20:35  9  ERR                          Task secureresources has failed with an unknown exception
09/15/2008 09:20:35  9  ERR                          Exception: System.Security.Principal.IdentityNotMappedException: Some or all identity references could not be translated.
   at System.Security.Principal.NTAccount.Translate(IdentityReferenceCollection sourceAccounts, Type targetType, Boolean forceSuccess)
   at System.Security.Principal.NTAccount.Translate(Type targetType)
   at System.Security.AccessControl.CommonObjectSecurity.ModifyAccess(AccessControlModification modification, AccessRule rule, Boolean& modified)
   at System.Security.AccessControl.CommonObjectSecurity.AddAccessRule(AccessRule rule)
   at Microsoft.SharePoint.PostSetupConfiguration.ResourceAccess.SetRegistryAccessRule()
   at Microsoft.SharePoint.PostSetupConfiguration.ResourceAccess.Secure()
   at Microsoft.SharePoint.PostSetupConfiguration.SecurityTask.SecureResources()
   at Microsoft.SharePoint.PostSetupConfiguration.SecurityTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

After some pondering, I realised the problem. The first line of the above log gives a big hint. Something went awry with the group WSS_WPG.

Of course! We just demoted a domain controller, back to a member server. Because SharePoint was installed when the machine was a domain controller the local group WSS_WPG does not exist. So I created this group and reran the configuration wizard. Sure enough we got further (but it still died as I expected it to).

09/15/2008 09:31:18  9  INF                          The securityGroup value is WSS_ADMIN_WPG
09/15/2008 09:31:18  9  ERR                          Task secureresources has failed with an unknown exception
09/15/2008 09:31:18  9  ERR                          Exception: System.Security.Principal.IdentityNotMappedException: Some or all identity references could not be translated.

Aha, WSS_ADMIN_WPG group also needs to be created. So I created this local group also and ran the wizard for a third time. This time we made it all the way to step 6 before another fatal error. This time the message and code is different to the first two.

09/15/2008 09:34:22  9  ERR                          Task adminvs has failed with an unknown exception
09/15/2008 09:34:22  9  ERR                          Exception: System.InvalidOperationException: 2220
   at Microsoft.SharePoint.Win32.SPNetApi32.NetLocalGroupSetMember(String groupName, String userName)
   at Microsoft.SharePoint.Administration.SPProvisioningAssistant.SetGroupMember(String username, Int32 group)
   at Microsoft.SharePoint.Administration.SPProvisioningAssistant.ProvisionProcessIdentity(String strUserName, SecureString secStrPassword, IdentityType identityType, Boolean isAdminProcess, Boolean isWindowsService, String strServiceName, Boolean dontRestartService)
   at Microsoft.SharePoint.Administration.SPProcessIdentity.ProvisionInternal(SecureString sstrPassword, Boolean isRunningInTimer)
   at Microsoft.SharePoint.Administration.SPApplicationPool.ProvisionInternal(SecureString sstrPassword)
   at Microsoft.SharePoint.Administration.SPApplicationPool.Provision()
   at Microsoft.SharePoint.Administration.SPWebServiceInstance.Provision()
   at Microsoft.SharePoint.PostSetupConfiguration.CentralAdministrationSiteTask.ProvisionAdminVs()
   at Microsoft.SharePoint.PostSetupConfiguration.CentralAdministrationSiteTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

This error proved harder to crack, because in all of their wisdom, the engineers behind this part of the Configuration Wizard neglected to mention the name of the group that was causing this problem with the NetLocalGroupSetMember function. I had to look at another SharePoint install to see what was missing.

As it happened, it was the WSS_RESTRICTED_WPG group. Once I manually added this local group to the server, the Configuration Wizard successfully completed.

And it was good… Not!

Although the configuration wizard completed successfully, my troubles were not over. I created a new AD account to run the SharePoint farm. This account had no domain level privileges apart from being a regular domain user account.

On the SQL Server (thankfully a separate box) I added this account as a login, and granted it the dbcreator and securityadmin server roles (minimum privileges required for a farm account).

Back on the newly demoted member server, I added this account to the WSS_ADMIN_WPG and WSS_RESTRICTED_WPG groups. I then added this account to the "Farm Administrators" group inside SharePoint Central Administration. (Operations->Update Farm Administrators group)

Finally, I used the STSADM command "updatefarmcredentials" to change the farm account from the domain administrator to my newly created farm account.

C:\stsadm -o updatefarmcredentials -userlogin MYDOMAIN\MOSS.Farm -password MyPassWord

I then browsed to the SharePoint Central Administration site.

GONG! Erorrs! A quick check of the eventlogs and what do we see? Seems the new account does not have enough rights to run SharePoint. Dammit! Despite the configuration wizard (and in particular step 3) applying various permissions to the server, it seems like we still have some work to do.

The first log was event ID 5021.

Event ID:      5021
Description:
The identity of application pool SharePoint Central Administration v3 is invalid. The user name or password that is specified for the identity may be incorrect, or the user may not have batch logon rights. If the identity is not corrected, the application pool will be disabled when the application pool receives its first request.  If batch logon rights are causing the problem, the identity in the IIS configuration store must be changed after rights have been granted before Windows Process Activation Service (WAS) can retry the logon. If the identity remains invalid after the first request for the application pool is processed, the application pool will be disabled. The data field contains the error number.

The above message talked about batch logon rights. I guess because I chose not to disconnect from the server farm when I re-ran the configuration wizard, I wasn’t asked to confirm the farm account to use. Therefore the wizard made no attempt to grant the farm account the local security policy right to log on as a batch job.

The next event log entry was 1309.

Event ID:      1309
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 15/09/2008 1:56:54 PM
Event time (UTC): 15/09/2008 5:56:54 AM
Event ID: 07afc239d8df4a13b10a19c3e026e979
Event sequence: 2
Event occurrence: 1
Event detail code: 0
Application information:
    Application domain: /LM/W3SVC/1926590645/ROOT-1-128659318128622880
    Trust level: WSS_Minimal
    Application Virtual Path: /
    Application Path: C:\inetpub\wwwroot\wss\VirtualDirectories\38138\
    Machine name: SHAREPOINT
Process information:
    Process ID: 3456
    Process name: w3wp.exe
    Account name: MYDOMIN\MOSS.farm
Exception information:
    Exception type: COMException
    Exception message: Either a required impersonation level was not provided, or the provided impersonation level is invalid. (Exception from HRESULT: 0×80070542)

The exception message here talks about impersonation. The most logical local security policy right is "Impersonate a client after authentication". So I granted the farm account this right and the above exception went away.

In the end I compared the local security policy rights with a known good SharePoint install and noted that the farm account had the following local security policy rights.

• Log on as a batch job
• Log on as a service
• Impersonate a client after authentication

Alas I was still not done! It turned out that some file-system and registry permissions were also required. Since the WSS_ADMIN_WPG and WSS_WPG groups did not exist when SharePoint was first installed, no permission changes were made to the 12 hive, the inetpub folder, or the registry. So I had to manually put these permissions back in, utilising a known good server as a reference.

I ended up modifying permissions for WSS_ADMIN_WPG and WS_WPG in two locations on the server.

• Permissions on C:\inetpub\wwwroot
• Permissions on 12 Hive (C:\program files\common files\microsoft shared\web server extensions\12)

I also had to modify permissions to one particular registry key for WSS_ADMIN_WPG

• HKLM\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\Secure\configdb

The key in particular that we needed access to was DSN. This hey holds the location of the farm configuration database.

Finally, after much teeth gnashing and grumbling about Bill Gates, we are working!

Let the bashing begin…

Running the SharePoint configuration wizard is possibly the most important activities that you will do as an administrator of a SharePoint farm. It is not a once off thing either. Service Packs, Infrastructure updates, hotfixes and the like all require the wizard to be re-run on every server in the farm.

Therefore, this thing has to be rock solid and not trip up. If you are half way through a service pack and this thing burps on something silly and trivial, then you are likely in for some serious pain.

My problems stemmed from the fact that the wizard *assumes* that the local WSS_* groups already exist and that certain permissions are already in place. In the scenario that I encountered, it is clear that this assumption does not always hold true. Therefore the wizard should detect this and rectify the situation, rather than bomb out with a runtime error and a stacktrace!

I have manually gotten things working again but what assurance do I have that I have caught everything? The annoying thing is that this is such a trivial thing to check for and fix. If the required groups so not exist, then create them! Sheesh!

Furthermore, how about showing some meaningful information in the logs? At least the first two log entries showed me the name of the group that I was having a problem with. Not so on the third error. You would think that as a developer that if you are going to make a call to a function called "NetLocalGroupSetmember", that someone debugging would like to know the name of the group that you are trying to use??? I had to make an educated guess as to the cause of that one. The first two had the friendly message "The securityGroup value is <groupname>" so I had a fighting chance. You can sooo tell when different developers have worked on different bits! Gotta love consistency!

Furthermore, the configuration wizard should check that the farm account has the appropriate local privileges on the server. In my case, despite using the STSADM -o  updatefarmcredentials, it seems that the local security rights needed by the new farm account wasn’t set and nor were registry and file level permissions. Why not? It would not be hard to make the check and rectify if things are not set up as expected!

Conclusion

If anybody says to me "Well why not just reinstall SharePoint and reconnect the content databases"? On the surface of it, it seems like a fairly logical thing to do. After all, reinstalling the product to a new farm would ask you for a farm account, create the necessary local groups and set the various correct permissions.

The answer is that the SharePoint database schema changes with hotfixes and service packs. A uninstall and reinstall would only work if you knew the exact patchlevel of the SharePoint that you uninstalled and patched the reinstallation to the same level. Failure to do this and you will hit the dreaded "“Your backup is from a different version of Windows SharePoint Services and cannot be restored to a server running the current version.” message. If you hit that then you have the equally fun task of trying to determine exactly which combination of Service Pack and/or Hotfix is required.

Think that through for a second and then consider this rhetorical question… Do you think that an administrator who installs MOSS 2007 onto a Domain controller as Domain Admin is likely to keep configuration management notes of all of the patches or service packs applied to SharePoint?

Need I answer?

thanks for reading

Paul Culmsee

                      

No Tags

I am working on an essay about IT complexity at the moment, and one thing that sprung into my mind while thinking about this, is the fact that many of my clients seem to think I am some sort of SQL Server guru.

There are two sad realities inferred by this.

Firstly, I am far from a SQL Server god. Yes I have experience with it, but the only reason people think I’m any good at it stems from the general lack of knowledge that they have about the product. Often all I have to do is waltz in with my Michael Buble like smooth charm and recommend a maintenance plan be set up and I am instantly the guy to talk to in all things SQL.

The truth of the matter is that I’m not fit to lick the boots of a skilled DBA. But like all former system administrators/infrastructure managers who have had to deal with the pressure and consequences of downtime, irrespective of the product, I developed a reflex to learn what I need to to cover my butt. Often I didn’t care less how a product operated from the end-user perspective. All I cared about was how it hung together so I could recover it when it inevitably failed in some way.

The second sad reality, is that a large percentage of sites that I visit, have poorly configured SQL Server boxes on a number of levels. This isn’t necessarily the fault of the overworked support staff as SQL Server is just one of the many IT devices and services that they have to maintain and they are simply too stretched to give SQL the sort of attention it needs.

Sad reality = Risk to the organisation

SQL Server underpins many products in the Microsoft ecosystem and often many 3rd party products ship with the free edition of SQL Server as a back-end data store. I suspect that this results in many administrators simply leaving SQL Server to "do its thing" and focus higher up (or lower down) the application stack. (Developers often make the same mistake too and write poorly optimised SQL code).

The thing that sucks about being a system administrator or infrastructure person is that when things are going well, you are essentially invisible to the success of the organisational machine. Then when something goes down, people scream at you to get it back again. Many system administrators are also perpetually busy, and therefore it is common to see corners cut in terms of overall quality around infrastructure management.

The reality for most organisations is that SQL Server is likely to be one of your most critical components its infrastructure even before SharePoint is added to the mix. It likely underpins various line of business tools, and a prolonged outage is likely to hurt. But to paint with a *really* generalised brush, users still spend a lot of their time working with stuff on the traditional file server and email. So if a line of business application is down due to a dead SQL box, one could argue that your user base is still somewhat productive if file/print and email still works.

But as soon as you add SharePoint to the SQL equation (and especially if you use SharePoint in a document central collaboration scenario), then SQL becomes a real focal-point. Now if SQL goes down, you are down to basic email because you have lost a core component that provides your file services, as well as your line of business applications. If you view this from a productivity standpoint, then the cost of downtime is now significantly higher as a result.

Be Afraid…

In my travels, I have consulted to and work with quite a variety of organisations, large and small, across most vertical markets and in various regional (and worldwide) locations. Despite the variety of organisations, I see the same sort of stuff over and over again. In fact, seeing a well set-up and managed SQL Server is really the exception rather than the norm.

Examples of the problems that I commonly see around SQL are:

Administrator/sa privilege abuse

This issue comes in two forms. The first one is when all of the SQL Service accounts run using an account with "local administrator", or worse, "Domain Administrator" privilege.

The second form is when applications access databases using the ‘sa’ account or a local/domain administrative account.

Both of these drive me nuts because it either reflects a lack of security awareness or a poor security posture. Unsurprisingly perhaps, when I find this setting at client sites, there is a very good chance that the SharePoint service accounts (and many other services) also run with far too much privilege. Perhaps the effects of SQL Slammer has been long forgotten eh?

In any event, the risk profile for this sort of setup for me jumps significantly, not because I believe that the SQL box is about to be hacked, but because it tells me so much more about the organisations awareness and acumen around the value of their information assets. Adding SharePoint to this mix? Sheesh! Why not attend a Michael Bolton concert instead? Both will hurt you, but the latter you can recover from.

No maintenance plans (or ancient ones)

…and people wonder why it’s slow.

This is a very serious issue, and I would have to say that more than half of the sites that I visit do not have any SQL Server maintenance plans.

If you are responsible for maintaining a SharePoint or SQL infrastructure and do not know what a maintenance plan is, then in all seriousness, I urge you to read up on them and check the example screengrab below. As I mention in this article, maintenance plans are the SQL equivalent of your very own personal fitness trainer who makes sure that you go to the gym, do your push-ups and maintain a healthy lifestyle. You use maintenance plans to perform integrity checks of your databases, optimise their performance, run backups and various other tasks. A SQL Server without a maintenance plan is like the band Queen without Freddie Mercury. Sure it will keep going, but it will provide a sub-optimal user experience.

The other big problem area is where there actually *are* maintenance plans, but they were set up in an age before time and have gathered dust ever since. They typically do not reflect current reality and more recent databases are not included in it. Often, although defined, they do not run because service accounts have changed, or the SQL Agent is stopped. This scenario is arguably worse that no maintenance plan at all, as it can lead to a false sense of security.

General "Baggage"

Another huge issue that I see all to often is "baggage". Often I am called in to manage a collaborative SharePoint implementation and the "SQL box" underpinning this critical service is in a mess from years of neglect and the client expects to drop SharePoint onto it. Typical examples? How about when SQL Server has a large list of databases, that nobody can identify, due to poor naming and a lack of change control/configuration management. Often I will see databases that were ‘temporarily’ created for a restore operation – only the restore was in 2006 and the database has sat there ever since.

Another telltale sign of baggage is to count the number of SQL jobs defined on the server. If there are none at all, then you know its a potentially suspect install. But if there are dozens of them in an error state with ambiguous names that make it difficult to work out what they actually do, then you know it is totally a suspect install! So often do I see SQL jobs listed that have long failed with no-one noticing because operators and alerts have not been set up (which I will get to next).

Another common occurrence I see in relation to failed database jobs is because they are set to run using the credentials of *end-users*, often developers. The very fact that jobs are defined using the personal account of a user implies that the user has some degree of privileged access to SQL Server. Of course, these same people will change their passwords 6 weeks later and then wonder why their account gets locked out from time to time because the job they set-up is long forgotten yet trying to execute in vain on its schedule. 

One has to ask, would you trust your SharePoint service (and potentially your business) to a server in such a state? SharePoint database tend to be pretty darn big compared to most databases and if it is clear the SQL box has been poorly managed until now, what sort of assurance is there that it will be managed any better going forward?

No alerting or decent logging

SQL is perfectly capable of providing very detailed auditing, logging and alerting of problems and exception type conditions. But in my experience, it is very rare to see Database Mail and SQL Operators/Alerts set up.

At the very least, if a maintenance plan job fails (such as a database integrity check), you would think it would be prudent to be informed about it no?

So if you have never set up operators and alerts, then do yourself a favour and make sure that any alert with a severity of 16 and up is sent to an operator as this indicates a backup failure. In fact, any alerts from severity 16 and up should ideally be alerted to an operator as the general rule is the higher the severity, the nastier the error.

For a good write-up of the steps to enable email notification, operators and alerts, I recommend Jeremy Kadlec’s article.

Never tested SQL backup agent

Okay I will admit it. I don’t trust backup agents. At various points in my dim, dark past I have been burnt by backup agents behaving badly. I still use them and test them out of course. However, past experience has taught me to always adhere to the principle of giving myself as many options as possible in the event of a disaster recovery situation.

Unfortunately, it is not hard to find backup/recovery horror stories where backups failures have gone unnoticed for months, only to be found the hard way (i.e when a catastrophic failure hits and a restore is needed). As recently as last month I was called in to a catastrophic restore situation where the much touted (and expensive) enterprise backup software failed to restore a SharePoint web application via its native (and untested) SharePoint agent. The backup software failed with a cryptic code and an even more cryptic log file. Fortunately for all concerned, I had previously set up a maintenance plan at this site and restored it the old fashioned simple way.

Moral? Assume nothing! Always test your backup to make sure that a restore works and that the process is documented.

Be your own SQL god!

There are other areas to look at too, but I have listed enough for you to get the idea. So remember, if you want to be considered a "SQL god" and be cool, then heed my advice. Take a good look at your SQL box. Drop in a maintenance plan or two and just like me, people will think you are some sort of genius

Thanks for reading

Paul Culmsee

CISSP, MCSE, "SQL God", MCT

                      

No Tags

I guess the entire SharePoint world now is aware of the post SP1 "Infrastructure updates" put out by Microsoft recently. Probably the best thing about them are that the flaky "content deployment" feature has had some serious work done on it. (My advice has always been use with extreme caution or avoid it, but now I will have to reassess).

Anyway, that is not what I am writing about. Being a former IT Security consultant, I have always installed SharePoint in a "least privilege" configuration. I use different service accounts for search, farm, SSP, reporting services and web applications. The farm account in particular, is one that needs to be very carefully managed given its control over the heart of SharePoint – the configuration database.

Specifically, the farm account never should have any significant rights on any SharePoint farm server, over and above what it minimally needs (which is some SQL Server rights and some DCOM permission stuff). For what it’s worth, I do not use the Network Service account either, as it is actually more risky than a low privileged domain or local user account.

Developers on the other hand tend to run their boxes as full admin. I have no problem with this, so long as there is a QA or test server that is running least privilege.

However, as always with tightening the screws, there are some potential side effects in doing this. There are occasions when the farm account actually needs to perform a task that requires higher privileges, over and above what it has by default. Upgrading SharePoint from a standard to enterprise license is one such example, and it seems that performing the infrastructure update may be another.

If you take the time to read the installation instructions for the infrastructure update, there is this gem of advice:

To ensure that you have the correct permissions to install the software update and run the SharePoint Products and Technologies Configuration Wizard, we recommend that you add the account for the SharePoint Central Administration v3 application pool identity to the Administrators group on each of the local Web servers and application servers and then log on by using that account

You may wonder why this even matters? After all, it is exceedingly likely that you logged into the server as an administrator or domain administrator anyway. Surely you have all the permission that you need, right?

The answer is that not all update tasks are run by your logged-in account. Although you start the installation using your account, at a certain point during the install, many tasks are performed by the SharePoint Central Administration application. Thus, despite you having administrative permissions, SharePoint (using the farm account) will not have.

So the advice above essentially says, temporarily add the SharePoint farm account to the local administrators group and then log into the server as that user account. Now perform the action as instructed. That way the account that starts the installation is the same account that runs SharePoint and thus we know that we are using the correct account with the correct server privileges.

Once the installation has been completed, you can log out of the farm account and then revoke its administrator access, and we are back to the original locked down configuration.

So keep this in mind when performing farm tasks that are likely to require some privileges at the operating system level. It is a good habit to get into (provided you remember to lock down permissions afterwards ).

Cheers

Paul

                      

No Tags

It is the typical scenario isn’t it. Site works fine for a week and then is officially launched on a Monday morning and the site breaks after an hour complaining that it cannot connect to the configuration database. Whoa?

The SQL Server was checked and confirmed to be running fine, and in checking the SharePoint web front end server I I noticed was MSSEARCH.EXE was chewing memory at a rapid rate of knots. Checking the event logs showed up a steady sequence of event ID 7888, 10036 and 3355 messages. Later I noticed that the search crawl was stuck on "Crawling".

If you search on this topic, many people recommend recreating your Shared Service Provider. In this case, this is unnecessary (not to mention drastic).

It turned out to be an unfortunate combination of factors.

• The client was using SQL Server 2005 with SP1
• The client had a SQL Server maintenance plan with a "rebuild index" task.

Non SQL reader? Maintenance plans are a "good thing". Think of it as a fitness regime for your SQL Server. You can regularly perform integrity checks, tasks to optimise performance, run backups and the like. A screenshot of a basic SQL2005 maintenance plan is to the right.

As it turns out, my client ran a maintenance plan each Sunday (hence why this broke on the Monday of go-live). It also turns out that the "rebuild index" maintenance plan task in SQL Server prior to SP2 has a teeny weenie problem. (By "rebuilding" an index, we mean that it is in effect, deleting and recreating it again).

When rebuilt, various options set on the indexes are not recreated the same way as they were originally created. As you may have guessed, this is extremely uncool, since SharePoint set various indexes in a certain way, it expects things to remain consistent.

There is a KB that you can read all about it here.

http://support.microsoft.com/kb/930887/en-us

Also, another great blog post about the issue is here:

http://blogs.vertigo.com/personal/michael/Blog/Lists/Posts/Post.aspx?ID=4

Some of effects of this dodgily recreated index are listed in the KB article, including:

1. In Shared Services, the "indexing status" remains stuck in the "Crawling" state
2. The number of handles that are opened by the MSSearch.exe process increases
3. The number of TCP connections to the computer that is running SQL Server increases
4. Several error are recorded in the SharePoint logfiles.

• "SqlCrawl::ExecuteCommand fails Error 0x80040e2f"
• CGathererQueueManager::FlushQueue failed with recoverable error 0x80040e2f
• CGathererFilterSink::CommitLinks : pGatherAddLink->AddLinkComplete error=0x80040e2f

Additionally, in my case, MSSEARCH.EXE leaked memory very rapidly when viewed in Task Manager and I received the event logs as shown at the end of this post.

To resolve it, follow the instructions at Michael’s blog as the KB article is poorly written, but make sure that you do it for all tables as listed in the KB article! Michael’s post only covers 2 indexes and there actually are more in the search database as shown below.

• Event Type:      Error
• Event Source:   Office SharePoint Server
• Event Category:            Office Server General
• Event ID:          7888
• Date:                1/07/2008
• Time:                12:03:03 PM
• User:                N/A
• Computer:         MyServer

Description:

Message: A connection was successfully established with the server, but then an error occurred during the pre-login handshake.  When connecting to SQL Server 2005, this failure may be caused by the fact that under the default settings SQL Server does not allow remote connections. (provider: Named Pipes Provider, error: 0 – No process is on the other end of the pipe.)

• Event Type:      Error
• Event Source:   Office Server Search
• Event Category:            Gatherer
• Event ID:          10036
• Date:                1/07/2008
• Time:                12:03:38 PM
• User:                N/A
• Computer:         MyServer

Description:

A database error occurred.

Source: Microsoft OLE DB Provider for SQL Server

Code: 17 occurred 1 time(s)

Description: [DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.

• Event Type:      Error
• Event Source:   Windows SharePoint Services 3
• Event Category:            Database
• Event ID:          3355
• Date:                1/07/2008
• Time:                12:04:08 PM
• User:                N/A
• Computer:         MyServer

Description:

Cannot connect to SQL Server. MyServer not found. Additional error information from SQL Server is included below.

[DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.

• Event Type:      Error
• Event Source:   Office SharePoint Server
• Event Category:            Office Server Shared Services
• Event ID:          6482
• Date:                1/07/2008
• Time:                12:05:12 PM
• User:                N/A
• Computer:         MyServer

Description:

Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (009e9001-5c8f-4705-9b4d-ee514c442fc7).

Reason: Exception from HRESULT: 0x80040D1B

Techinal Support Details:

System.Runtime.InteropServices.COMException (0x80040D1B): Exception from HRESULT: 0x80040D1B

   at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.SynchronizeDefaultContentSource(IDictionary applications)

   at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()

   at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

07/01/2008 12:25:01.85 mssearch.exe (0x0E20)                          0x0E38 Search Server Common             GathererSql                               0          Monitorable       CSqlCrawl::ExecuteCommand fails Error 0x80040e2f – File:d:\office\source\search\search\gather\server\gathersql.cxx Line:407 

                      

No Tags