Every so often I get back into geek mode and roll up the sleeves and get my SharePoint hands dirty. Not so long ago I was assisting a small WSS3 based organisation with a major disk subsystem upgrade (iSCSI SAN) and locally attached storage upgrade, driven by content database growth and a need to convert some sites to site collections. By the end of it all, we had a much better set-up with a much better performing disk subsystem, but I was hit by two problems. One was WSS Search being broken and needing a fix and the other was the appallingly slow speed of copying large files around.

So, let’s talk about fixing broken search and then talk about copying large files.

1. When SPSearch breaks…

The SharePoint install in question was a WSS3 site with SP1, and Search Server 2008 had not been installed. The partition that contained the indexes (F:\) had to be moved to another partition (G:), so to achieve this I used the command

stsadm –o spsearch indexlocation G:\DATA\INDEX

Now, 99.9% of the time this will happily do its thing. But today was the 0.01% of the time when it decided to be difficult. Upon executing this command, I received an RPC error. Unfortunately for me, I was out of time, so I decided to completely re-provision search and start all over again.

It didn’t matter whether I tried this in Central Administration->Operations->Services on Server, or via the command line below. Both methods would not work.

stsadm -o spsearch -action stop

On each attempt, search would get stuck on unprovisioning (stopping) with a sequence of events in the event log (eventID 2457, 2429 and 2428).

===========================================================================================

Event Source: Windows SharePoint Services 3 Search
Event Category: Gatherer
Event ID: 2457
Description: 

The plug-in manager <SPSearch.Indexer.1> cannot be initialized.
Context: Application 'Search index file on the search server' 

Details:
The system cannot find the path specified. (0x80070003) 

===========================================================================================
Event Source: Windows SharePoint Services 3 Search
Event Category: Gatherer
Event ID: 2429
Description: 

The plug-in in <SPSearch.Indexer.1> cannot be initialized.
Context: Application '93a1818d-a5ec-40e1-82d2-ffd8081e3b6e', Catalog 'Search' 

Details:
The specified object cannot be found. Specify the name of an existing object. (0x80040d06) 

===========================================================================================

Event Source: Windows SharePoint Services 3 Search
Event Category: Gatherer
Event ID: 2428
Description: 

The gatherer object cannot be initialized.
Context: Application 'Search index file on the search server', Catalog 'Search' 

Details:
The specified object cannot be found. Specify the name of an existing object. (0x80040d06) 

So, as you can see I was stuck. I couldn’t not clear the existing configuration and the search service would never actually stop. In the end, I started to wonder whether the problem was that my failed attempt to change the index partition had perhaps not reapplied permissions to the new location. To be sure I reapplied permissions using the following STSADM command

psconfig -cmd secureresources

This seemed to do the trick. Re-executing the stsadm spsearch stop command finally did not come up with an error and the service was listed as stopped.

Once stopped, we repartitioned the disks accordingly and now all I had to do was start the damn thing

Through the Central Administration GUI I clicked Start and re-entered all of the configuration settings, including service accounts and the new index location (G:\DATA\INDEX). After a short time, I received the ever helpful “Unknown Error” error message.

Rather than change debug settings in web.config, I simply checked the SharePoint logs and the event viewer. Now, I had a new event in the logs.

Event Type: Warning
Event Source: Windows SharePoint Services 3 Search
Event Category: Gatherer
Event ID: 10035
Description:
Could not import the registry hive into the registry because it does not exist in the configuration database.
Context: Application '93a1818d-a5ec-40e1-82d2-ffd8081e3b6e' 

Hmm… It suggests a registry issue, so I checked the registry.

Although the error message really made no sense to me, checking the registry turned out to be the key to solving this mystery. If you look carefully in the above screenshot, note that the registry key DataDirectory was set to “F:\DATA\INDEX”.

I was surprised at this, because I had re-provisioned the SPSearch to use the new location (G:\DATA\INDEX). I would have thought that changing the default index location would alter the value of this key. A delve into the ULS logs showed events like this.

STSADM.EXE (0x0B38) 0×0830 Search Server Common MS Search Administration 95k1 High WSS Search index move: Changing index location from ‘F:\data\index’ to ‘G:\data\index’.

STSADM.EXE (0x0B38) 0×0830 Search Server Common MS Search Administration 95k2 High WSS Search index move: Index location changed to ‘G:\data\index’.

STSADM.EXE (0x0B38) 0×0830 Search Server Common MS Search Administration 0 High CopyIndexFiles: Source directory ‘F:\data\index\93a1818d-a5ec-40e1-82d2-ffd8081e3b6e’ not found for application ’93a1818d-a5ec-40e1-82d2-ffd8081e3b6e’.

STSADM.EXE (0x0F10) 0×1558 Windows SharePoint Services Topology 8xqz Medium Updating SPPersistedObject SPSearchServiceInstance Parent=SPServer Name=DAPERWS03. Version: 218342 Ensure: 0, HashCode: 54267293, Id: 305c06d7-ec6d-465a-98be-1eafe64d8752, Stack: at Microsoft.SharePoint.Administration.SPPersistedObject.Update() at Microsoft.SharePoint.Administration.SPServiceInstance.Update() at Microsoft.SharePoint.Search.Administration.SPSearchServiceInstance.Update() at Microsoft.Search.Administration.CommandLine.ActionParameter.Run(StringBuilder& output) at Microsoft.SharePoint.Search.Administration.CommandLine.SPSearch.Execute() at Microsoft.Search.Administration.CommandLine.CommandBase.Run(String command, StringDictionary keyValues, String& output) at Microsoft.SharePoint.StsAdmin.SPStsAdmin.RunOperation(SPGlobalAdmin globalAdmin, String st…

mssearch.exe (0×1654) 0×1694 Search Server Common IDXPIPlugin 0 Monitorable CTripoliPiMgr::InitializeNew – _CopyNoiseFiles returned 0×80070003 – File:d:\office\source\search\ytrip\search\tripoliplugin\tripolipimgr.cxx Line:519

mssearch.exe (0×1654) 0×1694 Search Server Common Exceptions 0 Monitorable <Exception><HR>0×80070003</HR><eip>0000000001D4127F</eip><module>d:\office\source\search\ytrip\search\tripoliplugin\tripolipimgr.cxx</module><line>520</line></Exception>

Note the second last line above (marked bold and italic). It showed that a function called CopyNoiseFiles returned a code of 0×8007003. This code happens to be “The system cannot find the path specified,” so it appears something is missing.

It then dawned on me. Perhaps the SharePoint installer puts some files into the initially configured index location and despite moving the index to another location, SharePoint still looks to this original location for some necessary files. To test this, I loaded up a blank Windows 2003 VM and installed SharePoint SP1 *without* running the configuration wizard. When I looked in the location of the index files, sure enough – there are some files as shown below.

It turned out that during our disk reconfiguration, the path of F:\DATA\INDEX no longer existed. So I recreated the path specified in the registry (F:\DATA\INDEX) and copied the contents of the CONFIG folder from my fresh VM install. I then started the search service from Central Administration and… bingo! Search finally started successfully…Wohoo!

Now that I had search successfully provisioned, I re-ran the command to change the index location to G:\DATA\INDEX and then started a full crawl.

C:\>stsadm -o spsearch -indexlocation G:\DATA\INDEX

Operation completed successfully.

C:\>stsadm -o spsearch -action fullcrawlstart

Operation completed successfully.

I then checked the event logs and now it seems we are cooking with gas!

Event Type: Information
Event Source: Windows SharePoint Services 3 Search
Event Category: Gatherer
Event ID: 10045
Description: 

Successfully imported the application configuration snapshot into the registry.
Context: Application '93a1818d-a5ec-40e1-82d2-ffd8081e3b6e' 

Event Type: Information
Event Source: Windows SharePoint Services 3 Search
Event Category: Gatherer
Event ID: 10044
Description: 

Successfully stored the application configuration registry snapshot in the database.
Context: Application 'Serve search queries over help content' 

As a final check, I re-examined the registry and noted that the DataDirectory key had not changed to reflect G:\. Bear that in mind when moving your index to another location. The original path may still be referred to in the configuration.

2. There are RAID cards and there are RAID cards

Since part of the upgrade work was to improve disk performance, we had to detach databases and move them around while we upgraded the disk infrastructure and repartitioned existing disk arrays. The server had an on-board Intel RAID controller with two arrays configured. One was a two-disk RAID 0 SCSI and the other was a three-disk RAID 5 SATA array. The performance of the RAID 5 SATA had always been crap – even crappier than you would expect from onboard RAID 5. When I say crap, I am talking around 35 megabytes per second transfer rate – even on the two-disk SCSI RAID 0 array.

Now, 35MB/sec isn’t great, but not completely intolerable. But, what made this much, much worse however, was the extreme slowness in copying large files (ie >4GB). When trying to copy files like this to the array, the throughput dropped to as low as 2MBs.

No matter whether it was Windows Explorer drag and drop or a command line utility like ROBOCOPY, the behaviour was the same. Throughput would be terrific for around 60 seconds, and then it would drop as shown below.

My client called the server vendor and was advised to purchase 4 SCSI disks to replace the SATA’s. Apparently the poor SATA performance was actually because SCSI and SATA were mixed on the same RAID bard and bus. That was a no-no.

Sounded plausible, but of course, after replacing the RAID 5 with the SCSI disks, there was no significant improvement in disk throughput at all. The performance of large files still reflected the pattern illustrated in the screenshot above.

Monitoring disk queue length on the new RAID array showed that disk queues were off the planet in terms of within normal boundaries. Now, I know that some people view disk queue length as a bit of an urban myth in terms of disk performance, but copying the same files to the iSCSI SAN yielded a throughput rate of around 95MBs and the disk queue value rarely spiked above 2.

Damn! My client wasn’t impressed with his well known server vendor! Not only does the onboard RAID card have average to crappy performance to begin with, RAID 5 with large files makes it positively useless.

Fun with buffers

To me, this smelt like a buffer type of issue. Imagine you are pouring sand into a bucket and the bucket has a hole in it. If you pour sand into the bucket at a faster rate than the hole allows sand to pour out, then eventually you will overflow the bucket. I suspected this sort of thing was happening here. The periods of high throughput were when the bucket was empty and the sand filled it fast. Then the bucket filled up and things slowed to a crawl while all of that sand passed through the metaphorical hole in the bottom. Once the bucket emptied, there was another all-too-brief burst of throughput as it filled quickly again.

I soon found a terrific article from EPS Windows Server Performance Team that explain what was going on very clearly.

Most file copy utilities like Robocopy or Xcopy call API functions that try and improve performance by keeping data in a buffer. The idea is that files that are changed or accessed frequently can be pulled from the buffer, thereby improving performance and responsiveness. But there is a trade-off. Adding this cache layer introduces an overhead in creating this buffer in the first place. If you are never going to access to copy this file again, adding it to the buffer is actually a bad idea.

Now imagine a huge file. Not only do you have the buffer overhead, but you now are also filling the buffer (and forcing it to be flushed), over and over again.

With a large file, you are actually better off avoiding the buffer altogether and doing a raw file copy. Any large file on a slow RAID card will still take time, but it’s a heck of a lot quicker than when combined with the buffering overhead.

Raw file copy methods

In the aforementioned article from the Microsoft EPS Server performance team, they suggest ESEUTIL as an alternative method. I hope they don’t mind me quoting them…

For copying files around the network that are very large, my copy utility of choice is ESEUTIL which is one of the database utilities provided with Exchange.  To get ESEUTIL working on a non-Exchange server, you just need to copy the ESEUTIL.EXE and ESE.DLL from your Exchange server to a folder on your client machine.  It’s that easy.  There are x86 & x64 versions of ESEUTIL, so make sure you use the right version for your operating system.  The syntax for ESEUTIL is very simple: eseutil /y <srcfile> /d <destfile>.  Of course, since we’re using command line syntax – we can use ESEUTIL in batch files or scripts.  ESEUTIL is dependent on the Visual C++ Runtime Library which is available as a redistributable package

I found an alternative to this, however, that proved its worth to me. It is called Teracopy and we tried the free edition to see what difference it would make in terms of copy times. As it happens, the difference was significant and the time taken to transfer large files around was reduced by a factor of 5. Teracopy also produced a nice running summary of the copy thoughput in MB/sec.The product definitely proved its worth and at a whopping $15 bucks, is not going to break the bank.

So, if you are doing any sort of large file copies and your underlying disk subsystem is not overly fast, then I recommend taking a look at this product. Believe me, it will save you a heap of time.

3. Test your throughput

A final note about this saga. Anyone who deals with SQL Server will have likely read articles about best practice disk configuration in terms of splitting data/logs/backups to different disk arrays to maximise throughput. This client had done this, but since Teracopy gave us nice throughput stats, we took the opportunity to test the read/write performance of all disk subsystems and it turned out that putting ALL data onto the SAN array had significantly better performance than using any of the onboard arrays.

This meant that the by-the-book configuration was hamstrung by a poorly performing onboard RAID controller and even if the idea of using separate disks/spindles seemed logical, the cold hard facts of direct throughput testing proved otherwise.

After reconfiguring the environment to leverage this fact, the difference in response time of WSS, when performing bulk uploads was immediately noticeable.

The moral of the story?

If you are a smaller organisation and can’t afford the high end server gear like Compaq/IBM, then take the time to run throughput tests before you go to production. The results may surprise you.

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

I now find myself saying stuff to my kids that my parents said to me, I think today’s music is crap, I have taken a liking to drinking quality scotch. Essentially all I need now to complete the metamorphosis to being my father is for all my hair to fall out!

So when I write an article whining about an assertion that IT has a credibility issue and has gone backwards in its ability to cope with various challenges, I fear that I have now officially become my parents. I’ll sound like grandpa who always tells you that life was so much simpler back in the 1940′s.

Consequences of complexity…

Before I go and dump on IT as a discipline, how about we dump on finance as a discipline, just so you can be assured that my cynicism extends far beyond nerds.

I previously wrote about how Sarbanes Oxley legislation was designed to, yet ultimately failed to, provide assurance to investors and regulators that public companies had adequate controls over their financial risk. As I write this, we are in the midst of a once in a generation-or-two credit crisis where some seven hundred billion dollars ($700,000,000,000) of US taxpayers’ money will be used to take ownership of crap assets (foreclosed or unpaid mortgages).

Part of the problem with the credit crisis was through the use of "collateralized debt obligations". This is a fancy, yet complex, way of taking a bunch of mortgages, and turning them into an "asset" that someone else who has some spare cash invests in. If you are wondering why the hell someone would invest in such a thing, then consider people with home loans, supposedly happily paying interest on those mortgages. It is that interest that finds its way to the holder (investor) of the CDO. So a CDO is supposedly an income stream.

Now if that explanation makes your eyes glaze over then I have bad news for you: that’s supposed to be the easy part. The reality is that the CDO’s are actually extremely complex things. They can be backed by residential property, commercial property, something called mortgage backed securities, corporate loans – essentially anything that someone is paying interest on can find its way into a CDO that someone else buys into, to get the income stream from the interest paid.

To provide "assurance" that these CDO’s are "safe", ratings agencies give them a mark that investors rely upon when making their investment. So a "AAA" CDO is supposed to have been given the tick of approval by experts in debt instrument style finance.

Here’s the rub about rating agencies. Below is a news article from earlier in the year with some great quotes

http://www.nytimes.com/2008/03/23/business/23how.html?pagewanted=print

Credit rating agencies, paid by banks to grade some of the new products, slapped high ratings on many of them, despite having only a loose familiarity with the quality of the assets behind these instruments.

Even the people running Wall Street firms didn’t really understand what they were buying and selling, says Byron Wien, a 40-year veteran of the stock market who is now the chief investment strategist of Pequot Capital, a hedge fund. “These are ordinary folks who know a spreadsheet, but they are not steeped in the sophistication of these kind of models,” Mr. Wien says. “You put a lot of equations in front of them with little Greek letters on their sides, and they won’t know what they’re looking at.”

Mr. Blinder, the former Fed vice chairman, holds a doctorate in economics from M.I.T. but says he has only a “modest understanding” of complex derivatives. “I know the basic understanding of how they work,” he said, “but if you presented me with one and asked me to put a market value on it, I’d be guessing.”

What do we see here? How many people really *understand* what’s going on underneath the complexity?

Of course, we now know that many of the mortgages backing these CDO’s were made to people with poor credit history, or with a high risk of not being able to pay the loans back. Jack up the interest rate or the cost of living and people foreclose or do not pay the mortgage. When that happens en masse, we have a glut of houses for sale, forcing down prices, lowering the value of the assets, eliminating the "income stream" that CDO investors relied upon, making them pretty much worthless.

My point is that the complexity of the CDO’s were such that even a guy with a doctorate in economics only had a ‘modest understanding’ of them. Holy crap! If he doesn’t understand it then who the hell does?

Thus, the current financial crisis is a great case study in the relationship between complexity and risk.

Consequences of complexity (IT version)…

One thing about doing what I do, is that you spent a lot of time on-site. You get to see the IT infrastructure  and development at many levels. But more importantly, you also spend a lot of time talking to IT staff and organisation stakeholders with a very wide range of skills and experience. Finally and most important of all, you get to see first hand organisational maturity at work.

My conclusion? IT is completely f$%#ed up across all disciplines and many will have their own mini equivalent of the US $700 billion dollar haemorrhage. Not only that, it is far worse today than it previously was – and getting worse! IT staff are struggling with ever accelerating complexity and the "disconnect" between IT and the business is getting worse as well. To many businesses, the IT department has a credibility problem, but to IT the feeling is completely mutual

You can find a nice thread about this topic on slashdot. My personal favourite quote from that thread is this one

Let me just say, after 26 years in this business, of hearing this every year, the systems just keep getting more complex and harder to maintain, rather than less and easier.

Windows NT was supposed to make it so anyone who could use Windows could manage a server.

How many MILLION MSCEs do we have in the world now?

Storage systems with Petabytes of data are complex things. Cloud computing is a complex thing. Supercomputing clusters are complex things. World-spanning networks are complex things.

No offense intended, but the only people who think things are getting easier are people who don’t know how they work in the first place

Also there is this…

There are more software tools, programming languages, databases, report writers, operating systems, networking protocols, etc than ever before. And all these tools have a lot more features than they used to. It’s getting increasingly harder to know "some" of them well. Gone are the days when just knowing DOS, UNIX, MVS, VMS, and OS/400 would basically give you knowledge of 90% of the hardware running. Or knowing just Assembly/C/Cobol/C++ would allow you to read and maintain most of the source code being used. So I would argue that the need for IT staff is going to continue to increase.

I think the "disconnect" between IT and Business has a lot more to do with the fact that business "knows" they depend on IT, but they are frustrated that IT can’t seem to deliver what they want when they want it. On the other side, IT has to deal with more and more tools and IT staff has to learn more and more skills. And to increase frustration in IT, business users frequently don’t deliver clear requirements or they "change" their mind in the middle of projects….

So it seems that I am not alone

I mentioned previously that more often than not, SQL Server is poorly maintained – I see it all the time. Yet today I was speaking to a colleague who is a storage (SAN) and VMware virtualisation god. I asked him what the average VMware setup was like and his answer was similar to my SQL Server and SharePoint experience. In his experience, most of them were sub-optimally configured, poorly maintained, poorly documented and he could not provide any assurance as to the stability of the platform.

These sorts of quality assurance issues are rampant in application development too. I see the same thing most definitely in the security realm too.

As the above quote sates, "it’s increasingly harder to know *some* of them well". These days I am working with specialists who live and breathe their particular discipline (such as storage, virtualisation, security or comms). Those disciplines over time grow more complex and sub-disciplines appear.

Pity then, the poor developer/sysadmin/IT manager who is trying to keep a handle on all of this and try to provide a decent service to their organisation!

Okay, so what? IT has always been complex – I sound like a Gartner cliche. What’s this got to do with SharePoint?

Consequences of SharePoint complexity…

SharePoint, for a number of reasons, is one of those products that has a way of really laying bare any gaps that the organisation has in terms of their overall maturity around technology and strategy.

Why?

Because it is so freakin’ complex! That complexity transcends IT disciplines and goes right to the heart organisational/people issues as well.

It’s bad enough getting nerds to agree on something, let alone organisation-wide stakeholders!

Put simply, if you do a half-arsed job of putting SharePoint in, you will be punished in so many ways! The simple fact is that the odds are against you before you even start because it only takes a mistake in one particular part of the complex layers of hardware, systems, training, methodology, information architecture and governance, to devalue the whole project.

When I first started out, I was helping organizations get SharePoint installed. However lately I am visiting a lot of sites where SharePoint has already been installed, but it has not been a success. There are various reasons;I have cited them in detail in the project failure series, so I won’t rehash all that here. (I’d suggest reading parts three, four and five in particular).

I am firmly of the conclusion that much of SharePoint is more art than science, and what’s more, the organisation has to be ready to come with you. Due to differing learning styles and poor communication of strategy, this is actually pretty rare. Unfortunately, IT are not the people who are well suited to "getting the organisation ready for SharePoint."

If that wasn’t enough, then there is this question. If IT already struggle to manage the underlying infrastructure and systems that underpin SharePoint, then how can you have any assurance that IT will have a "governance epiphany" and start doing things the right way?

This translates to risk, people! I will be writing all about risk in a similar style to the CFO Return on Investment series very soon. I am very interested in methods to quantify the risk brought about by the complexity of SharePoint and the IT services it relies on. For me, I see a massive parallel from the complexity factor in the current financial crisis and I think that a lot can be learned from it. SOX was supposed to provide assurance and yet did nothing to prevent the current crisis. Therefore, SOX represents a great example of mis-focused governance where a lot of effort can be put in for no tangible gain.

A quick test of "assurance"…

Governance is like learning to play the guitar. It takes practice, and it does not give up its secrets easily and despite good intent, you will be crap at it for a while. It is easy to talk about, but putting it into practice is another thing.

Just remember this. The whole point of the exercise is to provide *assurance* to stakeholders. When you set any rule, policy, procedure, standard (or similar), just ask yourself: Does this provide me the assurance I need that gives me confidence to vouch for the service I am providing? Just because you may be adopting ITIL principles, does *not* mean that you are necessarily providing the right sort assurance that is required.

I’ll leave you with a somewhat biased, yet relatively easy litmus test that you can use to test your current level of assurance.

It might be simplistic, but if you are currently scared to apply a service pack to SharePoint, then you might have an assurance issue.

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

Hi again and welcome to this seemingly endless series of posts on the topic of SharePoint projects gone bad.

We spent a couple of posts looking at problem projects in general before focusing specifically on SharePoint. If you have followed the series closely, you will observe that haven’t talked much on technical aspects of the product yet. If you were expecting me to pick apart annoying aspects of the architecture then unfortunately, you will be disappointed because I really don’t believe that it is a big factor in why SharePoint projects fail. Besides which, 90% of SharePoint blogs are on technical/development content anyway.

So where am I going with part 5 then you ask?  I am indeed delving into technical aspects, but once again it is all about the people involved.

So now its time to take a few cheap-shots at the geeks. (After all, they are sensitive souls and we don’t want them to feel left out do we). For the purposes of this post, infrastructure people, tech support, system administrators can be lumped into the same ‘geek bucket’.

Geeks can also cop it like Project Managers do, when projects take on wicked tendencies. They will implement the agreed requirements, but the stakeholders feel that the end result isn’t what they wanted. In the ensuing fallout that happens when the project sponsor realises that say, half a million bucks has been blown with little to show for it, blame is inevitably directed their way, whether justified or not.

In some ways the geeks have an easier job than others in the project (but don’t tell them that you will bruise their ego’s). The infrastructure geeks simply have to plan, build and configure the SharePoint farm and assimilate it into existing infrastructure.They deal with servers, SAN’s, networks and in general, devices are much more agreeable than other humans . Infrastructure integration can range from easy to hellish, depending mainly on the attitude of management to its importance and the ability of the geek to sell that importance. Sadly, both sides equally suck at this – and it is a reflection of “organisational maturity” which we will get to a little later.  

In Homer Simpson’s imaginary land of chocolate where everything is wonderful (and edible), the infrastructure people and developers take the functional specification and build it without ever having to speak to the users/stakeholders. You see, in this land, the business analysts and stakeholders have done their job and the agreed off requirements are exactly right, thus all of the problems outlined in the first four posts of this series never happened

So to complete this article, we have to spend it in the land of chocolate, because we have to suspend reality and assume that somehow, we have managed to avoid all of the issues outlined in parts 1 to 4.

I realise that this requires a significant suspension of belief, but humour me and just go with it…

Product Complexity – Infrastructure

Let’s face it. SharePoint is big, complex, nasty, over-engineered yet under-engineered, skinnable yet unskinnable and depends on other products and supporting technologies that have big, thick books written just about them (SQL Server, ASP.NET, etc). Such a large number of components means there are equally as many points of failure or disruption, therefore points of governance to cover and experience required to cover it.

The combination of skills required to really become an expert in this product is not easy to develop unless you have a significant amount of real world experience over a period of time. To give you some idea of what is required, Joel Oleson wrote a blog post some time ago on what he considered was the combination of skills required to be a SharePoint Architect. The list was expansive. I’ve listed some below (and cut a heap of them out too)

Ready? Take a deep breath… 

IIS, Windows Server, TCP/IP & networks, SQL Server 2005 Advanced Administration, Firewalls, Proxies, Active Directory, Authentication, Security, IT Infrastructure Design, Hardware, Performance Monitoring, Capacity Planning, Workflow, IE, Firefox, Office Client tools, ASP.NET, HTML, Javascript, AJAX, XSL, XSLT, Exchange/SMTP, Clustering, NLB, SANs, Backup Solutions, Single Sign on, Monitoring & Troubleshooting, Global Deployments, Dev, Test, Staging, Production – Staged deployments, ITIL, Virtualisation.

… and that’s just the core skills apparently! There are many, many other skills listed by Joel – and I could add a few more myself.

The complexity of the infrastructure requirements are very much a product on the size of the organisation, the scope of the project and which SharePoint disciplines are involved. The biggest mistake project teams and infrastructure people make is to underestimate the implications of this infrastructure investment.

I remember sitting in on an early project team meeting for a widely scoped, document centric SharePoint deployment. It was for a company of some 500 staff with an existing 1TB+ file system. I told them that for what they wanted to do, and for the organisation size and utter reliance on its availability, they had better size up and budget for a clustered SQL Server and SAN. I also warned them, that their existing backup infrastructure may not be scaleable enough to accommodate the requirements of SharePoint and they should also investigate this. The look on the face of the CIO should have made my spider senses tingle. It turned out that they were not aware at all that all the documents lived inside SQL Server. Thus the possibility of a high speed, fault tolerant SAN, a review of their backup system and a clustered SQL server never entered their minds.

That small ‘oversight’ ended up costing some $200,000 all up. That came close to doubling the initial budget that was sold to the project sponsor – ouch!

Scope, Infrastructure and Organisational Maturity

This doesn’t really belong in part 5, but I forgot to mention it in part 3 (blush). When I talk of “organisational maturity”, I am not talking about the how crude the office dirty jokes are, but I am talking about maturity in relation to awareness and pragmatic adoption of best practice methodologies. Perhaps another term to use is corporate wisdom?

Anyway, many methodologies, such as COBiT and CMMI, contain a measurement system which allows an organisation to rate themselves against what the standard considers to be ‘mature’. In this way, the organisation has the means to know where they are and where they need to get to.

In this competitive, post SOX world of compliance and governance, this is becoming an increasing part of the corporate and IT landscape. Organisations need to provide assurance to their customers, shareholders and staff that all is well and that their investment in the organisation is safe.

Sarcastic Note: I’m very interested in whether compliance has made any difference in the current subprime mess. Expect more posts on that topic.

What I have noticed is that the ‘less mature’ an organisation is, the greater the propensity for wicked problems to occur, whether SharePoint or any other product. I believe this is because of all of the factors described in the first four parts, but particularly, panacea effect, described in part 3. The panacea effect can easily take root at organisations where the culture is not conducive to organisational maturity. A less mature organisation will find a tool with potential and then try and solve all of their problems at once, without the sort of soul searching, dialogue and raw data analysis that would give them a deeper understanding of the problems they face.

Another inexplicable problem that I have also noticed (but haven’t quantified), is that organisations with a low maturity often find it difficult to properly make use of the very methodologies and frameworks designed to improve maturity. They are well aware that they have problems and want to improve, but always seem to go about it the wrong way and go from complete disorganisation to complete over-the-top process and structure. Typically this does not end up improving productivity or resolving longstanding problems and in extreme cases, can do more harm than good. ISO9001, PMBOK, ITIL and process improvement methodologies like Six Sigma have suffered this fate. It is basically the panacea effect in action again, this time not around a platform like SharePoint, but around interpretation of the intent of the methodology.

Infrastructure and Project Budget

The result of product complexity and low organisational maturity is a vicious circle. The poor old infrastructure architect/engineer is asked to plan, design and size a solution to a system that will “do everything”. The infrastructure person usually have had little visibility or input into scope and the requirements. Therefore, with so many knowns and equally many unknowns, they have little choice but to produce an infrastructure design to accommodate all of these possibilities.

SharePoint’s complexity and aspects of its architecture mean that there can be a significant infrastructure cost for many projects. For example. A stakeholder would have very little idea just how much work the seemingly innocuous request of requiring that the SharePoint solution be extranet accessible, adds to a project. Another example would be that a stakeholder would also not appreciate the implications of mandating that all files in SharePoint are to have a complete version history.

Inevitably, infrastructure is going to be costly. As a general rule, the more ambitious the scope of a project, the more complex it is to plan for a cost effective infrastructure. When faced with an uncertain future, it is always preferable to add another server or two for more redundancy When it is all said and done, the infrastructure person can rarely give you a guarantee that their solution is adequate either. Instead they can rate the solution in how comfortable they feel about it.

Thus, in the low maturity organisation, the project manager and project sponsor will balk at the cost because it blows away their initial, relatively naive, time and budget estimate. This is despite the fact the scope of the project is likely so broad that is has some wicked problem elements in it. The Project Manager may inform the infrastructure architect that the cost is unacceptable and try and pin them down into reducing the cost to save the project, and in bad cases, save face with the project sponsor or senior management.

I used to ask senior management to provide me growth assumptions to try and work through a balance between cost and benefit when this occurred. This method placed accountability for the assumptions in the right place, but often did not solve the infrastructure and budget issues. One notable case I had was an infrastructure design for a private company owned by a single managing director. The director had a serious case of rose-coloured glasses and delusions of grandeur around the growth prospects of his company. His growth figures were wildly optimistic and thus so was the cost that I provided. But at least I was able to argue that the design was sound, based on the assumptions that I had been given.

Conversely when there is a budget issue in a higher maturity organisation, senior management, the project manager and project sponsor would initially be more interested in why there is such a disparity between their budget estimate and the actual cost. From there they would determine the root cause factors that allowed this to happen. I’d love to get into how an organisation gets from low to higher maturity, but that is not the intent of this series of posts, so we’ll have to leave it at that.

Infrastructure and Product Skills

As previously mentioned, SharePoint is complex and the products it relies on are also complex. In the wrong infrastructure/architect hands, this can cause costly problems. Interestingly however, I have noticed that a lack of product skills among architect and infrastructure staff does not necessarily impact the project in terms of the project metrics of time and budget. Nerds are rather adept at hacking away enough to “make it work”, and thus they have a talent for winging it. Instead the cost will be quality, but that cost will not be immediately apparent.

The geek contribution to project failure is not as overt. Instead, problems caused by cutting corners tend to manifest themselves post implementation, where bad decisions, poor governance and dodgy workarounds all have a habit of coming home to roost in some nasty, high profile and costly way. By then however the project team can deny all knowledge . I really should do a blog post about infrastructure complexity separately to this series for the above reason. Since the risks around infrastructure complexity manifest themselves usually after the project has been delivered and is into its operating phase.

That said, In the project planning process however, it is common that several infrastructure oriented tasks are under estimated due to a lack of appreciation of what is involved. Some notable ones are:

• Capacity planning
• Impact on existing infrastructure
• Information Architecture
• Governance (procedures, standards, guidelines, change control, configuration management, etc

In the final section of Part 5, we will touch on each of these areas

Infrastructure – Capacity Planning

This is a particularly complex and overlooked area in document oriented collaboration projects. I specifically modelled a mythical project budget estimate in my return on investment series on this exact project type. Being document centric, it is the scenario that puts the most pressure on your infrastructure in terms of disk IO performance and growth due to the added overhead of version control, full text indexing, backup and recycle bin requirements. It is also impacts (and is impacted by) the information architecture and logical architecture design of SharePoint because of the architectural limitations of the product.

Microsoft’s top support call reasons were listed by Joel some time back and many of them are basic issues addressed via a capacity planning exercise.

• Are you monitoring all your disks?  If you did a simple/basic all in one install is it all on one drive that you are watching?  Disk space available on WFE and SQL, common support issue is Basic install running out of drive space for SQL
• If large file support is enable are the myriad of configuration settings to make this work configured, http://support.microsoft.com/?id=925083
• Scan the SharePoint farm and report if any of our capacity limits are exceeded, http://technet2.microsoft.com/WindowsServer/WSS/en/library/2aa12954-2ea7-475c-9dce-663f543820811033.mspx
• Check if /3GB switch is enabled in boot.ini, WSS does not support /3GB switch
  http://support.microsoft.com/kb/933560

Impact on existing infrastructure

Closely related to capacity planning is the impact that SharePoint has on existing infrastructure. As I mentioned in one of the examples above, the client gave utterly no thought to how adequate their existing backup system was for accommodating the database oriented nature of SharePoint. As it turned out, it was completely inadequate as it was, before the complexity of SharePoint was added to the mix. To rectify the problem required a new backup system that cost in the region of $50,000.

Extranet requirements are also typically unappreciated too. It is easy to ‘extend’ a SharePoint virtual server to be accessible via the extranet, utilising say, SQL authentication. But there are firewalls, reverse proxy as well as logical architecture to consider. Do you really want say, your internal documents potentially accessible via an external facing web server? No? Then you may require separate content databases, web applications and site collections to improve the “chinese wall” between internal and external oriented content. Ah, but that has its own implications on design, governance, custom development costs and information architecture.

Information Architecture

Information architecture is a large topic and I cannot do it justice here. Suffice to say, that we are talking about concepts such as design decisions around the set-up of web applications, site collections, sites, sub sites, site columns, site content types, libraries, lists, workflows, forms, navigation, user profiles and search to name a few.

Prior to SharePoint, many users will have known nothing else but messy shared file systems where the only means of classification was folders and file naming conventions (that no-one ever followed anyway). Now we have many other tools at our disposal which in theory is a good thing.

But as I said, many people know nothing but folders. This has been the predominant file classification mechanism for more than 30 years. Trying to “un-learn” more than 30 years of operating fundamentally the same way is not something that will come naturally. Thinking in terms of columns, views and content types requires a deeper understanding of the divisions, disciplines, compliance and vertical market of an organisation.

Information Architecture is one of those phases in a project that should come with a safety warning. “Please kids, do not try this at home”

I’d love to write more on information architecture because it cuts to the heart of what I currently spend a lot of time doing, but this post is getting long . So, I’ll simply say this: Information architecture is a very, very difficult area to get right and it can seem like more art than science. (don’t worry, there is science). Even in a good project, the amount of time and effort to develop it is grossly underestimated.

Yep, you heard right, even in a good project.

My absolute worst scenario is for people come at it with a naive notion that you can say, pick 5 columns to classify *all* files in an organisation the same way. This is records management thinking, and if you read this post, and think that I am mistaken and standard classification is the way document management should be, then I say to you in all seriousness, you have picked the wrong product and your definition of document management is different to mine!

Governance

Governance thankfully has been getting much more press in blogs, webcasts, whitepapers and the like in the last half year. A lot of it is still of questionable quality, but at least people are starting to factor it into their SharePoint project planning.

It is all well and good to use the term, but project teams still underestimate exactly what is required for governance. For all of SharePoint’s good points, it does have a relatively high cost of ownership in my opinion and that ownership cost is completely in the area of governance. In other words, I don’t think day to day management of a SharePoint farm is a big task. But if there is not a certain degree of discipline around change management, configuration management, procedures, standards and guidelines to administrators, users, site owners and developers, bad things will happen. Failure to maintain this discipline, means that the ongoing reliability of the system will be slowly but surely eroded.

Now rather that do a big write-up here about governance, I urge you to read Microsoft’s “Governance Checklist” found here. In your project plans, you should use this checklist and ensure you cover off all of the sections applicable to your SharePoint deployment. Additionally, when I wrote my ROI Series, I actually used the aforementioned governance checklist as the methodology to estimate project costs. You may find that post useful as a reference for your own project planning.

Conclusion

It is when you get to this part in the series, that you realise that you missed some important points in the previous posts. That is the nature of blogs I guess, since I post them as I write them. I think by the time this series is finished, I might release the overall content as a much more coherent and comprehensive whitepaper. Please let me know what you think about this idea and whether you’d pay a few bucks for it.

Part 5 was the longest post so far in this series, yet I still firmly believe that these topics are not the overriding reasons why SharePoint projects fail. Some of the topics covered here are simply symptoms of the fallout from a project that was wicked in the first place. So really, I believe that parts 1 to 4 of this series is where you will find most of your answers.

The next post is going to cover application development factors in terms of failed projects, which will be interesting, since I’m not really a developer, so I might miss a few things that are obvious to developers.

I’ll then follow that up with the end user factors.. after that it’s a little grey at this stage

I’m not sure how long it will take to get the next post out either, as the next couple of weeks are busy for me. In any event, if you have found this article or this series of articles useful, please let me know.

Later

Paul

                      

No Tags

Okay, well maybe I didn’t quite do that. But what I did do was forward the KB article to a colleague whose company is the leading reseller for EMC Documentum in my town. We’d previously had one of those conversations over a few beers where we questioned the wisdom of SharePoint’s potentially unwise, yet unsurprising use of SQL Server as the storage engine.

…so what’s wrong with SQL?

I am going to briefly dump on SQL here for a minute. But first, let me tell you, I actually like SQL Server! Always have. I hated other Office Server products like Exchange until the 2003 version and SharePoint until the 2007 version. But on the whole, I found SQL to be pretty good. So hopefully that will stop the SQL fanboys from flaming me!

Those readers who appreciate capacity planning issues would appreciate the challenges SQL based storage brings to the table.Additionally, those who have used Enterprise Information Management products like Documentum or Hummingbird (now OpenText) would nod as if Microsoft have finally realised the error of their ways with this updated API.

All of the SharePoint goodies like version control, full text indexing and records management come at a price. Disk consumption and performance drain.  Microsoft say to plan for 1.5 times your previous growth in disk usage. In my own real-world results it is more like 2.5 times previous growth. Disk I/O is also increased markedly.

“So what? Disk is cheap”, you reply. Perhaps so, but the disk itself was never the major cost. Given that this is a SQL database we are talking about, a backup of a 100 gigabyte SQL database could take hours and a restore possibly longer. A differential backup of a SQL database would be the entire 100 gig as it is generally one giant database file! So the whole idea of differential backups during the week and full backups on weekends suddenly has to be re-examined. Imagine a disk partition gets corrupted rendering the data useless. In a file server, this might mean 20% of shared files are unavailable while a restore takes place. In a SQL world, you have likely toasted the whole thing and a full restore is required. Often organisations overlook the common issue of existing backup infrastructure not being scalable enough to deal with SQL databases of this size.

“100 gig”, you scoff, don’t be ridiculous”. Sorry but I have news for you. At an application level, there is a scalability issue in that the lowest logical SharePoint object that can have its own database is a Site Collection, not an individual site. For many reasons, it is usually better to use a single site collection where possible. But if one particular SharePoint site has a library with a lot of files, then the entire content database for the site collection is penalised.

Now the above reasons may be the big ticket items that vendors use to sell SAN’s and complex Backup/Storage solutions, but that’s not the real issue.

The real issue (drumroll…)

It may come as a complete shock to you, but documents are not all created equal. No! Really? If they were, those crazy cardigan wearing, file-plan obsessed document controllers and records managers wouldn’t exist. But as it happens, different content is handled and treated completely differently, based on its characteristics.

Case in point: Kentucky Fried Chicken would have some interesting governance around the recipe for its 11 herbs and spices, as would the object of Steve Balmer’s chair throwing with their search engine page ranking algorithms.

I picked out those two obvious examples to show an extreme in documents with high intrinsic value to an organisation. The reality is much more mundane. For example, you may be required by law to store all financial records for seven years. In this day and age, invoices can be received electronically, via fax or email. Once processed by accounts payable, invoices largely have little real value.

By using SQL Server, Microsoft is in effect allocating an identical cost of each document in terms of infrastructure cost. Since all documents of a site collection reside inside a SQL content database, you have limited flexibility to shift lower value documents to lower cost storage infrastructure.

How do the big boys do it then?

Documentum as an example stores the content itself in traditional file shares and then stores the name and location of that document (and any additional metadata) in the SQL database. Those of you who have only seen SharePoint may think this is a crazy idea and introduce much more complex disaster recovery issues. But the reality is the opposite.

Consider the sorts of things you can do with this set-up. You can have many file shares on many servers or SAN’s. Documentum for example, would happily allow an administrator to automatically move all documents not accessed in three months to older, slower file storage. It would move the files and then update the file location in SQL so the new location is hidden from the user and they don’t even know it has been moved. Conversely, documents on older, slower storage that have been accessed recently can be moved back to the faster storage automatically.

It also facilitates geographically dispersed organisations having a central SQL repository for the document metadata, but each remote site has a local file store, to make retrieval work at LAN speeds for most documents. This is a much simpler geographically dispersed scenario than SharePoint can ever do right now.

Restores from backup are quite simple. If a file server corrupts, it only affects the documents stored on that file server. Individual file restores are easy to perform and you don’t have to do a major 100gig database restore for a few files.

Furthermore, documents that have a compliance requirement, but do not need to be immediately available, can easily be archived off to read-only media, thus reducing disk space consumption. The metadata detail of the file can still be retrieved from the SQL database, but location information in the SQL database can now refer to a DVD or tape number.

For this reason, it is clear that SharePoint’s architecture has some cost and scalability limitations when it comes to disk usage and management, largely due to SQL Databases and the limitation of Site Collections for content databases.

So how can we move less valuable documents onto less expensive disk hardware? Multiple databases? Possibly, but that requires multiple site collections and this complicates your architecture significantly. (Doing that is the Active Directory equivalent of using separate forests for each of your departments).

Note to SharePoint fanboys: I am well aware that you can ‘sort of’ do some of this stuff via farm design, site design and 3rd party tools. But until you have seen an high end enterprise content management system, there is no contest.

So you might wonder why SharePoint is all the rage then – even for organisations that already have high end ECM systems? Well the short answer is other ECM vendors’ GUIs suck balls and users like SharePoint’s front end better. (And I am not going to provide a long answer )

Utopia then?

As I said at the start of this post, I was very happy to hear about Microsoft’s external storage API. In my mind’s eye, I envisaged a system where you create two types of document libraries: ‘standard’ document libraries that use SQL as the store and ‘enhanced’ document libraries that look and feel identical to a regular document library, but it stores the data outside of SQL. Each ‘enhanced document library’ would be able to point to various different file stores, configured from within the properties of the document library itself.

Utopia my butt!

Then a few weeks back some more detail emerged in SDK documentation and my dream was shattered. This really smells like a “just get version 1 out there and we will fix it properly in version 2″ release. I know all software companies partake in this sales technique, but it is Microsoft we are talking about here. Therefore it it my god given right…no…my god given privilege to whine about it as much as I see fit.

Essentially this new feature defines an external BLOB store (EBS). The EBS runs parallel to the SQL Server content database, which stores the site’s structured data. You will note that this is pretty much the Documentum method.

In SharePoint, you must implement a COM interface (called the EBS Provider) to keep these two stores in sync. The COM interface recognizes file Save and Open commands and invokes redirection calls to the EBS. The EBS Provider also ensures that the SQL Server content database contains metadata references to their associated BLOB streams in the external BLOB store.

You install and configure the EBS Provider on each Web front end server in your farm. In its current version, external BLOB storage is supported only at the scope of the farm (SPFarm).

Your point being?

If you haven’t realised why I marked the previous sentence in bold, it is this. Since this EBS provider can only be supported at farm scope, then every document library on every site on every site collection in your farm now saves its data via the EBS provider.

So there is utterly nil granularity with this approach. It’s an all or nothing deal. (There goes my utopian dream of two different types of document libraries). All of your documents in the farm are doing to be stored in this EBS provider!

But it gets worse!

The external BLOB storage feature in the present release will not remain syntactically consistent with external BLOB storage technology to be released with the next full-version release of Microsoft Office and Windows SharePoint Services. Such compatibility was not a design goal, so you cannot assume that your implementation using the present version will be compatible with future versions of Microsoft Office or Windows SharePoint Services.

So basically, if you invest time and resources into implementing an EBS provider, then you’re pretty much have to rewrite it all for the next version. (At least you find this out up front).

No utility is available for moving BLOB data from the content database into the external BLOB store. Therefore, when you install and enable the EBS Provider for the first time, you must manually move existing BLOBs that are currently stored in the content database to your external BLOB store.

Okay that makes sense. It is annoying but I can forgive that. Basically if you implement an EBS provider and enable it, your choices for migrating your existing content into it is a backup and restore/overwrite operation or simply wait it out and allow the natural process of file updates do the job for you.

When using an external BLOB store with the EBS Provider, you must re-engineer your backup and restore procedures, as well as your provisions for disaster recovery, because some backup and restore functions in Windows SharePoint Services operate on the content database but not on the external BLOB store. You must handle the external BLOB store separately.

I would have preferred Microsoft to flesh this statement out, as this will potentially cause much grief if people are not aware of this. It implies that STSADM isn’t going to give you the sort of full-fidelity backup that you expect. Yeouch! I feel I might get a few late night call-outs on that one!

Ah, but wait a minute there, sunshine, is that any different to now? STSADM backup and restore is not exactly rock solid now!

Any error conditions, resource drag, or system latency that is introduced by using the EBS Provider, or in the external BLOB store itself, are reflected in the performance of the SharePoint site generally.

Yeah whatever, this is code word for Microsoft’s tech support way of getting out of helping you. “I’m sorry sir, but call your EBS vendor. Thank you come again!”

Conclusion

I can’t say I am surprised at this version 1 implementation, but I am disappointed. If only the granularity extended to a site collection or better still an individual site, I could forgo the requirement to extend it to individual document libraries or content types.

So it will be interesting to see if this API gets any real uptake and if it does, who would actually use it!

later

Paul

                      

No Tags

For what its worth, I don’t do this sort of work anymore.. its been a long time since geeky low level stuff pushed my buttons. But hey, you never know – maybe someone may find this of some use..

So here we go. This post simply outlines the process of upgrading an IBM blade server to the latest greatest (at the time of writing) drivers, firmware and stuff. The operating system was installed by IBM’s fancy schamzy ServerGuide CD’s. But from my experience with IBM in particular, by the time you rip open the packaging, the CD is out of date and there are much newer versions on IBM’s site. Given that we are talking about disk subsystems, BIOS, drivers and that sort of stuff here, you should ideally ensure you always do a comprehensive check of all components of your server and system infrastructure and make sure you have the latest.

Upgrading this stuff before the server is in production is less risky and allows you to capture really useful information that helps for next time.

Example HBA and storage subsystem driver install for WINDOWS Cluster on IBM BladeServers

1. Qlogic San Surfer management agent

Run the QLogic SANsurfer install program on the blade server and click on the custom button and then click on the Next button:

Deselect everthing except SANsurfer FC Windows Agent (we do not need all the other crap) and click on the Next button:

Click on the Install button:

Click on the Done button:

2. Update the Qlogic drivers

So now that we have the SanSurfer management agent installed, we can use the SanSurfer Administration utility to connect to this server and check things out. You can collect some useful stats from this utility which I will blog about some other time. (I used it to corroborate disk I/O performance as reported by my SAN as well as Windows physicaldisk performance counters)

Anyway I digress. So I am assuming here that you have installed the SanSurfer management components (the stuff you skipped in step 1) onto management PC. So using SANSurfer Administration utility, Connect to the IP of the server running the agent and then click on each HBA/Port listed.

Click UTILITIES tab and click Update Driver and choose “from the Qlogic Website”

If there is a new driver, then you will see a message like the one below

You will be asked to enter a password.. (you can find the default password in the documentation that comes with the blade if you have never set it)

Now the update has completed..

3. Option ROM and Firmware

Note the OS driver was not the only thing upgradable via this utility. It is also possible to upgrade firmware and BIOS for the controller itself

So I went snooping at the IBM website and came across the is the IBM OEM Support page

http://support.qlogic.com/support/oem_detail_all.asp?oemid=224

Listed on this page is:

4Gb Expansion Card ROM Image for HS & LS blades 1.38 4Gb Expansion Card BIOS 1.24, Fcode 1.25, Firmware 4.00.24.

Examination of the current BIOS using SanSurfer showed an older BIOS.

BIOS is 1.04, FW 4.00.23, fcode 1.08

Downloaded ibm_qmix472cf1.38risc4.00.24.zip from the above site and saved it to my PC.

Now back in the SANSurfer GUI, choose UPDATE OPTION ROM

Choose the downloaded zip file

enter password

Reboot the server and check the option rom Information. Well what do you know.. now it matches the version on the IBM OEM site.

4. Install IBM Storage Manager host agent for failover

This site has a nice new IBM SAN, and to achieve redundancy within the SAN, you have to install an IBM component that sets up the HBA’s on the blade for failover. This software came with the SAN and once again I checked IBM’s web site for the latest version.

Here, I used Storage Manager v9.19 for Windows 64-bit\Windows

Choose HOST from the list here.

Choose the RDAC Driver as recommended. This is holding my entire farm and I’ll stick to the recommendation

Restart and hey presto! You will see in a later post my test plan for stress testing this SAN once set up as I test this failover by pulling cables

                      

No Tags