SharePoint development/deployment governance…

Send to Kindle

My home-town compatriot Jeremy Thake has hit onto a governance topic that I think will turn into a very popular series once he is done with it (in committing to writing it, he will be a busy boy indeed for a while I suspect 🙂 ).

He has written a post on current methods and common issues of deployment of SharePoint customisations from dev to staging to prod, with particular reference to the current pain associated with iterative development and deployment.

The state of play in this area out in SharePoint land is not the best, due to a wide variety of factors and Jeremy sums it up very well.

Certainly, there are some extremely annoying architectural quirks. Back when I did the branding series I encountered some, but Jeremy has hit some nastier ones than what I found.

Anyway, take a look at his post – its a good read.

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

Why do SharePoint Projects Fail – Part 6

Send to Kindle

Hi again and welcome to part 6 of my series on the factors of why SharePoint projects fail. Joel Oleson’s write-up a while back gave me 5 minutes of fame, but like any contestant on Big Brother, I’ve had my time in the limelight, been voted out of the house (as in Joel’s front page) and I’m back to being an ordinary citizen again.

If you have followed events thus far, I covered off some wicked problem theory, before delving into the bigger ticket items that contribute to SharePoint project failure. In the last post, we pointed our virtual microscope at the infrastructure aspects that can cause a SharePoint problem to go off the rails.

Now we turn our magnifying glass onto application development issues and therefore application developers. Ah, what fun you can have with application developer stereotyping, eh! A strange breed indeed they are. As a group they have had a significant contribution to the bitter and twisted individual that I am today.

The CleverWorkarounds tequila shot rating is back!

image imageimageimageimageimageimage for a project manager in denial 🙂

imagefor the rest of us!

Continue reading

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

SharePoint External Storage API – Crushing My Dream

Send to Kindle

When I read several months back, that Microsoft was going to supply an API for external storage in MOSS/WSS, I sprang from my desk and danced around the room and babbled incoherently as if I’d been touched by Benny Hinn.

Okay, well maybe I didn’t quite do that. But what I did do was forward the KB article to a colleague whose company is the leading reseller for EMC Documentum in my town. We’d previously had one of those conversations over a few beers where we questioned the wisdom of SharePoint’s potentially unwise, yet unsurprising use of SQL Server as the storage engine.

…so what’s wrong with SQL?

I am going to briefly dump on SQL here for a minute. But first, let me tell you, I actually like SQL Server! Always have. I hated other Office Server products like Exchange until the 2003 version and SharePoint until the 2007 version. But on the whole, I found SQL to be pretty good. So hopefully that will stop the SQL fanboys from flaming me!

Those readers who appreciate capacity planning issues would appreciate the challenges SQL based storage brings to the table.Additionally, those who have used Enterprise Information Management products like Documentum or Hummingbird (now OpenText) would nod as if Microsoft have finally realised the error of their ways with this updated API.

All of the SharePoint goodies like version control, full text indexing and records management come at a price. Disk consumption and performance drain.  Microsoft say to plan for 1.5 times your previous growth in disk usage. In my own real-world results it is more like 2.5 times previous growth. Disk I/O is also increased markedly.

“So what? Disk is cheap”, you reply. Perhaps so, but the disk itself was never the major cost. Given that this is a SQL database we are talking about, a backup of a 100 gigabyte SQL database could take hours and a restore possibly longer. A differential backup of a SQL database would be the entire 100 gig as it is generally one giant database file! So the whole idea of differential backups during the week and full backups on weekends suddenly has to be re-examined. Imagine a disk partition gets corrupted rendering the data useless. In a file server, this might mean 20% of shared files are unavailable while a restore takes place. In a SQL world, you have likely toasted the whole thing and a full restore is required. Often organisations overlook the common issue of existing backup infrastructure not being scalable enough to deal with SQL databases of this size.

“100 gig”, you scoff, don’t be ridiculous”. Sorry but I have news for you. At an application level, there is a scalability issue in that the lowest logical SharePoint object that can have its own database is a Site Collection, not an individual site. For many reasons, it is usually better to use a single site collection where possible. But if one particular SharePoint site has a library with a lot of files, then the entire content database for the site collection is penalised.

Now the above reasons may be the big ticket items that vendors use to sell SAN’s and complex Backup/Storage solutions, but that’s not the real issue.

The real issue (drumroll…)

It may come as a complete shock to you, but documents are not all created equal. No! Really? 🙂 If they were, those crazy cardigan wearing, file-plan obsessed document controllers and records managers wouldn’t exist. But as it happens, different content is handled and treated completely differently, based on its characteristics.

Case in point: Kentucky Fried Chicken would have some interesting governance around the recipe for its 11 herbs and spices, as would the object of Steve Balmer’s chair throwing with their search engine page ranking algorithms.

I picked out those two obvious examples to show an extreme in documents with high intrinsic value to an organisation. The reality is much more mundane. For example, you may be required by law to store all financial records for seven years. In this day and age, invoices can be received electronically, via fax or email. Once processed by accounts payable, invoices largely have little real value.

By using SQL Server, Microsoft is in effect allocating an identical cost of each document in terms of infrastructure cost. Since all documents of a site collection reside inside a SQL content database, you have limited flexibility to shift lower value documents to lower cost storage infrastructure.

How do the big boys do it then?

Documentum as an example stores the content itself in traditional file shares and then stores the name and location of that document (and any additional metadata) in the SQL database. Those of you who have only seen SharePoint may think this is a crazy idea and introduce much more complex disaster recovery issues. But the reality is the opposite.

Consider the sorts of things you can do with this set-up. You can have many file shares on many servers or SAN’s. Documentum for example, would happily allow an administrator to automatically move all documents not accessed in three months to older, slower file storage. It would move the files and then update the file location in SQL so the new location is hidden from the user and they don’t even know it has been moved. Conversely, documents on older, slower storage that have been accessed recently can be moved back to the faster storage automatically.

It also facilitates geographically dispersed organisations having a central SQL repository for the document metadata, but each remote site has a local file store, to make retrieval work at LAN speeds for most documents. This is a much simpler geographically dispersed scenario than SharePoint can ever do right now.

Restores from backup are quite simple. If a file server corrupts, it only affects the documents stored on that file server. Individual file restores are easy to perform and you don’t have to do a major 100gig database restore for a few files.

Furthermore, documents that have a compliance requirement, but do not need to be immediately available, can easily be archived off to read-only media, thus reducing disk space consumption. The metadata detail of the file can still be retrieved from the SQL database, but location information in the SQL database can now refer to a DVD or tape number.

For this reason, it is clear that SharePoint’s architecture has some cost and scalability limitations when it comes to disk usage and management, largely due to SQL Databases and the limitation of Site Collections for content databases.

So how can we move less valuable documents onto less expensive disk hardware? Multiple databases? Possibly, but that requires multiple site collections and this complicates your architecture significantly. (Doing that is the Active Directory equivalent of using separate forests for each of your departments).

Note to SharePoint fanboys: I am well aware that you can ‘sort of’ do some of this stuff via farm design, site design and 3rd party tools. But until you have seen an high end enterprise content management system, there is no contest.

So you might wonder why SharePoint is all the rage then – even for organisations that already have high end ECM systems? Well the short answer is other ECM vendors’ GUIs suck balls and users like SharePoint’s front end better. (And I am not going to provide a long answer 🙂 )

Utopia then?

As I said at the start of this post, I was very happy to hear about Microsoft’s external storage API. In my mind’s eye, I envisaged a system where you create two types of document libraries: ‘standard’ document libraries that use SQL as the store and ‘enhanced’ document libraries that look and feel identical to a regular document library, but it stores the data outside of SQL. Each ‘enhanced document library’ would be able to point to various different file stores, configured from within the properties of the document library itself.

Utopia my butt!

Then a few weeks back some more detail emerged in SDK documentation and my dream was shattered. This really smells like a “just get version 1 out there and we will fix it properly in version 2” release. I know all software companies partake in this sales technique, but it is Microsoft we are talking about here. Therefore it it my god given right…no…my god given privilege to whine about it as much as I see fit.

Essentially this new feature defines an external BLOB store (EBS). The EBS runs parallel to the SQL Server content database, which stores the site’s structured data. You will note that this is pretty much the Documentum method.

In SharePoint, you must implement a COM interface (called the EBS Provider) to keep these two stores in sync. The COM interface recognizes file Save and Open commands and invokes redirection calls to the EBS. The EBS Provider also ensures that the SQL Server content database contains metadata references to their associated BLOB streams in the external BLOB store.

You install and configure the EBS Provider on each Web front end server in your farm. In its current version, external BLOB storage is supported only at the scope of the farm (SPFarm).

Your point being?

If you haven’t realised why I marked the previous sentence in bold, it is this. Since this EBS provider can only be supported at farm scope, then every document library on every site on every site collection in your farm now saves its data via the EBS provider.

So there is utterly nil granularity with this approach. It’s an all or nothing deal. (There goes my utopian dream of two different types of document libraries). All of your documents in the farm are doing to be stored in this EBS provider!

But it gets worse!

The external BLOB storage feature in the present release will not remain syntactically consistent with external BLOB storage technology to be released with the next full-version release of Microsoft Office and Windows SharePoint Services. Such compatibility was not a design goal, so you cannot assume that your implementation using the present version will be compatible with future versions of Microsoft Office or Windows SharePoint Services.

So basically, if you invest time and resources into implementing an EBS provider, then you’re pretty much have to rewrite it all for the next version. (At least you find this out up front).

No utility is available for moving BLOB data from the content database into the external BLOB store. Therefore, when you install and enable the EBS Provider for the first time, you must manually move existing BLOBs that are currently stored in the content database to your external BLOB store.

Okay that makes sense. It is annoying but I can forgive that. Basically if you implement an EBS provider and enable it, your choices for migrating your existing content into it is a backup and restore/overwrite operation or simply wait it out and allow the natural process of file updates do the job for you.

When using an external BLOB store with the EBS Provider, you must re-engineer your backup and restore procedures, as well as your provisions for disaster recovery, because some backup and restore functions in Windows SharePoint Services operate on the content database but not on the external BLOB store. You must handle the external BLOB store separately.

I would have preferred Microsoft to flesh this statement out, as this will potentially cause much grief if people are not aware of this. It implies that STSADM isn’t going to give you the sort of full-fidelity backup that you expect. Yeouch! I feel I might get a few late night call-outs on that one!

Ah, but wait a minute there, sunshine, is that any different to now? STSADM backup and restore is not exactly rock solid now!

Any error conditions, resource drag, or system latency that is introduced by using the EBS Provider, or in the external BLOB store itself, are reflected in the performance of the SharePoint site generally.

Yeah whatever, this is code word for Microsoft’s tech support way of getting out of helping you. “I’m sorry sir, but call your EBS vendor. Thank you come again!”

Conclusion

I can’t say I am surprised at this version 1 implementation, but I am disappointed. If only the granularity extended to a site collection or better still an individual site, I could forgo the requirement to extend it to individual document libraries or content types.

So it will be interesting to see if this API gets any real uptake and if it does, who would actually use it!

later

Paul

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

More SharePoint Branding – Customisation using JavaScript – Part 6

Send to Kindle

God help me, I’m up to part 6 of series about a technology I dislike and still going. For those of you that have just joined us, then you might want to go back to the very beginning of this series where I used JavaScript to improve the SharePoint user experience. Since then, I’ve been trying to pick a path through the thorny maze of what you could term, ‘sustainable customisation’.

By that, I mean something that hopefully will not cause you grief and heartache the next time a service pack is applied!

So no mood for jokes this time – I want to get this over with so let’s get straight to it and finish this thing!

So where are we at?

  • Part 1 looked at how we can use JavaScript to deal with the issue of hiding form elements from the user in lists and document libraries.
  • Part 2 examined some of the issues with the part 1 JavaScript hacks and wrapped it into a web part using the content editor web part.
  • Part 3 then examined the various issues of adding this new web part to certain SharePoint pages (NewForm.aspx, EditForm.aspx and DispForm.aspx). I also covered using SharePoint Audience targeting to make the hiding/unhiding of form elements personalised to particular groups of users.
  • Part 4 started to address a couple of remaining usability issues, and introduced ‘proper’ web-part development using Visual Studio and STSDEV. I created a project to perform the same functionality in part 3, but would not requiring the user to have any JavaScript knowledge or experience.
  • Part 5 then used STSDEV to create a solution package that allowed easy debugging, deployment and updating of the web part developed in part 4.

So what could we possibly have left to cover? Basically this article will revisit the web part code and make some functionality improvements and then I will cover off some remaining quirks/issues that you should be aware of.

[Quick Navigation: Part 1, Part 2, Part 3, Part 4, Part 5 and Part 6]

Continue reading

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

More SharePoint Branding – Customisation using Javascript – Part 5

Send to Kindle

Hello and welcome to part 5 of another epic CleverWorkArounds blog post.

If you think I write a lot on my blog, you should see my documentation and training material! I seem to be rare insofar as I actually like to write documentation and can churn out reasonable quality pretty fast. So if you need your scary SharePoint farm/infrastructure audited and fully documented, you know who to call! 🙂

Anyhow, here is the current state of play.

  • Part 1 of this series looked at how we can use JavaScript to deal with the common request of hiding form elements from the user in lists and document libraries. We looked at a Microsoft documented method, then a better, more flexible method.
  • Part 2 wrapped this JavaScript code into a web part which has been loaded into the SharePoint web part gallery.
  • Part 3 then examined the trials and tribulations of getting this new web part added to certain SharePoint pages (NewForm.aspx, EditForm.aspx and DispForm.aspx), and then with a few simple edits, use this web part to hide form fields as desired. Finally, I demonstrated the power of combining this with SharePoint Audiences targeting functionality to make the hiding/unhiding of form elements personalised to particular groups of users.
  • Part 4 introduced Visual Studio and STSDEV. I created a project to perform the same functionality in part 3, but not requiring any JavaScript knowledge or experience. By the end of part 4 I had a STSDEV project that compiled with no errors.

And now we are onto Part 5 where we turn our attention to the packaging and deployment of our web part. As you are about to see, STSDEV makes this a very quick and painless experience. If you aren’t convinced of the merits of STSDEV and the SharePoint solution framework by the time you finish this article, then I don’t know what will convince you.

[Quick Navigation: Part 1, Part 2, Part 3, Part 4 and Part 6]

Continue reading

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

More SharePoint Branding – Customisation using JavaScript – Part 4

Send to Kindle

Hi there. As I write this post, the media are telling me that the stock market is stuffed, the US economy is going to the dogs and banks are writing down billions from sub-prime excess. I dare not check my online broker, road traffic this morning was abysmal, I was late, brought in the wrong laptop and left an important DVD at home.

Could it get any worse? Who knows, but it sounds like the sort of day to re-visit JavaScript and get frustrated with writing a web part for the first time.

So to recap on our journey thus far..

  • Part 1 of this series looked at how we can use JavaScript to deal with the common request of hiding form elements from the user in lists and document libraries. It looked at a Microsoft documented method, then a better, more flexible method.
  • Part 2 wrapped this JavaScript code into a web part which has been loaded into the SharePoint web part gallery.
  • Part 3 then examined the trials and tribulations of getting this new web part added to certain SharePoint pages (NewForm.aspx, EditForm.aspx and DispForm.aspx), and then with a few simple edits, use this web part to hide form fields as desired. Finally, I demonstrated the power of combining this with SharePoint Audiences targeting functionality to make the hiding/unhiding of form elements personalised to particular groups of users.

All in all a pretty clever workaround so far if I say so myself. 🙂

My original goals for this JavaScript was to find an effective, easily repeatable way to customise SharePoint form pages by hiding fields or form elements when we need to. Specifically:

  • Allow hidden fields based on identity/audience
  • Avoid use of SharePoint Designer
  • Avoid customisations to the form pages that unghosted the pages from the site definition

We achieved these goals in part three, but was I satisfied? No. The quest for more clever workarounds always goes on!

[Quick Navigation: Part 1, Part 2, Part 3, Part 5 and Part 6]

Continue reading

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle