Why SharePoint training sometimes doesn’t deliver (and what to do about it)

Send to Kindle

image

I was surprised to see the recent SharePoint Fatigue Syndrome post got some traction in the interweb. As it happened, that particular post was kicking around in an unfinished state for months. The thing is, its not the only “home truth” type of post that I have sitting in my “drafts” folder. I also have one on the state of the SharePoint training market. Given that I have a training announcement to make, I thought that I would combine them.

A day in the life…

We recently worked on a SharePoint upgrade project, where the previous developers did an excellent job overall. That is…if you judge them on the SharePoint governance metrics of writing clean and maintainable code, packaging it up properly, not hacking away at system files and actually writing documentation.

Unfortunately, although they did an excellent job through that lens, the actual solution, when judged on whether users found that it made their life easier, it was an epic fail. Users hated it with passion and like many solution that users hate, the system was soon relegated to being a little-used legacy platform where the maintenance costs now outweighed the benefits. The organisation had invested a couple-hundred thousand dollars on this solution and saw very little value for that money. Accordingly, they took their business elsewhere…to us. After a workshop, the client had one of those inverse “aha” moments when they realised that if they had taken a little more time to understand SharePoint, the custom solution would have never been developed in the first place.

This sort of example, to me, highlights where SharePoint governance goes so wrong. The care and diligence the developers exercised was necessary, but clearly not sufficient. No matter what the quality of the code, the unit testing regime and its packaging, at the end of the day a blueberry pie was baked and the client wanted an apple pie. The problem was not in the ingredients or the baking. The problem was that by the time they delivered the pie, it was clear that the wrong recipe was used. In the above case, the developer had omitted a whole raft of critical considerations in creating the solution – none of which were covered in developer training.

Necessary but not sufficient…

image

When you think about it, the current approach to SharePoint training seems not to be about recipes, but all about ingredients. Trainees get shipped off to “boot camps” for an indoctrination of all of the ingredients in the cupboard (and SharePoint is a bloody big cupboard!). SharePoint features and components are examined in individual detail, usually with an accompanying exercise or lab to demonstrate competency in that particular component. Graduates then return with a huge list of ingredients, but still no skills in how to develop the right recipes.

What exacerbates this problem is that training is siloed across disciplines. As an example: An “IT Pro” bootcamp will go into meticulous detail about performance, scalability and design aspects. Any considerations around development, information architecture and user engagement are seen through the lens of the infrastructure nerd. (Ah – who am I kidding… user engagement in an IT pro bootcamp has never happened. Smile)

Now consider for a second, how we design SharePoint sites. These days, it is common for people to actively discourage designing SharePoint solutions based on organisational departmental boundaries. (By organisational departmental boundaries I mean Marketing, HR, IT etc.) Why is this design approach frowned upon? Proponents claim that it tends to perpetuate  the problem of information silos and doesn’t stand the test of time, given that organisations tend to restructure just when your information architecture masterpiece is ready for prime time. In fact, the research organisation Jackob Neilsen did a study and found that task based structures (characterised by “My…” and “I need to…”) endured better than organisational based structures. Quoting from them:

In our study, task-based structures often endured better than intranets organized departmentally. In our user testing of intranets, we’ve also found that task-based navigation tends to facilitate ease-of-learning. Thus, the benefits for IA durability are just one more argument in favor of adopting a task-based structure for your intranet.

So what I find ironically funny is the second sentence of the Jackob Neilsen quote: “Ease-of-learning.” I wonder what sort of learning they are talking about? Presumably something other than delivering a failed solution with some really nice programming governance behind it! Yet the way SharePoint training is designed and marketed actually compartmentalises SharePoint training into similar silos. The result? Students get a rose coloured view of the SharePoint world, based on their discipline. This is because, as Ackoff brilliantly put it,  “complexity is in the eye of the beholder – the other persons job always looks simple”.

By the way, what I am highlighting is not the fault of the trainers because at the end of the day, they respond to what they think the market wants. Sadly, what the market thinks it wants is often not what it needs.

I feel that the missing link – and most critical aspect of SharePoint training for practitioners – is not about how many ingredients you know, but how you go about creating those recipes. Yet SharePoint training overly focuses on what each ingredient does in isolation – whether a job discipline or a particular component. Whilst I fully accept that knowing the ingredients is a necessity, it is clearly not sufficient. This is an airbrushed version of reality, without due consideration of how ingredients combine in unique scenarios. Accordingly, this training does nothing to teach how to achieve shared understanding between practitioner and the eventual users who have to live with the legacy of what is delivered.

When you think about it, shared understanding is what makes or breaks SharePoint success because it is the pre-requisite to shared commitment to a solution. As demonstrated by the example of great code underpinning a crap solution, lack of shared understanding and commitment will always trump any other good work performed.

What to do about it…

SharePoint is a product that often requires adaptive change on the part of users. Learning the capabilities of the product is one thing – changing entrenched collaborative practice is another altogether. In case you haven’t noticed before, users tend not to be charmed by new, shiny features if they cannot see how it will make their jobs easier. (Nerdy knowledge workers like you and me easily get seduced by shiny things but our world view is seriously skewed compared to those who live on the coal face of organisations). Thus, the skills required to facilitate change and align various roles, require a different type of training course: one that integrates rather than compartmentalises. One that teaches how to synthesise the whole, rather than reductionise into the parts.

For such a course, no virtual machines are needed because there are no labs to demonstrate competence in some SharePoint component that will be out of date by SharePoint vNext. Instead, such a course needs to focus on the concepts, patterns and practices that are typically not seen in the IT practitioners toolkit (and for that matter, not seen in many complex mainstream IT/PM methodologies). The added bonus for such a course is that the skills and learning’s it provides are applicable beyond SharePoint and even beyond IT itself. While a typical SharePoint might give you mileage for the current version, a course like what I describe will give you tools that you can use anywhere, irrespective of the technology and project.

Does such a class exist? (Is that the longest post you have ever read to get to such a rhetorical question? Smile )

Of course it exists – I’ve been running it around the world for a couple of years now. It’s called the SharePoint Governance and Information Architecture Class (#SPGovIA) and it was a year in the making and comes with lots of goodies, such as a CD with a sample performance framework, governance plan, SharePoint ROI calculator (spreadsheet) and sample mind maps of Information Architecture. The class was originally designed for Microsoft New Zealand, on behalf of 3Grow for the Elite program that used to certify gold partners for serious SharePoint competence. Since then its been run in the UK, Netherlands, US, Australia and New Zealand. Next month I will run classes in Singapore and Hong Kong.

For my US readers, early next year I will be taking the course on the road, specifically Canada and the USA in Feb 2012. This course is not run often, because for me the US is a damn long way to travel and my time is tight these days! So I sincerely hope that if this sort of class sounds interesting to you, then you will consider being part of it. Michal Pisarek has already made an announcement for classes in Vancouver, and more details will be forthcoming for one or two US cities. I only have time for 2 classes in North America, so which city should it be?

For more detail on the class, head on over to www.spgovia.com. While there, click the Media link and watch the first half hour of the class. I look forward to seeing you there.

image

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

Consequences of complexity–the evilness of the SharePoint 2010 User Profile Service

Send to Kindle

Hiya

A few months back I posted a relatively well behaved rant over the ridiculously complex User Profile Service Application of SharePoint 2010. I think this component in particular epitomises SharePoint 2010’s awful combination of “design by committee” clunkiness, along with real-world sheltered Microsoft product manager groupthink which seems to rate success on the number of half baked features packed in, as opposed to how well those features install logically, integrate with other products and function properly in real-world scenarios.

Now truth be told, until yesterday, I have had an unblemished record with the User Profile Service – being able to successfully provision it first time at all sites I have visited (and no I did not resort to running it all as administrator). Of course, we all have Spence to thank for this with his rational guide. Nevertheless, I am strongly starting to think that I should write the irrational guide as a sort of bizzaro version of Spencers articles, which combines his rigour with some mega-ranting ;-).

So what happened to blemish my perfect record? Bloody Active Directory policies – that’s what.

In case you didn’t know, SharePoint uses a scaled down, pre-release version of Forefront Identify Manager. Presumably the logic here to this was to allow more flexibility, by two-way syncing to various directory services, thereby saving the SharePoint team development time and effort, as well as being able to tout yet another cool feature to the masses. Of course, the trade-off that the programmers overlooked is the insane complexity that they introduced as a result. I’m sure if you asked Microsoft’s support staff what they think of the UPS, they will tell you it has not worked out overly well. Whether that feedback has made it way back to the hallowed ground of the open-plan cubicles of SharePoint product development I can only guess. But I theorise that if Microsoft made their SharePoint devs accountable for providing front-line tech support for their components, they will suddenly understand why conspiracy theorist support and infrastructure guys act the way they do.

Anyway I better supress my desire for an all out rant and tell you the problem and the fix. The site in question was actually a fairly simple set-up. Two server farm and a single AD forest. About the only thing of significance from the absolute stock standard setup was that the active directory NETBIOS name did not match the active directory fully qualified domain name. But this is actually a well known and well covered by TechNet and Spence. A quick bit of PowerShell goodness and some AD permission configuration sorts the issue.

Yet when I provisioned the User Profile Service Application and then tried to start the User Profile Synchronisation Service on the server (the big, scary step that strikes fear into practitioners), I hit the sadly common “stuck on starting” error. The ULS logs told me utterly nothing of significance – even when i turned the debug juice to full throttle. The ever helpful windows event logs showed me Event ID 3:

ForeFront Identity Manager,
Level: Error

.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: HostId is not registered
at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
at Microsoft.ResourceManagement.Data.DataAccess.RetrieveWorkflowDataForHostActivator(Int16 hostId, Int16 pingIntervalSecs, Int32 activeHostedWorkflowDefinitionsSequenceNumber, Int16 workflowControlMessagesMaxPerMinute, Int16 requestRecoveryMaxPerMinute, Int16 requestCleanupMaxPerMinute, Boolean runRequestRecoveryScan, Boolean& doPolicyApplicationDispatch, ReadOnlyCollection`1& activeHostedWorkflowDefinitions, ReadOnlyCollection`1& workflowControlMessages, List`1& requestsToRedispatch)
at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RetrieveWorkflowDataForHostActivator()
at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.ActivateHosts(Object source, ElapsedEventArgs e)

The most common issue with this message is the NETBIOS issue I mentioned earlier. But in my case this proved to be fruitless. I also took Spence’s advice and installed the Feb 2011 cumulative update for SharePoint 2010, but to no avail. Every time I provisioned the UPS sync service, I received the above persistent error – many, many, many times. 🙁

For what its worth, forget googling the above error because it is a bit of a red herring and you will find issues that will likely point you to the wrong places.

In my case, the key to the resolution lay in understanding my previously documented issue with the UPS and self-signed certificate creation. This time, I noticed that the certificates were successfully created before the above error happened.  MIISCLIENT showed no configuration had been written to Forefront Identity Manager at all. Then I remembered that the SharePoint User Profile Service Application talks to Forefront over HTTPS on port 5725. As soon as I remembered that HTTP was the communication mechanism, I had a strong suspicion on where the problem was – as I have seen this sort of crap before…

I wondered if some stupid proxy setting was getting in the way. Back in the halcyon days of SharePoint 2003, I had this issue when scheduling SMIGRATE tasks, where the account used to run SMIGRATE is configured to use a proxy server, would fail. To find out if this was the case here, a quick execute of the GPRESULT tool and we realised that there was a proxy configuration script applied at the domain level for all users. We then logged in as the farm account interactively (given that to provision the UPS it needs to be Administrator anyway this was not a problem). We then disabled all proxy configuration via Internet explorer and tried again.

Blammo! The service provisions and we are cooking with gas! it was the bloody proxy server. Reconfigure group policy and all is good.

Conclusion

The moral of the story is this. Anytime windows components communicate with each-other via HTTP, there is always a chance that some AD induced dumbass proxy setting might get in the way. If not that, stateful security apps that check out HTTP traffic or even a corrupted cache (as happened in this case). The ULS logs will never tell you much here, because the problem is not SharePoint per se, but the registry configuration enforced by policy.

So, to ensure that you do not get affected by this, configure all SharePoint servers to be excluded from proxy access, or configure the SharePoint farm account not to use a proxy server at all. (Watch for certificate revocation related slowness if you do this though).

Finally, I called this post “consequences of complexity” because this sort of problem is very tricky to identify the root cause. With so many variables in the mix, how the hell can people figure this sort of stuff out?

Seriously Microsoft, you need to adjust your measures of success to include resiliency of the platform!

 

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

Un-Managed Metadata: A couple of gotchas

Send to Kindle

As the SharePoint 2010 dust settles, gushing praise and inflated expectations are slowly replaced by the cold hard reality, as people come to grips with the limitations of the product. One such area is with the managed metadata service. Don’t get me wrong, I like managed metadata a lot and I can see a little ecosystem building around that functionality specifically. But it does have a couple of big gotchas that you should be aware of before making a big investment with it.

The sad irony is that these issues are actually not the fault of the managed metadata service, but the applications that are supposed to embrace and extend SharePoint and therefore accommodate it.

The reason I am calling out these two particular issues, is that I can see many people making assumptions that this will just work, make a significant investment in time and effort to develop an IA based around that assumption and then face the painful truth of having to work around them. After examining two issues that I suspect will cause some pain, we will then have a quick look through some of the implications and mixed messages that Microsoft are sending to organisations.

InfoPath Web Suckiness

The first issue that has gotten a bit of attention is the fact that the managed metadata columns cannot be used in browser based InfoPath forms. In other words, if you have a list with a managed metadata column and think that it would be cool to customise that list forms using InfoPath, you will be in for a nasty surprise. You will receive the following error message:

image

"The following fields in the SharePoint list are not supported because of their data type and will not be available in InfoPath Designer:

MyColumn (TaxonomyFieldType)”

I have a screenshot pasted above – which actually has come from a nice explanation of the problem made by Alana Helbig (hope you don’t mind Alana). Alana shows that if you persist and open the form in InfoPath, the managed metadata field will be hidden away, never to be edited again (and therefore pointless). She also also demonstrates that the behaviour is even worse if the managed metadata column is marked as mandatory. In this case, SharePoint totally spits the dummy if you modify the form with InfoPath and then try to load it. You will get a message along the lines of: “The following required fields are missing from the form” and a ULS correlation ID for your trouble.

Paradocially, InfoPath does support managed metadata when forms are displayed natively (ie not web based). This is proven by the fact that the MSOffice Document Information Panel (DIP) contains a control to display managed metadata information (in case you are not aware the DIP is an InfoPath form). The screengrab below shows Word showing two managed metadata columns (one with the imaginative name of “aaa” which I have clicked on) allowing me to pick terms from the term set.

image

Taking a closer look, if I edit the Document Information Panel settings in InfoPath, I can clearly see that there is a Managed Metadata picker control.

image

I never bothered with the SP2010 betas because I was doing a lot of non SharePoint work at the time. But from my reading, it seems that at one point, InfoPath could display managed metadata in the browser but it was yanked from the RTM because of quality issues. Some forums suggest it won’t be corrected in any service packs soon. I certainly hope they are wrong.

Conclusion? I assume Microsoft knew the implications of this decision – yet still, I feel that this will cause a lot of frustration and grief.

SharePoint Workspace 2010 Suckiness

This is the same issue, just using a different Microsoft client application: SharePoint Workspace 2010. SPW2010, if you haven’t seen it, provides a client for SharePoint 2010 that enables real-time synchronization of desktop content with SharePoint documents and lists.

This gotcha is one I fear might be even more insidious than the InfoPath one in certain geographic locations. This is because offline access tends to be an area people will think about later in the project. Where I live (Western Australia), is remote and dominated by mining. As a result, Groove had considerable popularity when you are in the middle of nowhere with nothing but a poor satellite link with >1 second latency ;-). Many organisations will flock to SharePoint Workspace 2010 because of its much improved compatibility with synchronising SharePoint lists, libraries and views.

The problem is that managed metadata columns can be viewed in SharePoint Workspace 2010 but not edited at all. 

Below I show a custom list with a managed metadata column called Projects.  The next image shows the same list in SharePoint Workspace 2010.  Note how the Project column is displayed in the list of projects, but is not displayed in the view/edit item form below it.

image

image

Now some of you might be thinking that this is fairly minor, and that not being able to modify managed metadata columns is not a problem. But check out what happens when the managed metadata column is made mandatory. SharePoint Workspace 2010 displays the error below when attempting to view the list.

image

Ouch! When you click on the More Info link in the ribbon, you are presented with a scarily similar message to InfoPath.

image

It gets better (mixed messages)

Office 2010 has finally gotten past the use-case I described in my “folders are bad and other urban legends” post. In Office 2010, application centric users have the option to browse document libraries not just by folders, but by metadata as shown below. Note how we are browsing a managed metadata term store in the File>Open dialog box in Word 2010.

image

The rub with this functionality though, is it only works for managed metadata columns. You might have configured a choice field for metadata navigation and in the browser, you can sort, slide and dice via those columns as well. But in Office 2010, you can only use managed metadata or folders. No views, and no other column types. This will inevitably lead organisations to invest time and effort to create an information architecture around the managed metadata construct. Yet by utilising managed metadata in this way, we consign ourselves to not being able to edit any of this data when we take it offline using SharePoint Workspace 2010.

*sigh* So basically, the more you try and move to a metadata driven, taxonomy approach, the more you make yourself rigid and inflexible.

But there is more…

By the way, managed metadata is not the only column type that suffers this fate. If you enable ratings on a list or library you will see the same problem. The first screengrab below is InfoPath and the next two are SharePoint Workspace 2010.

image

image 

image

Conclusion: Violating the laws of motion

More than ever, SharePoint is a minefield of caveats. These examples conclusively disprove Newtons laws of motion because for every possible action, there are just not equal and opposite reactions, but potentially many more opposite reactions. More then ever, practitioners have to understand these complex dependencies, and then somehow explain them to stakeholders without giving them a brain explosion. Is it little wonder that there is commonly a big gap between the slick demos and the reality on the ground?

 

Thanks for reading

 

Paul Culmsee

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

Sack Justin Bieber with SPD2010 and Forms Services – Part 2

Send to Kindle

Hi

This is part 2 of a quick (but huge) post on my experiences working with SharePoint Designer 2010 workflows and Forms Services. In part 1, we used the scenario of an employee termination form, and sacked Justin Bieber. Now we want to ensure that the SharePoint user experience for sacking Justin Bieber is seamless and intuitive.

Truth be told, I have never actually heard a Justin Bieber song because we have not had a television in the house for over a year. Ignorance is bliss, but I have seen enough news reports that I still want to sack him!

In part 1, we examined the ability of SPD2010 to leverage InfoPath for tailoring forms used by workflows. We then covered creating a workflow utilising the Start Approval Process action, which enables us to do a couple of cool things without custom programming.

Now we are onto the next two steps.

Continue reading

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

Sack Justin Bieber with SPD2010 and Forms Services – Part 1

Send to Kindle

Hi all

A very long time ago now, I had the ambition to write an end-to-end blog post series called “A Humble Tribute to the Leave Form”. The intent was to show InfoPath Forms Services 2007 in all its glory – from its initially seductive, demo friendly first impressions, through to all of the dodgy workarounds and .net code required to get it to adequately handle a relatively simple business process like employee leave applications.

As it happened, I got through seven and a half blog posts and never finished it as events kind of overtook me. Its a pity because I didn’t get to the nasty bits. But one of the side effects of getting as far as I did, was that I ended up getting a lot of work developing leave forms!

Thus, now that SharePoint 2010 is upon us, it was inevitable that I would eventually get called to develop a leave form for this new edition. I now have done so, and in the process learnt a couple of new things that I thought were blogworthy – especially around getting things to play nice in a sustainable manner.

I came to realise that anytime you write any content that involves InfoPath, you end up with a stupid number of screenshots, and you are perpetually torn on the amount of detail to cover. So this time, rather than do a twelve post monster, I’ll just do 2 posts (real programmers will still find this post waffly but hopefully normal humans won’t).

I am not going to do a total beginners course here, I will assume instead you have done some basic InfoPath and SharePoint Designer workflows previously, and now want to know some interesting ways to do a general approval type process using SharePoint 2010. My main focus here is to deal with handling browser based InfoPath forms with SharePoint Designer workflows.

Continue reading

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

Announcing the SamePage Alliance

Send to Kindle

image

This is really great, and something that’s been a long time coming. On behalf of my partners at Seven Sigma, I’m announcing the formation of the SamePage Alliance. A strategic partnership with Seven Sigma and 21Apps, founded by Andrew Woodward, as founding members. SamePage is a commercial relationship where we will be pooling the respective talents of our organisations together and expanding our service offerings to clients.

I first met Andrew in San Diego in 2008, the SharePoint Best Practices Conference, where I was a very nervous first-time presenter, wondering if all of my wicked problem stuff would resonate with the US audience. Andrew was there, presenting on TDD and Scrum, and apart from having someone in the US I could talk about the cricket with, it was immediately clear that we had a hell of a lot in common. It was like he held a big piece to a puzzle, and I held another piece. The irony was that I never got to see his talk as, if I recall, we presented at the same time. But back then (Feb 08) I made a rather prophetic statement at the end of my report of that conference.

“I feel some future collaboration in the very near future.  Andrew Woodward will definitely be a part of it (although he doesn’t know it yet…Hehehe).”

Funny how things turn out. We have collaborated on a number of different things since then, both within the SharePoint realm and beyond it. The common interests run deep and between 21apps and Seven Sigma, there is a lot of experience there. During the SharePoint Evolutions conference, where a certain volcano prevented me attending, Andrew ran my wicked problems/SharePoint/IBIS talk for me and did a tremendous job (I watched all the tweets from Perth).

In terms of practicalities, we will be reselling each-others products and services. Seven Sigma entered the training space this year, writing the SharePoint 2010 Governance and Information Architecture course that 3Grow and Microsoft New Zealand use to certify gold partners for SharePoint prowess. Seven Sigma also developed a unique 1 and 2-day SharePoint Governance f-Laws course, with content drawn from our sensemaking work that we ran in the New Zealand and Sydney conferences. When it came to who could possibly teach Seven Sigma courseware, the obvious answer was Andrew Woodward, given our shared interests and his sterling job at Evolutions.

21apps released their first SharePoint product into the marketplace this year – 21scrum, and 21apps authors and teaches workshops and training for development teams looking to improve their quality of development around the SharePoint space.

Further to this, we will be co-developing products as well. Seven Sigma has been brewing some things in the cauldron for some time and 21apps will be part of this development effort.

In general terms, we offer great SharePoint competencies across training, governance, infrastructure, development and delivery. Our combined offerings means that we can offer:

  • Global software development and round the clock SharePoint managed services and support
  • World-unique strategic advisory services and collaborative facilitation services, incorporating goal alignment, shared visioning and performance framework development, large group facilitation, user and community engagement, enquiry by design, risk analysis, critical thinking and decision methodologies, process improvement
  • Beyond SharePoint, we can provide full enterprise architecture and analysis services over the program life cycle
  • The first output of this new arrangement is a two-day course to be run in London in mid November. Andrew will be there too, and we will cover my SharePoint Governance f-Laws course as well as material from the recent Information Architecture course in New Zealand. If you have SharePoint competencies and find yourself having to bridge the gap between organisational aspirations and SharePoint as the enabler to that aspiration, then this session is for you.

    You can find out more about this event and register at the 21apps site.

    Looking forward to seeing you all there!

    www.samepage.co

    www.21apps.co.uk

    www.sevensigma.com.au

     Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

    No Tags

    Send to Kindle

    Why me? Web part errors on new web applications

    Send to Kindle

    Oh man, it’s just not my week. After nailing a certificate issue yesterday that killed user profile provisioning, I get an even better one today! I’ve posted it here as a lesson on how not to troubleshoot this issue!

    The symptoms:

    I created a brand new web application on a SP2010 farm, and irrespective of the site collection I subsequently create, I get the dreaded error "Web Part Error: This page has encountered a critical error. Contact your system administrator if this problem persists"

    Below is a screenshot of a web app using the team site template. Not so good huh?

    image

    The swearing…

    So faced with this broken site, I do what any other self respecting SharePoint consultant would do. I silently cursed Microsoft for being at the root of all the world’s evils and took a peek into that very verbose and very cryptic place known as the ULS logs. Pretty soon I found messages like:

    0x3348 SharePoint Foundation         General                       8sl3 High     DelegateControl: Exception thrown while building custom control ‘Microsoft.SharePoint.SPControlElement’: This page has encountered a critical error. Contact your system administrator if this problem persists. eff89784-003b-43fd-9dde-8377c4191592

    0x3348 SharePoint Foundation         Web Parts                     7935 Information http://sp:81/default.aspx – An unexpected error has been encountered in this Web Part.  Error: This page has encountered a critical error. Contact your system administrator if this problem persists.,

    Okay, so that is about as helpful as a fart in an elevator, so I turned up the debug juice using that new, pretty debug juicer turner-upper (okay, the diagnostic logging section under monitoring in central admin). I turned on a variety of logs at different times including.

    • SharePoint Foundation           Configuration                   Verbose
    • SharePoint Foundation           General                         Verbose
    • SharePoint Foundation           Web Parts                       Verbose
    • SharePoint Foundation           Feature Infrastructure          Verbose
    • SharePoint Foundation           Fields                          Verbose
    • SharePoint Foundation           Web Controls                    Verbose
    • SharePoint Server               General                         Verbose
    • SharePoint Server               Setup and Upgrade               Verbose
    • SharePoint Server               Topology                        Verbose

    While my logs got very big very quickly, I didn’t get much more detail apart from one gem,to me, seemed so innocuous amongst all the detail, yet so kind of.. fundamental 🙂

    0x3348 SharePoint Foundation         Web Parts                     emt7 High     Error: Failure in loading assembly: Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a eff89784-003b-43fd-9dde-8377c4191592

    That rather scary log message was then followed up by this one – which proved to be the clue I needed.

    0x3348 SharePoint Foundation         Runtime                       6610 Critical Safe mode did not start successfully. This page has encountered a critical error. Contact your system administrator if this problem persists. eff89784-003b-43fd-9dde-8377c4191592

    It was about this time that I also checked the event logs (I told you this post was about how not to troubleshoot) and I saw the same entry as above.

    Log Name:      Application
    Source:        Microsoft-SharePoint Products-SharePoint Foundation
    Event ID:      6610
    Description:
    Safe mode did not start successfully. This page has encountered a critical error. Contact your system administrator if this problem persists.

    I read the error message carefully. This problem was certainly persisting and I was the system administrator, so I contacted myself and resolved to search google for the “Safe mode did not start successfully” error.

    The 46 minute mark epiphany

    image

    If you watch the TV series “House”, you will know that House always gets an epiphany around the 46 minute mark of the show, just in time to work out what the mystery illness is and save the day. Well, this is the 46 minute mark of this post!

    I quickly found that others had this issue in the past, and it was the process where SharePoint checks web.config to process all of the controls marked as safe. If you have never seen this, it is the section of your SharePoint web application configuration file that looks like this:

    image 

    This particular version of the error is commonly seen when people deploy multiple servers in their SharePoint farm, and use a different file path for the INETPUB folder. In my case, this was a single server. So, although I knew I was on the right track, I knew this wasn’t the issue.

    My next thought was to run the site in full trust mode, to see if that would make the site work. This is usually a setting that makes me mad when developers ask for it because it tells me they have been slack. I changed the entry

    <trust level="WSS_Minimal" originUrl="" />

    to

    <trust level="Full" originUrl="" />

    But to no avail. Whatever was causing this was not affected by code access security.

    I reverted back to WSS_Minimal and decided to remove all of the SafeControl entries from the web.config file, as shown below. I knew the site would bleat about it, but was interested if the “Safe Mode” error would go away.

    image

    The result? My broken site was now less broken. It was still bitching, but now it appeared to be bitching more like what I was expecting.

    image

    After that, it was a matter of adding back the <safecontrol> elements and retrying the site. It didn’t take long to pinpoint the offending entry.

    <SafeControl Assembly="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" Namespace="Microsoft.SharePoint.WebPartPages" TypeName="ContentEditorWebPart" Safe="False" />

    As soon as I removed this entry the site came up fine. I even loaded up the content editor web part without this entry and it worked a treat. Therefore, how this spurious entry got there is still a mystery.

    The final mystery

    My colleague and I checked the web.config file in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG. This is the one that gets munged with other webconfig.* files when a new web application is provisioned.

    Sure enough, its modified date was July 29 (just outside the range of the SharePoint and event logs unfortunately). When we compared against a known good file from another SharePoint site, we immediately saw the offending entry.

    image

    The solution store on this SharePoint server is empty and no 3rd party stuff to my knowledge has been installed here. But clearly this file has been modified. So, we did what any self respecting SharePoint consultant would do…

    …we blamed the last guy.

     

    Thanks for reading

    Paul Culmsee

    www.sevensigma.com.au

     Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

    No Tags

    Send to Kindle

    More User Profile Sync issues in SP2010: Certificate Provisioning Fun

    Send to Kindle

    Wow, isn’t the SharePoint 2010 User Profile Service just a barrel of laughs. Without a bit of context, when you compare it to SP2007, you can do little but shake your head in bewilderment at how complex it now appears.

    I have a theory about all of this. I think that this saga started over a beer in 2008 or so.

    I think that Microsoft decided that SharePoint 2010 should be able to write back to Active Directory (something that AD purists dislike but sold Bamboo many copies of their sync tool). Presumably the SharePoint team get on really well with the Forefront Identify Manager team and over a few Friday beers, the FIM guys said “Why write your own? Use our fit for purpose tool that does exactly this. As an added bonus, you can sync to other directories easily too”.

    “Damn, that *is* a good idea”, says the SharePoint team and the rest is history. Remember the old saying, the road to hell is paved with good intentions?

    Anyways, when you provision the UPS enough times, and understand what Forefront Identity Manager does, it all starts to make sense. Of course, to have it make sense, requires you to mess it up in the first place and I think that everyone universally will do this – because it is essentially impossible to get it right the first time unless you run everything as domain administrator. This is a key factor that I feel did not get enough attention within the product team. I have now visited three sites where I have had to correct issues with the user profile service. Remember, not all of us do SharePoint all day – for the humble system administrator that is also catering with the overall network, this implementation is simply too complex. Result? Microsoft support engineers are going to get a lot of calls here – and its going to cost Microsoft that way.

    One use-case they never tested

    I am only going to talk about one of the issues today because Spence has written the definitive article that will get you through if you are doing it from scratch.

    I went to a client site where they had attempted to provision the user profile synchronisation unsuccessfully. I have no idea of the things they tried because I wasn’t there unfortunately, but I made a few changes to permissions, AD rights and local security policy as per Spencers post. I then provisioned user profile sync again and I hit this issue. A sequence of 4 event log entries.

    Event ID:      234
    Description:
    ILM Certificate could not be created: Cert step 2 could not be created: C:\Program Files\Microsoft Office Servers\14.0\Tools\MakeCert.exe -pe -sr LocalMachine -ss My -a sha1 -n CN=”ForefrontIdentityManager” -sky exchange -pe -in “ForefrontIdentityManager” -ir localmachine -is root

    Event ID:      234
    Description:
    ILM Certificate could not be created: Cert could not be added: C:\Program Files\Microsoft Office Servers\14.0\Tools\CertMgr.exe -add -r LocalMachine -s My -c -n “ForefrontIdentityManager” -r LocalMachine -s TrustedPeople

    Event ID:      234
    Description:
    ILM Certificate could not be created: netsh http error:netsh http add urlacl url=
    http://+:5725/ user=Domain\spfarm sddl=D:(A;;GA;;;S-1-5-21-2972807998-902629894-2323022004-1104)

    Event ID:      234
    Description:
    Cannot get the self issued certificate thumbprint:

    The theory

    Luckily this one of those rare times where the error message actually makes sense (well – if you have worked with PKI stuff before). Clearly something went wrong in the creation of certificates. Looking at the sequence of events, it seems that as part of provisioning ForeFront Identity Manager, a self signed certificate was created for the Computer Account, added to the Trusted People certificate store and then is used for SSL on a web application or web service listening on port 5725.

    By the way, don’t go looking for the web app listening on such a port in IIS because its not there. Just like SQL Reporting Services, FIM likely uses very little of IIS and doesn’t need the overhead.  

    The way I ended up troubleshooting this issue was to take a good look at the first error in the sequence and what the command was trying to do. Note the description in the event log is important here. “ILM Certificate could not be created: Cert step 2 could not be created”. So this implies that this command is the second step in a sequence and there was a step 1 that must have worked. Below is the step 2 command that was attempted.

    C:\Program Files\Microsoft Office Servers\14.0\Tools\MakeCert.exe -pe -sr LocalMachine -ss My -a sha1 -n CN=”ForefrontIdentityManager” -sky exchange -pe -in “ForefrontIdentityManager” -ir localmachine -is root

    When you create a certificate, it has to have a trusted issuer. Verisign and Thawte are examples and all browsers consider them trustworthy issuers. But we are not using 3rd party issuers here. Forefront uses a self-signed certificate. In other words, it trusts itself. We can infer that step 1 is the creation of this self-trusted certificate issuer by looking at the parameters of the MakeCert command that step 2 is using.

    Now I am not going to annotate every Makecert parameter here, but the English version of the command above says something like:

    Make me a shiny new certificate for the local machine account and call it “ForefrontIdentityManager”, issued by a root certificate that can be found in the trusted root store also called ForeFrontIdentityManager.

    So this command implies that step 1 was the creation of that root certificate that issues the other certificates. (Product team – you could have given the name of the root issuer certificate something different to the issued certificate)

    The root cause

    Now that we have established a theory of what is going on, the next step is to run the failing Makecrt command from a prompt and see what we get back. Make sure you do this as the Sharepoint farm account so you are comparing apples with apples.

    C:\Program Files\Microsoft Office Servers\14.0\Tools>MakeCert.exe -pe -sr LocalMachine -ss My -a sha1 -n CN=”ForefrontIdentityManager” -sky exchange -pe -in “ForefrontIdentityManager” -ir localmachine -is root

    Error: There are more than one matching certificate in the issuer’s root cert store. Failed

    Aha! so what do we have here? The error message states that we have more than 1 matching certificate in the issuers root certificate store.

    For what its worth it is the parameters “-ir localmachine -is root” that specifies the certificate store to use. In this case, it is the trusted root certificate store on the local computer.

    So lets go and take a look. Run the Microsoft Management Console (MMC) and Choose “Add/Remove Snap In” from the File Menu.

    image

    From the list of snap ins choose Certificates and then choose “Computer Account”

    image

    Now in the list of certificate stores, we need to examine the one that the command refers to: The Trusted Root Certification Authorities store. Well, look at that, the error was telling the truth!

    image

    Clearly the Forefront Identity Manager provisioning/unprovisioning code does not check for all circumstances. I can only theorise what my client did to cause this situation because I wasn’t privy to what was done on this particular install before I got there. but step 1 of this provisioning process would create an issuing certificate whether one existed already or not. Step 2 then failed because it had no way to determine which of these certificates is the authoritative one.

    This was further exacerbated because each re-attempt creates another root certificate because there is no check whether a certificate already exists.

    The cure is quite easy. Just delete all of the ForefrontIdentityManager certificates from the Trusted Root Certification Authorities and re-provision the user profile sync in SharePoint. Provided that there is no certificate in this store to begin with, step 1 will create it and step 2 will then be able to create the self signed certificate using this issuer just fine.

    Conclusion (and minor rant)

    Many SharePoint pros have commented on the insane complexity of the new design of the user profile sync system. Yes I understand the increased flexibility offered by the new regime, leveraging a mature product like Forefront, but I see that with all of this flexibility comes risk that has not been well accounted for. SP2010 is twice as tough to learn as SP2007 and it is more likely that you will make a mistake than not making one. The more components added, the more points of failure and the less capable over-burdened support staff are in dealing with it when it happens.

    SharePoint 2010 is barely out of nappies and I have already been in a remediation role for several sites over the user profile service alone.

    I propose that Microsoft add a new program level KPI to rate how well they are doing with their SharePoint product development. That KPI should be something like % of time a system administrator can provision a feature without making a mistake or resorting to running it all as admin. The benefit to Microsoft would be tangible in terms of support calls and failed implementations. Such a KPI would force the product team to look at an example like the user profile service and think “How can we make this more resilient?”. “How can we remove the number of manual steps that are not obvious?”, “how can we make the wizard clearer to understand?” (yes they *will* use the wizard).

    Right now it feels like the KPI was how many features could be crammed, in as well as how much integration between components there is. If there is indeed a KPI for that they definitely nailed it this time around.

    Don’t get me wrong – its all good stuff, but if Microsoft are stumping seasoned SharePoint pros with this stuff, then they definitely need to change the focus a bit in terms of what constitutes a good product.

    Thanks for reading

    Paul Culmsee

    www.sevensigma.com.au

     Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

    No Tags

    Send to Kindle