I’ve been nerding it up lately SharePointwise, doing the geeky things that geeks like to do like ADFS and Claims Authentication. So in between trying to get my book fully edited ready for publishing, I might squeeze out the odd technical SharePoint post. Today I had to troubleshoot a broken SharePoint people search for the first time in a while. I thought it was worth explaining the crawl process a little and talking about the most likely ways in which is will break for you, in order of likelihood as I see it. There are articles out on this topic, but none that I found are particularly comprehensive.

Background stuff

If you consider yourself a legendary IT pro or SharePoint god, feel free to skip this bit. If you prefer a more gentle stroll through SharePoint search land, then read on…

When you provision a search service application as part of a SharePoint installation, you are asked for (among other things), a windows account to use for the search service. Below shows the point in the GUI based configuration step where this is done. First up we choose to create a search service application, and then we choose the account to use for the “Search Service Account”. By default this is the account that will do the crawling of content sources.

Now the search service account is described as so: “.. the Windows Service account for the SharePoint Server Search Service. This setting affects all Search Service Applications in the farm. You can change this account from the Service Accounts page under Security section in Central Administration.”

In reading this, suggests that the windows service (“SharePoint Server Search 14”) would run under this account. The reality is that the SharePoint Server Search 14 service account is the farm account. You can see the pre and post provisioning status below. First up, I show below where SharePoint has been installed and the SharePoint Server Search 14 service is disabled and with service credentials of “Local Service”.

The next set of pictures show the Search Service Application provisioned according to the following configuration:

• Search service account: SEVENSIGMA\searchservice
• Search admin web service account: SEVENSIGMA\searchadminws
• Search query and site settings account: SEVENSIGMA\searchqueryss

You can see this in the screenshots below.

Once the service has been successfully provisioned, we can clearly see the “Default content access account” is based on the “Search service account” as described in the configuration above (the first of the three accounts).

Finally, as you can see below, once provisioned, it is the SharePoint farm account that is running the search windows service.

Once you have provisioned the Search Service Application, the default content access (in my case SEVENSIGMA\searchservice), it is granted “Read” access to all web applications via Web Application User Policies as shown below. This way, no matter how draconian the permissions of site collections are, the crawler account will have the access it needs to crawl the content, as well as the permissions of that content. You can verify this by looking at any web application in Central Administration (except for central administration web application) and choosing “User Policy” from the ribbon. You will see in the policy screen that the “Search Crawler” account has “Full Read” access.

In case you are wondering why the search service needs to crawl the permissions of content, as well as the content itself, it is because it uses these permissions to trim search results for users who do not have access to content. After all, you don’t want to expose sensitive corporate data via search do you?

There is another more subtle configuration change performed by the Search Service. Once the evilness known as the User Profile Service has been provisioned, the Search service application will grant the Search Service Account specific permission to the User Profile Service. SharePoint is smart enough to do this whether or not the User Profile Service application is installed before or after the Search Service Application. In other words, if you install the Search Service Application first, and the User Profile Service Application afterwards, the permission will be granted regardless.

The specific permission by the way, is “Retrieve People Data for Search Crawlers” permission as shown below:

Getting back to the title of this post, this is a critical permission, because without it, the Search Server will not be able to talk to the User Profile Service to enumerate user profile information. The effect of this is empty "People Search results.

How people search works (a little more advanced)

Right! Now that the cool kids have joined us (who skipped the first section), lets take a closer look at SharePoint People Search in particular. This section delves a little deeper, but fear not I will try and keep things relatively easy to grasp.

Once the Search Service Application has been provisioned, a default content source, called – originally enough – “Local SharePoint Sites” is created. Any web applications that exist (and any that are created from here on in) will be listed here. An example of a freshly minted SharePoint server with a single web application, shows the following configuration in Search Service Application:

Now hopefully http://web makes sense. Clearly this is the URL of the web application on this server. But you might be wondering that sps3://web is? I will bet that you have never visited a site using sps3:// site using a browser either. For good reason too, as it wouldn’t work.

This is a SharePointy thing – or more specifically, a Search Server thing. That funny protocol part of what looks like a URL, refers to a connector. A connector allows Search Server to crawl other data sources that don’t necessarily use HTTP. Like some native, binary data source. People can develop their own connectors if they feel so inclined and a classic example is the Lotus Notes connector that Microsoft supply with SharePoint. If you configure SharePoint to use its Lotus Notes connector (and by the way – its really tricky to do), you would see a URL in the form of:

notes://mylotusnotesbox

Make sense? The protocol part of the URL allows the search server to figure out what connector to use to crawl the content. (For what its worth, there are many others out of the box. If you want to see all of the connectors then check the list here).

But the one we are interested in for this discussion is SPS3: which accesses SharePoint User profiles which supports people search functionality. The way this particular connector works is that when the crawler accesses this SPS3 connector, it in turns calls a special web service at the host specified. The web service is called spscrawl.asmx and in my example configuration above, it would be http://web/_vti_bin/spscrawl.asmx

The basic breakdown of what happens next is this:

1. Information about the Web site that will be crawled is retrieved (the GetSite method is called passing in the site from the URL (i.e the “web” of sps3://web)
2. Once the site details are validated the service enumerates all of the use profiles
3. For each profile, the method GetItem is called that retrieves all of the user profile properties for a given user. This is added to the index and tagged as content class of “urn:content-class:SPSPeople” (I will get to this in a moment)

Now admittedly this is the simple version of events. If you really want to be scared (or get to sleep tonight) you can read the actual SP3 protocol specification PDF.

Right! Now lets finish this discussion by this notion of contentclass. The SharePoint search crawler tags all crawled content according to its class. The name of this “tag” – or in correct terminology “managed property” – is contentclass. By default SharePoint has a People Search scope. It is essentially a limits the search to only returning content tagged as “People” contentclass.

Now to make it easier for you, Dan Attis listed all of the content classes that he knew of back in SharePoint 2007 days. I’ll list a few here, but for the full list visit his site.

• “STS_Web” – Site
• “STS_List_850″ – Page Library
• “STS_List_DocumentLibrary” – Document Library
• “STS_ListItem_DocumentLibrary” – Document Library Items
• “STS_ListItem_Tasks” – Tasks List Item
• “STS_ListItem_Contacts” – Contacts List Item
• “urn:content-class:SPSPeople” – People

(why some properties follow the universal resource name format I don’t know *sigh* – geeks huh?)

So that was easy Paul! What can go wrong?

So now we know that although the protocol handler is SPS3, it is still ultimately utilising HTTP as the underlying communication mechanism and calling a web service, we can start to think of all the ways that it can break on us. Let’s now take a look at common problem areas in order of commonality:

1. The Loopback issue.

This has been done to death elsewhere and most people know it. What people don’t know so well is that the loopback fix was to prevent an extremely nasty security vulnerability known as a replay attack that came out a few years ago. Essentially, if you make a HTTP connection to your server, from that server and using a name that does not match the name of the server, then the request will be blocked with a 401 error. In terms of SharePoint people search, the sps3:// handler is created when you create your first web application. If that web application happens to be a name that doesn’t match the server name, then the HTTP request to the spscrawl.asmx webservice will be blocked due to this issue.

As a result your search crawl will not work and you will see an error in the logs along the lines of:

• Access is denied: Check that the Default Content Access Account has access to the content or add a crawl rule to crawl the content (0×80041205)
• The server is unavailable and could not be accessed. The server is probably disconnected from the network.   (0x80040d32)
• ***** Couldn’t retrieve server http://web.sevensigma.com policy, hr = 80041205 – File:d:\office\source\search\search\gather\protocols\sts3\sts3util.cxx Line:548

There are two ways to fix this. The quick way (DisableLoopbackCheck) and the right way (BackConnectionHostNames). Both involve a registry change and a reboot, but one of them leaves you much more open to exploitation. Spence Harbar wrote about the differences between the two some time ago and I recommend you follow his advice.

(As an slightly related side note, I hit an issue with the User Profile Service a while back where it gave an error: “Exception occurred while connecting to WCF endpoint: System.ServiceModel.Security.MessageSecurityException: The HTTP request was forbidden with client authentication scheme ‘Anonymous’. —> System.Net.WebException: The remote server returned an error: (403) Forbidden”. In this case I needed to disable the loopback check but I was using the server name with no alternative aliases or full qualified domain names. I asked Spence about this one and it seems that the DisableLoopBack registry key addresses more than the SMB replay vulnerability.)

2. SSL

If you add a certificate to your site and mark the site as HTTPS (by using SSL), things change. In the example below, I installed a certificate on the site http://web, removed the binding to http (or port 80) and then updated SharePoint’s alternate access mappings to make things a HTTPS world.

Note that the reference to SPS3://WEB is unchanged, and that there is also a reference still to HTTP://WEB, as well as an automatically added reference to HTTPS://WEB

So if we were to run a crawl now, what do you think will happen? Certainly we know that HTTP://WEB will fail, but what about SPS3://WEB? Lets run a full crawl and find out shall we?

Checking the logs, we have the unsurprising error “the item could not be crawled because the crawler could not contact the repository”. So clearly, SPS3 isn’t smart enough to work out that the web service call to spscrawl.asmx needs to be done over SSL.

Fortunately, the solution is fairly easy. There is another connector, identical in function to SPS3 except that it is designed to handle secure sites. It is “SPS3s”. We simple change the configuration to use this connector (and while we are there, remove the reference to HTTP://WEB)

Now we retry a full crawl and check for errors… Wohoo – all good!

It is also worth noting that there is another SSL related issue with search. The search crawler is a little fussy with certificates. Most people have visited secure web sites that warning about a problem with the certificate that looks like the image below:

Now when you think about it, a search crawler doesn’t have the luxury of asking a user if the certificate is okay. Instead it errs on the side of security and by default, will not crawl a site if the certificate is invalid in some way. The crawler also is more fussy than a regular browser. For example, it doesn’t overly like wildcard certificates, even if the certificate is trusted and valid (although all modern browsers do).

To alleviate this issue, you can make the following changes in the settings of the Search Service Application: Farm Search Administration->Ignore SSL warnings and tick “Ignore SSL certificate name warnings”.

The implication of this change is that the crawler will now accept any old certificate that encrypts website communications.

3. Permissions and Change Legacy

Lets assume that we made a configuration mistake when we provisioned the Search Service Application. The search service account (which is the default content access account) is incorrect and we need to change it to something else. Let’s see what happens.

In the search service application management screen, click on the default content access account to change credentials. In my example I have changed the account from SEVENSIGMA\searchservice to SEVENSIGMA\svcspsearch

Having made this change, lets review the effect in the Web Application User Policy and User Profile Service Application permissions. Note that the user policy for the old search crawl account remains, but the new account has had an entry automatically created. (Now you know why you end up with multiple accounts with the display name of “Search Crawling Account”)

Now lets check the User Profile Service Application. Now things are different! The search service account below refers to the *old* account SEVENSIGMA\searchservice. But the required permission of “Retrieve People Data for Search Crawlers” permission has not been granted!

If you traipsed through the ULS logs, you would see this:

Leaving Monitored Scope (Request (GET:https://web/_vti_bin/spscrawl.asmx)). Execution Time=7.2370958438429 c2a3d1fa-9efd-406a-8e44-6c9613231974
mssdmn.exe (0x23E4) 0x2B70 SharePoint Server Search FilterDaemon e4ye High FLTRDMN: Errorinfo is "HttpStatusCode Unauthorized The request failed with HTTP status 401: Unauthorized." [fltrsink.cxx:553] d:\office\source\search\native\mssdmn\fltrsink.cxx
mssearch.exe (0x02E8) 0x3B30 SharePoint Server Search Gatherer cd11 Warning The start address sps3s://web cannot be crawled. Context: Application ‘Search_Service_Application’, Catalog ‘Portal_Content’ Details: Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled. (0×80041205)

To correct this issue, manually grant the crawler account the “Retrieve People Data for Search Crawlers” permission in the User Profile Service. As a reminder, this is done via the Administrators icon in the “Manage Service Applications” ribbon.

Once this is done run a fill crawl and verify the result in the logs.4.

4. Missing root site collection

A more uncommon issue that I once encountered is when the web application being crawled is missing a default site collection. In other words, while there are site collections defined using a managed path, such as http://WEB/SITES/SITE, there is no site collection defined at HTTP://WEB.

The crawler does not like this at all, and you get two different errors depending on whether the SPS or HTTP connector used.

• SPS:// – Error in PortalCrawl Web Service (0×80042617)
• HTTP:// – The item could not be accessed on the remote server because its address has an invalid syntax (0×80041208)

The fix for this should be fairly obvious. Go and make a default site collection for the web application and re-run a crawl.

5. Alternative Access Mappings and Contextual Scopes

SharePoint guru (and my squash nemesis), Nick Hadlee posted recently about a problem where there are no search results on contextual search scopes. If you are wondering what they are Nick explains:

Contextual scopes are a really useful way of performing searches that are restricted to a specific site or list. The “This Site: [Site Name]”, “This List: [List Name]” are the dead giveaways for a contextual scope. What’s better is contextual scopes are auto-magically created and managed by SharePoint for you so you should pretty much just use them in my opinion.

The issue is that when the alternate access mapping (AAM) settings for the default zone on a web application do not match your search content source, the contextual scopes return no results.

I came across this problem a couple of times recently and the fix is really pretty simple – check your alternate access mapping (AAM) settings and make sure the host header that is specified in your default zone is the same url you have used in your search content source. Normally SharePoint kindly creates the entry in the content source whenever you create a web application but if you have changed around any AAM settings and these two things don’t match then your contextual results will be empty. Case Closed!

Thanks Nick

6. Active Directory Policies, Proxies and Stateful Inspection

A particularly insidious way to have problems with Search (and not just people search) is via Active Directory policies. For those of you who don’t know what AD policies are, they basically allow geeks to go on a power trip with users desktop settings. Consider the image below. Essentially an administrator can enforce a massive array of settings for all PC’s on the network. Such is the extent of what can be controlled, that I can’t fit it into a single screenshot. What is listed below is but a small portion of what an anal retentive Nazi administrator has at their disposal (mwahahaha!)

Common uses of policies include restricting certain desktop settings to maintain consistency, as well as enforce Internet explorer security settings, such as proxy server and security settings like maintaining the trusted sites list. One of the common issues encountered with a global policy defined proxy server in particular is that the search service account will have its profile modified to use the proxy server.

The result of this is that now the proxy sits between the search crawler and the content source to be crawled as shown below:

Crawler —–> Proxy Server —–> Content Source

Now even though the crawler does not use Internet Explorer per se, proxy settings aren’t actually specific to Internet Explorer. Internet explorer, like the search crawler, uses wininet.dll. Wininet is a module that contains Internet-related functions used by Windows applications and it is this component that utilises proxy settings.

Sometimes people will troubleshoot this issue by using telnet to connect to the HTTP port. "ie: “Telnet web 80”. But telnet does not use the wininet component, so is actually not a valid method for testing. Telnet will happily report that the web server is listening on port 80 or 443, but it matters not when the crawler tries to access that port via the proxy. Furthermore, even if the crawler and the content source are on the same server, the result is the same. As soon as the crawler attempts to index a content source, the request will be routed to the proxy server. Depending on the vendor and configuration of the proxy server, various things can happen including:

• The proxy server cannot handle the NTLM authentication and passes back a 400 error code to the crawler
• The proxy server has funky stateful inspection which interferes with the allowed HTTP verbs in the communications and interferes with the crawl

For what its worth, it is not just proxy settings that can interfere with the HTTP communications between the crawler and the crawled. I have seen security software also get in the way, which monitors HTTP communications and pre-emptively terminates connections or modifies the content of the HTTP request. The effect is that the results passed back to the crawler are not what it expects and the crawler naturally reports that it could not access the data source with suitably weird error messages.

Now the very thing that makes this scenario hard to troubleshoot is the tell-tale sign for it. That is: nothing will be logged in the ULS logs, not the IIS logs for the search service. This is because the errors will be logged in the proxy server or the overly enthusiastic stateful security software.

If you suspect the problem is a proxy server issue,  but do not have access to the proxy server to check logs, the best way to troubleshoot this issue is to temporarily grant the search crawler account enough access to log into the server interactively. Open internet explorer and manually check the proxy settings. If you confirm a policy based proxy setting, you might be able to temporarily disable it and retry a crawl (until the next AD policy refresh reapplies the settings). The ideal way to cure this problem is to ask your friendly Active Directory administrator to either:

• Remove the proxy altogether from the SharePoint server (watch for certificate revocation slowness as a result)
• Configure an exclusion in the proxy settings for the AD policy to that the content sources for crawling are not proxied
• Create a new AD policy specifically for the SharePoint box so that the default settings apply to the rest of the domain member computers.

If you suspect the issue might be overly zealous stateful inspection, temporarily disable all security-type software on the server and retry a crawl. Just remember, that if you have no logs on the server being crawled, chances are its not being crawled and you have to look elsewhere.

7. Pre-Windows 2000 Compatibility Access Group

In an earlier post of mine, I hit an issue where search would yield no results for a regular user, but a domain administrator could happily search SP2010 and get results. Another symptom associated with this particular problem is certain recurring errors event log – Event ID 28005 and 4625.

• ID 28005 shows the message “An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user ‘DOMAIN\someuser’, error code 0×5”.
• The 4625 error would complain “An account failed to log on. Unknown user name or bad password status 0xc000006d, sub status 0xc0000064” or else “An Error occured during Logon, Status: 0xc000005e, Sub Status: 0×0”

If you turn up the debug logs inside SharePoint Central Administration for the “Query” and “Query Processor” functions of “SharePoint Server Search” you will get an error “AuthzInitializeContextFromSid failed with ERROR_ACCESS_DENIED. This error indicates that the account under which this process is executing may not have read access to the tokenGroupsGlobalAndUniversal attribute on the querying user’s Active Directory object. Query results which require non-Claims Windows authorization will not be returned to this querying user.

The fix is to add your search service account to a group called “Pre-Windows 2000 Compatibility Access” group. The issue is that SharePoint 2010 re-introduced something that was in SP2003 – an API call to a function called AuthzInitializeContextFromSid. Apparently it was not used in SP2007, but its back for SP2010. This particular function requires a certain permission in Active Directory and the “Pre-Windows 2000 Compatibility Access” group happens to have the right required to read the “tokenGroupsGlobalAndUniversal“ Active Directory attribute that is described in the debug error above.

8. Bloody developers!

Finally, Patrick Lamber blogs about another cause of crawler issues. In his case, someone developed a custom web part that had an exception thrown when the site was crawled. For whatever reason, this exception did not get thrown when the site was viewed normally via a browser. As a result no pages or content on the site could be crawled because all the crawler would see, no matter what it clicked would be the dreaded “An unexpected error has occurred”. When you think about it, any custom code that takes action based on browser parameters such as locale or language might cause an exception like this – and therefore cause the crawler some grief.

In Patricks case there was a second issue as well. His team had developed a custom HTTPModule that did some URL rewriting. As Patrick states “The indexer seemed to hate our redirections with the Response.Redirect command. I simply removed the automatic redirection on the indexing server. Afterwards, everything worked fine”.

In this case Patrick was using a multi-server farm with a dedicated index server, allowing him to remove the HTTP module for that one server. in smaller deployments you may not have this luxury. So apart from the obvious opportunity to bag programmers , this example nicely shows that it is easy for a 3rd party application or code to break search. What is important for developers to realise is that client web browsers are not the only thing that loads SharePoint pages.

If you are not aware, the user agent User Agent string identifies the type of client accessing a resource. This is the means by which sites figure out what browser you are using. A quick look at the User Agent parameter by SharePoint Server 2010 search reveals that it identifies itself as “Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot)“. At the very least, test any custom user interface code such as web parts against this string, as well as check the crawl logs when it indexes any custom developed stuff.

Conclusion

Well, that’s pretty much my list of gotchas. No doubt there are lots more, but hopefully this slightly more detailed exploration of them might help some people.

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

www.spgovia.com

                      

No Tags

Hiya

A few months back I posted a relatively well behaved rant over the ridiculously complex User Profile Service Application of SharePoint 2010. I think this component in particular epitomises SharePoint 2010’s awful combination of “design by committee” clunkiness, along with real-world sheltered Microsoft product manager groupthink which seems to rate success on the number of half baked features packed in, as opposed to how well those features install logically, integrate with other products and function properly in real-world scenarios.

Now truth be told, until yesterday, I have had an unblemished record with the User Profile Service – being able to successfully provision it first time at all sites I have visited (and no I did not resort to running it all as administrator). Of course, we all have Spence to thank for this with his rational guide. Nevertheless, I am strongly starting to think that I should write the irrational guide as a sort of bizzaro version of Spencers articles, which combines his rigour with some mega-ranting .

So what happened to blemish my perfect record? Bloody Active Directory policies – that’s what.

In case you didn’t know, SharePoint uses a scaled down, pre-release version of Forefront Identify Manager. Presumably the logic here to this was to allow more flexibility, by two-way syncing to various directory services, thereby saving the SharePoint team development time and effort, as well as being able to tout yet another cool feature to the masses. Of course, the trade-off that the programmers overlooked is the insane complexity that they introduced as a result. I’m sure if you asked Microsoft’s support staff what they think of the UPS, they will tell you it has not worked out overly well. Whether that feedback has made it way back to the hallowed ground of the open-plan cubicles of SharePoint product development I can only guess. But I theorise that if Microsoft made their SharePoint devs accountable for providing front-line tech support for their components, they will suddenly understand why conspiracy theorist support and infrastructure guys act the way they do.

Anyway I better supress my desire for an all out rant and tell you the problem and the fix. The site in question was actually a fairly simple set-up. Two server farm and a single AD forest. About the only thing of significance from the absolute stock standard setup was that the active directory NETBIOS name did not match the active directory fully qualified domain name. But this is actually a well known and well covered by TechNet and Spence. A quick bit of PowerShell goodness and some AD permission configuration sorts the issue.

Yet when I provisioned the User Profile Service Application and then tried to start the User Profile Synchronisation Service on the server (the big, scary step that strikes fear into practitioners), I hit the sadly common “stuck on starting” error. The ULS logs told me utterly nothing of significance – even when i turned the debug juice to full throttle. The ever helpful windows event logs showed me Event ID 3:

ForeFront Identity Manager,
Level: Error

.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: HostId is not registered
at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)
at Microsoft.ResourceManagement.Data.DataAccess.RetrieveWorkflowDataForHostActivator(Int16 hostId, Int16 pingIntervalSecs, Int32 activeHostedWorkflowDefinitionsSequenceNumber, Int16 workflowControlMessagesMaxPerMinute, Int16 requestRecoveryMaxPerMinute, Int16 requestCleanupMaxPerMinute, Boolean runRequestRecoveryScan, Boolean& doPolicyApplicationDispatch, ReadOnlyCollection`1& activeHostedWorkflowDefinitions, ReadOnlyCollection`1& workflowControlMessages, List`1& requestsToRedispatch)
at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RetrieveWorkflowDataForHostActivator()
at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.ActivateHosts(Object source, ElapsedEventArgs e)

The most common issue with this message is the NETBIOS issue I mentioned earlier. But in my case this proved to be fruitless. I also took Spence’s advice and installed the Feb 2011 cumulative update for SharePoint 2010, but to no avail. Every time I provisioned the UPS sync service, I received the above persistent error – many, many, many times.

For what its worth, forget googling the above error because it is a bit of a red herring and you will find issues that will likely point you to the wrong places.

In my case, the key to the resolution lay in understanding my previously documented issue with the UPS and self-signed certificate creation. This time, I noticed that the certificates were successfully created before the above error happened.  MIISCLIENT showed no configuration had been written to Forefront Identity Manager at all. Then I remembered that the SharePoint User Profile Service Application talks to Forefront over HTTPS on port 5725. As soon as I remembered that HTTP was the communication mechanism, I had a strong suspicion on where the problem was – as I have seen this sort of crap before…

I wondered if some stupid proxy setting was getting in the way. Back in the halcyon days of SharePoint 2003, I had this issue when scheduling SMIGRATE tasks, where the account used to run SMIGRATE is configured to use a proxy server, would fail. To find out if this was the case here, a quick execute of the GPRESULT tool and we realised that there was a proxy configuration script applied at the domain level for all users. We then logged in as the farm account interactively (given that to provision the UPS it needs to be Administrator anyway this was not a problem). We then disabled all proxy configuration via Internet explorer and tried again.

Blammo! The service provisions and we are cooking with gas! it was the bloody proxy server. Reconfigure group policy and all is good.

Conclusion

The moral of the story is this. Anytime windows components communicate with each-other via HTTP, there is always a chance that some AD induced dumbass proxy setting might get in the way. If not that, stateful security apps that check out HTTP traffic or even a corrupted cache (as happened in this case). The ULS logs will never tell you much here, because the problem is not SharePoint per se, but the registry configuration enforced by policy.

So, to ensure that you do not get affected by this, configure all SharePoint servers to be excluded from proxy access, or configure the SharePoint farm account not to use a proxy server at all. (Watch for certificate revocation related slowness if you do this though).

Finally, I called this post “consequences of complexity” because this sort of problem is very tricky to identify the root cause. With so many variables in the mix, how the hell can people figure this sort of stuff out?

Seriously Microsoft, you need to adjust your measures of success to include resiliency of the platform!

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

Hiya

My colleague Chris Tomich, recently saved the day with a nice PowerShell script that makes the management and migration of managed metadata between farms easier. If you are an elite developer type, you can scroll down to the end of this post. if you are a normal human, read on for background and context.

The Issue

Recently, I had a client who was decommissioning an old active directory domain and starting afresh. Unfortunately for me, there was a SharePoint 2010 farm on the old domain and they had made considerable effort in leveraging managed metadata.

Under normal circumstances, migrating SharePoint in this scenario is usually done via a content database migration into a newly set up farm. This is because the built in SharePoint 2010 backup/restore, although better than 2007, presumes that the underlying domain that you are restoring to is the same (bad things happen if it is not). Additionally, the complex interdependencies between service applications (especially the user profile service) means that restoring one service application like Managed metadata is fiddly and difficult.

I was constrained severely by time, so I decided to try a quick and dirty approach to see what would happen. I created a new, clean farm and provisioned a managed metadata service application. The client in question had half a dozen term sets in the old SP2010 farm and around 20 managed metadata columns in the site collection that leveraged them. I migrated the content database from the old farm to the new farm via the database attach method and that worked fine.

I then used the SolidQ export tool from codeplex to export those term sets from the managed metadata service of the source farm. Since the new SP2010 farm had a freshly provisioned managed metadata service application, re-importing those terms and term sets into this farm was a trivial process and went without a hitch.

I knew that managed metadata stores each term as a unique identifier (GUID I presume because programmers love GUID’s). Since I was migrating a content database to a different farm there were two implications.

1. The terms stored in lists and libraries in that database would reference the GUID’s of the original managed metadata service
2. The site columns themselves, would reference the GUID of the term sets of the source managed metadata service

But the GUIDs on the managed metadata service on the new farm will not match the originals. I did not import the terms with their GUID’s intact. Instead I simply imported terms which created new GUID’s. Now everything was orphaned. Managed metadata columns would not know which term store they are assigned to. For example: just because a term set called, say “Clients” exists in the source farm, when a term set is created in the destination farm with the same name, the managed metadata columns will not automatically “re-link” to the clients term set.

So what does a managed metadata column orphaned from a term set look like? Fortunately, SharePoint handles this gracefully. The leftmost image below shows a managed metadata column happily linked to its term set. The rightmost image shows what it looks like when that term set is deleted while the column remains. As you can see we now have no reference to a term set. However, it is easy to re-point the column to another term set.

When you examine list or library items that have managed metadata entered for them prior to the term set being deleted, SharePoint prevents the managed metadata control from being used (the Project Phase column below is greyed out). Importantly, the original data is not lost since SharePoint actually stores the term along with the rest of the item metadata, but the term cannot be updated of course as there is no term set to refer to.

Once you reconnect a managed metadata column to a term set, the existing term will still be orphaned. The process of reconnecting does not do any sort of processing of terms to reconnect them. I show this below. In the example below, notice how this time, the managed metadata control is enabled again (as it has a valid term set), but the terms within it are marked red, signalling that they are invalid terms that do not exist in this term set. Notice though the second screen. By clicking on the orphaned term, managed metadata finds it in the associated term set and offers the right suggestion. With a single click, the term is re-linked back to the term store and the updated GUID is stored for the item. Too easy.

As the sequence of events above show, it is fairly easy to re-link a term to the term set by clicking on it and letting managed metadata find that term in the term set. However, it is not something you would want to do for 10000 items in a list! In addition, the relative ease of relinking a term like the Location column above is elegant only because it is a single term that does not show the hierarchy. This is not the only configuration available for managed metadata though.

If you have set up your column to use multiple managed metadata entries, as well as allow multiple entries, you will have something like the screen below. In this example, we have three terms selected. “Borough”, “District” and “Neighbourhood”. All three of these terms are at the bottom of a deep hierarchy of terms. (eg Continent > Political Entity > Country > province or State > County or Region > City). As a result, things look ugly.

Unfortunately in this use-case, our single click method will not work. Firstly, we have to click each and every term that has been added to this column one at a time. Secondly and more importantly, even if we do click it, the term will not be found in in the new term set. The only way to deal with this is to manually remove the term and re-enter or use the rather clumsy term picker as the sequence below shows.

Note how as each term is selected, it is marked in black. We have to manually delete the orphaned terms in red.

Finally, after far too many mouse clicks, we are back in business.

A better approach?

I showed this behaviour to my Seven Sigma colleague, Chris Tomich. Chris took a look at this issue and within a few hours trotted out a terrific PowerShell script to programmatically reconnect terms to a new term set. You can grab the source code for this script from Chris’s blog. The one thing Chris did tell me was that he wasn’t able to re-link a site or list column to its replacement term set. That bit you have to do yourself. This is because only the GUID of a term set is used against a column, which means without access to the original server, you do not know which term set to reattach.

Fortunately, there are not that many term sets to deal with, and the real pain is reattaching orphaned terms anyway.

Chris also added a ‘test’ mode to the script, so that it will flag if a field is currently connected to an invalid term set. This is very handy because you can run the script against a site collection and it will help you narrow down which columns are managed metadata and which need to be updated.

From the SharePoint 2010 Management Shell, the script syntax is: (assuming you call your script ReloadTermSets.ps1) -

.\ReloadTermSets.ps1 -web “http://localhost” -test -recurse

web – This is the web address to use. test – This uses a ‘test’ mode that will output lines detailing found fields and the values found for them in the term store and won’t save the items. recurse – This flag will cause the script to recurse through all child sites.

Of course, the usual disclaimers apply. Use the script/advice at your own risk. By using/viewing/distributing it you accept responsibility for any loss/corruption of data that may be incurred by said actions.

I hope that people find this useful. I sure did

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

This is really great, and something that’s been a long time coming. On behalf of my partners at Seven Sigma, I’m announcing the formation of the SamePage Alliance. A strategic partnership with Seven Sigma and 21Apps, founded by Andrew Woodward, as founding members. SamePage is a commercial relationship where we will be pooling the respective talents of our organisations together and expanding our service offerings to clients.

I first met Andrew in San Diego in 2008, the SharePoint Best Practices Conference, where I was a very nervous first-time presenter, wondering if all of my wicked problem stuff would resonate with the US audience. Andrew was there, presenting on TDD and Scrum, and apart from having someone in the US I could talk about the cricket with, it was immediately clear that we had a hell of a lot in common. It was like he held a big piece to a puzzle, and I held another piece. The irony was that I never got to see his talk as, if I recall, we presented at the same time. But back then (Feb 08) I made a rather prophetic statement at the end of my report of that conference.

“I feel some future collaboration in the very near future.  Andrew Woodward will definitely be a part of it (although he doesn’t know it yet…Hehehe).”

Funny how things turn out. We have collaborated on a number of different things since then, both within the SharePoint realm and beyond it. The common interests run deep and between 21apps and Seven Sigma, there is a lot of experience there. During the SharePoint Evolutions conference, where a certain volcano prevented me attending, Andrew ran my wicked problems/SharePoint/IBIS talk for me and did a tremendous job (I watched all the tweets from Perth).

In terms of practicalities, we will be reselling each-others products and services. Seven Sigma entered the training space this year, writing the SharePoint 2010 Governance and Information Architecture course that 3Grow and Microsoft New Zealand use to certify gold partners for SharePoint prowess. Seven Sigma also developed a unique 1 and 2-day SharePoint Governance f-Laws course, with content drawn from our sensemaking work that we ran in the New Zealand and Sydney conferences. When it came to who could possibly teach Seven Sigma courseware, the obvious answer was Andrew Woodward, given our shared interests and his sterling job at Evolutions.

21apps released their first SharePoint product into the marketplace this year – 21scrum, and 21apps authors and teaches workshops and training for development teams looking to improve their quality of development around the SharePoint space.

Further to this, we will be co-developing products as well. Seven Sigma has been brewing some things in the cauldron for some time and 21apps will be part of this development effort.

In general terms, we offer great SharePoint competencies across training, governance, infrastructure, development and delivery. Our combined offerings means that we can offer:

The first output of this new arrangement is a two-day course to be run in London in mid November. Andrew will be there too, and we will cover my SharePoint Governance f-Laws course as well as material from the recent Information Architecture course in New Zealand. If you have SharePoint competencies and find yourself having to bridge the gap between organisational aspirations and SharePoint as the enabler to that aspiration, then this session is for you.

You can find out more about this event and register at the 21apps site.

Looking forward to seeing you all there!

www.samepage.co

www.21apps.co.uk

www.sevensigma.com.au

                      

No Tags

Oh man, it’s just not my week. After nailing a certificate issue yesterday that killed user profile provisioning, I get an even better one today! I’ve posted it here as a lesson on how not to troubleshoot this issue!

The symptoms:

I created a brand new web application on a SP2010 farm, and irrespective of the site collection I subsequently create, I get the dreaded error "Web Part Error: This page has encountered a critical error. Contact your system administrator if this problem persists"

Below is a screenshot of a web app using the team site template. Not so good huh?

The swearing…

So faced with this broken site, I do what any other self respecting SharePoint consultant would do. I silently cursed Microsoft for being at the root of all the world’s evils and took a peek into that very verbose and very cryptic place known as the ULS logs. Pretty soon I found messages like:

0×3348 SharePoint Foundation         General                       8sl3 High     DelegateControl: Exception thrown while building custom control ‘Microsoft.SharePoint.SPControlElement’: This page has encountered a critical error. Contact your system administrator if this problem persists. eff89784-003b-43fd-9dde-8377c4191592

0×3348 SharePoint Foundation         Web Parts                     7935 Information http://sp:81/default.aspx – An unexpected error has been encountered in this Web Part.  Error: This page has encountered a critical error. Contact your system administrator if this problem persists.,

Okay, so that is about as helpful as a fart in an elevator, so I turned up the debug juice using that new, pretty debug juicer turner-upper (okay, the diagnostic logging section under monitoring in central admin). I turned on a variety of logs at different times including.

• SharePoint Foundation           Configuration                   Verbose
• SharePoint Foundation           General                         Verbose
• SharePoint Foundation           Web Parts                       Verbose
• SharePoint Foundation           Feature Infrastructure          Verbose
• SharePoint Foundation           Fields                          Verbose
• SharePoint Foundation           Web Controls                    Verbose
• SharePoint Server               General                         Verbose
• SharePoint Server               Setup and Upgrade               Verbose
• SharePoint Server               Topology                        Verbose

While my logs got very big very quickly, I didn’t get much more detail apart from one gem,to me, seemed so innocuous amongst all the detail, yet so kind of.. fundamental

0×3348 SharePoint Foundation         Web Parts                     emt7 High     Error: Failure in loading assembly: Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a eff89784-003b-43fd-9dde-8377c4191592

That rather scary log message was then followed up by this one – which proved to be the clue I needed.

0×3348 SharePoint Foundation         Runtime                       6610 Critical Safe mode did not start successfully. This page has encountered a critical error. Contact your system administrator if this problem persists. eff89784-003b-43fd-9dde-8377c4191592

It was about this time that I also checked the event logs (I told you this post was about how not to troubleshoot) and I saw the same entry as above.

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Event ID:      6610
Description:
Safe mode did not start successfully. This page has encountered a critical error. Contact your system administrator if this problem persists.

I read the error message carefully. This problem was certainly persisting and I was the system administrator, so I contacted myself and resolved to search google for the “Safe mode did not start successfully” error.

The 46 minute mark epiphany

If you watch the TV series “House”, you will know that House always gets an epiphany around the 46 minute mark of the show, just in time to work out what the mystery illness is and save the day. Well, this is the 46 minute mark of this post!

I quickly found that others had this issue in the past, and it was the process where SharePoint checks web.config to process all of the controls marked as safe. If you have never seen this, it is the section of your SharePoint web application configuration file that looks like this:

This particular version of the error is commonly seen when people deploy multiple servers in their SharePoint farm, and use a different file path for the INETPUB folder. In my case, this was a single server. So, although I knew I was on the right track, I knew this wasn’t the issue.

My next thought was to run the site in full trust mode, to see if that would make the site work. This is usually a setting that makes me mad when developers ask for it because it tells me they have been slack. I changed the entry

<trust level="WSS_Minimal" originUrl="" />

to

<trust level="Full" originUrl="" />

But to no avail. Whatever was causing this was not affected by code access security.

I reverted back to WSS_Minimal and decided to remove all of the SafeControl entries from the web.config file, as shown below. I knew the site would bleat about it, but was interested if the “Safe Mode” error would go away.

The result? My broken site was now less broken. It was still bitching, but now it appeared to be bitching more like what I was expecting.

After that, it was a matter of adding back the <safecontrol> elements and retrying the site. It didn’t take long to pinpoint the offending entry.

<SafeControl Assembly="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" Namespace="Microsoft.SharePoint.WebPartPages" TypeName="ContentEditorWebPart" Safe="False" />

As soon as I removed this entry the site came up fine. I even loaded up the content editor web part without this entry and it worked a treat. Therefore, how this spurious entry got there is still a mystery.

The final mystery

My colleague and I checked the web.config file in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG. This is the one that gets munged with other webconfig.* files when a new web application is provisioned.

Sure enough, its modified date was July 29 (just outside the range of the SharePoint and event logs unfortunately). When we compared against a known good file from another SharePoint site, we immediately saw the offending entry.

The solution store on this SharePoint server is empty and no 3rd party stuff to my knowledge has been installed here. But clearly this file has been modified. So, we did what any self respecting SharePoint consultant would do…

…we blamed the last guy.

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

I have been doing a bit more tech work than normal lately – SP2010 popularity I guess, and was asked to remediate a few issues on a problematic server that I hadn’t set up. The server in question had a number of issues (over and above the usual “lets all run it as one account” type stuff) that had a single root cause, so I thought I’d quickly document the symptoms and the cause here.

Symptom 1: SQL Server Event ID 28005:

“An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user ‘DOMAIN\someuser’, error code 0×5”

This error would be reported in the Application log around 70 times per minute. As it happened, I had removed the account in question from running any of the SharePoint web, application or windows services, but this still persisted. I suspect SharePoint was installed as this account because it was the db owner of many of the databases on the SQL Server. Whatever the case, SQL was whinging about it despite its lack of actual need to be there.

Symptom 2: Event ID 4625:

At a similar rate of knots as the SQL error, was the rate of 4625 errors in the security log. These logs were not complaining about the account that the SQL event complained about, but instead it complained about ANY account running the SQL Server instance. I tried network service, a domain account and a local account and saw similar errors (although the local one had a different code).

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Event ID:      4625
Task Category: Logon
Keywords:      Audit Failure
Description:  An account failed to log on. 

Subject:
    Security ID:        NETWORK SERVICE
    Account Name:        COMPUTER$
    Account Domain:        DOMAIN
    Logon ID:        0x3e4 

Failure Information:
    Failure Reason:        Unknown user name or bad password.
    Status:            0xc000006d
    Sub Status:        0xc0000064 

Process Information:
    Caller Process ID:    0x7e0
    Caller Process Name:    E:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe 

Detailed Authentication Information:
    Logon Process:        Authz   
    Authentication Package:    Kerberos

When using a local, rather than a domain user account the code was:

Failure Information:
    Failure Reason:        An Error occured during Logon.
    Status:            0xc000005e
    Sub Status:        0x0 

Symptom 3: Search only working for domain administrators

On top of the logs being filled by endless entries of the previous two, I had another error (the original reason why I was called in actually). A SharePoint search would yield zip, nada, zero, no results for a regular user, but a domain administrator could happily search SP2010 and get results. (well actually regular users did get some results – people searched actually worked fine).

The crawler was fine and dandy and the default content source had not been messed with. There were no errors or logs to suggest anything untoward.

The resolution:

It was the second symptom that threw me because I thought that the problem must have been kerberos config. But I quickly discounted that after checking SPN’s and the like (notwithstanding the fact this was a single server install anyway!)

On a hunch (helped by the fact that I had dealt with the issue of registering managed accounts not so long ago), I concentrated on the user account that was causing SQL Server all the trouble (Event ID 28005). I loaded up Active Directory and temporarily changed the security of this user account so that “Authenticated Users” had “READ” access to it.

As soon as I did this, both event ID 28005 and 4625 stopped.

I then checked the search (symptom 3) and it was still barfing. In this case I decided to turn up the debug juice on the “Query” and “Query Processor” functions of “SharePoint Server Search”.

After upping the level of verbosity, I found what I was looking for.

08/12/2010 22:13:41.00     w3wp.exe (0×2228)                           0x1F0C    SharePoint Server Search          Query Processor                   g2j3    High        AuthzInitializeContextFromSid failed with ERROR_ACCESS_DENIED. This error indicates that the account under which this process is executing may not have read access to the tokenGroupsGlobalAndUniversal attribute on the querying user’s Active Directory object. Query results which require non-Claims Windows authorization will not be returned to this querying user.    debd2c54-d6a5-41b8-bf26-c4697b36f4d4

I knew immediately that this was very likely the same issue as the first two symptoms and when I googled this result, my Perth compatriot, Jeremy Thake had hit the same issue in July. The fix is to add your search service account to a group called “Pre-Windows 2000 Compatibility Access” group. This group happens to have the right required to read this attribute. Whether it is the same attribute I needed for my SQL issue, or the registering managed accounts issue I don’t know, but what I do know is that this group loosens up permissions enough to cure all four of these issues.

The little security guy in me keeps telling me I should confirm the least privilege in each of these scenarios, but hey if Microsoft are saying to put the accounts into this group, then who am I to argue?

Finally, it turns out that SP2010 re-introduced something that was in SP2003. A call to a function called AuthzInitializeContextFromSid which seems to be the root of it all. Apparently it was not used in SP2007, but its sure there now. I assume that one of the many stored procedures that SharePoint would call in SQL may have been the cause of Symptom 1. When you look at Symptom 2, it now makes sense because AD was faithfully reporting an access denied when the call to AuthzInitializeContextFromSid was made. It reported SQL Server as the culprit, I assume, because of a stored proc doing the work perhaps? It just sucks that the security event logged is a fairly stock message that doesn’t give you enough specifics to really work out what is going on.

Anyway, I hope that helps someone else – as googling the event ID details is not overly helpful

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

Note: For those of you who do not wish to read my usual verbose writing, then skip to the last section where there is a free web part to download and try out.

Unless you are a complete SharePoint nerd and world events don’t interest you while you spend your hours in a darkened room playing with the SP2010 beta, you would no doubt be aware that one of the most significant collaborative events in the world is currently taking place.

The United Nations climate change conference in Copenhagen this month is one of the most important world gatherings of our time. You might wonder why, as a SharePoint centric blog, I am writing about this. The simple answer is that this conference in which the world will come together to negotiate and agree on one of the toughest wicked problems of our time. How to tackle international climate change in a coordinated global way. As I write this, things do not seem to be going so well .

Climate change cuts to the heart of the wellbeing expected by every one of us. Whether you live in an affluent country or a developing nation, the stakes are high and the issues at hand are incredibly complex and tightly intertwined. It might all seem far away and out of sight/out of mind, but it is clear that we will all be affected by the outcomes for better and worse. The spectre of the diminishing window of opportunity to deal with this issue means that an unprecedented scale of international cooperation will be required to produce an outcome that can satisfy all stakeholders in an environmentally, economically and social bottom line.

Can it be done? For readers who are practitioners of SharePoint solutions, you should have an appreciation of the difficulty that a supposedly “collaborative tool” actually is to improve collaboration. Therefore, I want you to imagine your most difficult, dysfunctional project that you have ever encountered and just try and now multiply it by a million, gazzilion times. If there are ever lessons to be learned about effective collaboration among a large, diverse group on a hugely difficult issue, then surely it is this issue and this event.

Our contribution

My colleagues and I became interested in sense-making and collaboration on wicked problems some time back, and through the craft of Dialogue Mapping, we have had the opportunity to help diverse groups successfully work through some very challenging local issues. I need to make it clear that much of what we do in this area is far beyond SharePoint in terms of project difficulty, and in fact we often deal with non IT projects and problems that have significant social complexity.

Working with people like city planners, organisational psychologists, environmental scientists and community leaders to name a few, has rubbed off on myself and my colleagues. Through the sense-making process that we practice with these groups, we have started to see a glimpse of the world through their eyes. For me in particular, it has challenged my values, social conscience and changed the entire trajectory of where I thought my career would go. I feel that the experience has made me a much better practitioner of collaborative tools like SharePoint and I am a textbook case of the the notion that the key to improving in your own discipline, is to learn from people outside of it.

We have now become part of a global sense-making community, much like the global SharePoint community in a way. A group of diverse people that come together via common interest. To that end, my colleague at Seven Sigma, Chris Tomich has embarked on a wonderful initiative that I hope you may find of interest. He has enlisted the help of several world renowned sense-makers, such as Jeff Conklin of Cognexus and David Price of Debategraph, and created a site, http://www.copenhagensummitmap.org/, where we will attempt to create a global issue map of the various sessions and talks at the Copenhagen summit. The aim of this exercise is to try and help interested people cut through the fog of issues and understand the points of view of the participants. We are utilising IBIS, the grammar behind dialogue mapping, and the DebateGraph tool for the shared display.

How you can help

If you feel that issue mapping is for you, then I encourage you to sign up to Debategraph and help contribute to the Copenhagen debate by mapping the dialogue of the online sessions (which you can view from the site).

Otherwise, Chris has written a simple, free web part, specifically for Copenhagen which can be downloaded “Mapping tools” section of the Copenhagen site. The idea is that if you or your organisation wish to keep up with the latest information from the conference, then installing this web part onto your site, will allow all of your staff to see the Copenhagen debate unfold live via your SharePoint portal. Given that SharePoint is particularly powerful at surfacing data for business intelligence, think of this web part as a means to display global intelligence (or lack thereof, depending on your political view ).

Installing is the usual process for a SharePoint solution file. Add the solution to central admin, deploy it to your web application of choice and then activate the site collection scoped feature called “Seven Sigma Debategraph Components”. The web part will be then available to add to a page layout or web part page.

The properties of this web part allow you some fine grained control over how the Debategraph map renders inside SharePoint. The default is to show the Debategraph stream view, which is a twitter style view of the recent updates as shown in the example below.

Stream view is not the only view available. Detail view is also very useful for rationale that has supplementary information, as shown in the example below.

By the way, you can use this web part to display any Debategraph debate – not just Copenhagen. The Debategraph map to display is also controlled via the web part properties.

For information on how to change the default map, then check out this webcast I recorded for the previous version here.

I hope that some of you find this web part of use and look forward to any feedback.

Kind regards

Paul Culmsee

www.sevensigma.com.au

                      

No Tags

I had a problem this week that got resolved with something quite obvious, but I learned a lot in the process, so I am detailing it here.

Symptoms: Using MOSS 2007 SP2 and Office 2003 SP3, users complained that metadata on documents went missing. Consider the scenario below where we have a sample document library.

Our user opens the “Memo to Council re Convenor” document into Word 2003, makes a change and saves it. Note the difference. Where’s my metadata?

I asked about this on the ozmoss mailing list, and others noted the same issue. Some had applied the June 2009 cumulative update for WSS3 and the problem was solved for them. I applied this update, and it improved the situation, but did not cure the problem totally.

Upon further investigation, we were able to ascertain that the problem would only happen on certain PCs. The same user could update the same document on a different PC and it would work fine. A fiddler trace of the process allowed me to narrow down the issue and understand the sequence of events. The root cause of the issue was incorrect handling of HTML/JavaScript by Office 2003 and/or Internet Explorer. This manifests itself in an inconsistency in the display of properties in the Office “Web File Properties” dialog box (file->properties in an office 2003 app). Consider the example below.

Note that the first dialog shows that values for the metadata columns are blank or default. Now consider the same document opened on a different PC. Well what do you know – we have metadata!

Fiddler confirmed that when saving a document to SharePoint from an Office 2003 application, the same HTTP request is made to SharePoint as is done when displaying the document properties in the above example.

In both cases, A HTTP GET request is made to the owssvr.dll  (WSS RPC)  and the dialogview method is called. Therefore, the root of my problem has something to do with the fact that the data returned by this call was not being parsed properly by MSWord on one PC (as illustrated by the first of the above dialog box screenshots). When a user than saved their edits, blank or default values are being written back to SharePoint, in effect “losing” the metadata.

So let’s take a look at the HTTP call and the data returned. The call to owssvr.dll  looks like this.

GET /somesite/_vti_bin/owssvr.dll?location=My%20Committee/Agenda%20attachment%20July%2009.doc&dialogview=SaveForm HTTP/1.1

The output returned by SharePoint is a bunch of HTML and JavaScript that MSWord then renders by calling the Internet Explorers rendering framework programmatically. (I’ve included sample output at the end of the post for the hardcore nerds).

So the question now becomes, what was preventing the correct HTML rendering on one PC and not the other? This was a difficult question to answer because I was unable to find a way to debug JavaScript when the output was rendered in Office 2003 apps.

When you think about it, there can be many potential causes here. Given that Internet Explorer is effectively doing the work (WinINet), this whole process could be adversely affected by add-ins, zone settings, AD policies, virus scanning, etc. On one affected PC I disabled all all-ins to Internet Explorer and retried without success. After around half an hour of frustration, I decided to reset IE’s configuration as shown below.

After accepting the various warnings, I retried the test and it all worked! Metadata was now being properly saved. “Awesome”, I thought. I’m not quite sure what the true root cause is, but at least I know how to cure it.

Then it suddenly dawned on me that I’d been stupid. For all of my low level examination of the office 2003/RPC interaction with SharePoint, picking the brain of gurus like MCM Spence Harbar, I’d never thought to clear the temporary internet files cache. Doh!

On the next affected PC I did so, and the problem also went away instantly.

*Blush*

In my defence of my dumbness however, I had never examined the behind the scenes integration with Office 2003 and SharePoint, and it did not even twig that Internet Explorer would be involved in the picture. Given that this problem manifests itself within Office 2003 applications, it is not immediately obvious that your temporary internet files would cause an issue here – but now I know better .

And now I get it…

Now Office 2007 is not affected by this because it does not use this method at all. In 2007, the document information panel uses InfoPath to render metadata. At the time I naively thought that was a dumb idea because the document information panel cannot display custom column types. For example, let’s say you created a custom column type to hold a phone number with the correct formatting for country code and the like. In a SharePoint site you can display it fine, but in the Office 2007 document information panel, you will never see it. InfoPath only will work with the out of the box in column types.

My rationale was that if Office 2007 instead used JavaScript and HTML, then it should be able to display these sorts of custom columns. That may actually be true, but it is irrelevant. When you think of the sheer number of ways that the rendering of the code could be interfered with (Temporary Internet Files being one precedent), you can see why the Office team might have opted for the safety of InfoPath instead.

Now that I understand that this is how Office 2003 does it, I can plainly see that it leaves too much room for error.

So even though my problem was solved by a simple clearing of the cache, it still remains that things like a bad add-in or a bad AD policy setting could interfere with the Office 2003/SharePoint integration. So if you ever have this issue crop up, try the temporary internet files thing or reset IE’s configuration completely. It may save you a lot of time and pain troubleshooting.

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

<html dir="ltr">

<HEAD>

    <META Name="GENERATOR" Content="Microsoft SharePoint">

    <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">

    <META HTTP-EQUIV="Expires" content="0">

    <!-- _locID="id_PageTitle" _locComment="{StringCategory=TTL}" -->

    <Title ID=onetidTitle>File Properties</Title>

<Link REL="stylesheet" Type="text/css" HREF="/_layouts/1033/styles/core.css">

    <META Name="Microsoft Theme" Content="default">

    <META Name="Microsoft Border" Content="none">

<script src="/_layouts/1033/init.js"></script>

<script src="/_layouts/1033/core.js"></script>

<script src="/_layouts/1033/bform.js"></script>

</HEAD>

<Script ID="Form_Validate">

function Form_Validate(fVisible)

{

    if (frm.FValidate(fVisible))

        document.OWSForm.IsFormValid.value="true";

    else

        document.OWSForm.IsFormValid.value="false";

}

</Script>

<Script ID="Update_UI_From_Values">

var bFormFieldsInited = false;

function Update_UI_From_Values()

{

    frm.SetFirstFocus(bFormFieldsInited);

    bFormFieldsInited = true;

    frm.DataBind();

    DefaultControls();

}

</Script>

<Script ID="Update_Values_From_UI">

function Update_Values_From_UI()

{

    frm.SetFirstFocus(bFormFieldsInited);

    bFormFieldsInited = true;

    frm.FValidate(false);

}

</Script>

<script language="JavaScript">

L_tooltipfile_Text = "";

L_tooltipfolder_Text = "";

selectedElement = null

inChangeSelection = false

slElem = null;

oldSelection = "";

bIsFileDialogView = true;

function selectrow()

{

    if (slElem) {

        slElem.className=oldSelection;

        slElem.title="";

        }

    selectedElement = window.event.srcElement;

    while (selectedElement.tagName!="TR") {

           selectedElement=selectedElement.parentElement;

           }

    slElem = selectedElement;

    oldSelection=slElem.className;        

    slElem.className="ms-selected";

    if (slElem.getAttribute("fileattribute") == "file")

        slElem.title=L_tooltipfile_Text;

    else

        slElem.title=L_tooltipfolder_Text;

    document.selection.empty();

}

function checkScroll()

{

    if (document.body.scrollHeight > document.body.offsetHeight ||

        document.body.scrollWidth > document.body.offsetWidth)

       document.body.scroll="yes";

}

</script>

<BODY marginwidth=0 marginheight=0 onload="checkScroll()" onresize="checkScroll()" scroll=no>

<!-- Banner -->

<TABLE  CELLPADDING=0 CELLSPACING=0 BORDER=0 WIDTH="100%" >

  <TR>

   <TD COLSPAN=3 STYLE="width:100%">

    <TABLE class=ms-bannerframe CELLPADDING=0 CELLSPACING=0 BORDER=0 STYLE="width:100%">

     <TR>

      <TD VALIGN=BOTTOM WITDH=25>

      <font size=1>&nbsp;</font>

      </TD>

     </TR>

    </TABLE>

   </TD>

  </TR>

</table>

<table width=100% cellpadding=4 cellspacing=0>

 <tr>

  <td>

   <TABLE CELLPADDING=0 CELLSPACING=0 BORDER=0 WIDTH="100%" >

    <tr>

     <td>

      <table cellpadding=0 cellspacing=0 style="margin-left: 3px;margin-bottom:2px">

       <tr>

        <td nowrap class="ms-titlearea">Some User</td>

       </tr>

       <tr>

        <td ID=onetidPageTitle nowrap class="ms-pagetitle">My Committee</td>

       </tr>

      </table>

     </td>

    </tr>

    <tr>

     <td>

      <table border=0 width=100% cellpadding=0 cellspacing=0 style="margin-left: 3px;margin-bottom: 3px">

       <tr>

        <td class="ms-sectionline" height=1><img src="/_layouts/images/blank.gif"></td>

       </tr>

      </table>

     </td>

    </tr>

        <!-- Item Form -->

<SCRIPT>

var frm = new OWSForm("OWSForm", false, "http:\u002f\u002fintranetdev\u002fsomesite\u002f_layouts\u002f");

</SCRIPT>

<SCRIPT> frm.dopt.chDateSep = "\u002f"; frm.dopt.chTimeSep = ":"; frm.dopt.SetTimeFormat(0); frm.dopt.SetDateOrder(1); frm.dopt.SetDOW(0); frm.dopt.stAM = "AM"; frm.dopt.stPM = "PM"; frm.dopt.TimeMarkPosn = 0; frm.dopt.webTZOffsetMin = -480;  frm.nopt.chDigSep = ","; frm.nopt.chDecimal = "."; frm.nopt.chMinus = "-"; frm.nopt.iNegNumber = 1; frm.nopt.SetGrouping("3;0"); 

frm.stFieldPrefix = "urn:schemas-microsoft-com:office:office#";

frm.stImagesPath = "\u002f_layouts\u002fimages\u002f";

frm.wBaseType = 1;

</SCRIPT><Form Name="OWSForm" id=OWSForm EncType="multipart/form-data" Action="http://intranetdev/somesite/_vti_bin/owssvr.dll?CS=65001" Method=POST onSubmit="return false;">

<INPUT Type=Hidden Name="_charset_" Value="utf-8">

<INPUT ID=onetidCmd Type=Hidden Name="Cmd" Value="Save">

<INPUT ID=onetidIsFormValid type=hidden name="IsFormValid">

<INPUT ID=onetidFormWasPosted type=hidden name="FormWasPosted">

<INPUT ID="MustUpdateForm" type=hidden name="MustUpdateForm" value="true">

<INPUT type=hidden name="NextID" id="NextID" value="-1">

<INPUT type=hidden name="NextUsing" id="NextUsing" value="">

<SPAN id='part1'><INPUT ID=onetidIOHidden TYPE=HIDDEN NAME="List" VALUE="{EF494645-1D24-4761-A874-0CB866FA494C}">

<INPUT ID=onetidIOHidden TYPE=HIDDEN NAME="ID" VALUE="New">

<TABLE border=0 cellpadding=2>

<SCRIPT>var _g_tp_fNewForm = true;</SCRIPT>

<TR style="display:none"><TH nowrap valign=top class="ms-formlabel"><nobr>Comment<font color=red></font></nobr></TH><TD class="ms-formbody"><SCRIPT>fld = new NoteField(frm,"Comment1","Comment","");fld.stNumLines = "20";fld.IMEMode="";fld.BuildUI();</SCRIPT>&nbsp;<br><SPAN class="ms-formdescription"></SPAN></TD></TR><TR style="display:none"><TH nowrap valign=top class="ms-formlabel"><nobr>Document&nbsp;Status<font color=red></font></nobr></TH><TD class="ms-formbody"><SCRIPT>fld = new ChoiceField(frm,"Corro_x0020_Status","Document Status","Not Started"); fld.format = "Dropdown"; fld.AddChoice("Not Started", "");fld.AddChoice("No Action Required", "");fld.AddChoice("In Progress", "");fld.AddChoice("Completed", "");fld.AddChoice("Deferred", "");fld.AddChoice("Waiting on someone else", "");fld.IMEMode="";fld.BuildUI();</SCRIPT><SPAN class="ms-formdescription"></SPAN></TD></TR><TR style="display:none"><TH nowrap valign=top class="ms-formlabel"><nobr>Content&nbsp;Type<font color=red></font></nobr></TH><TD class="ms-formbody"><SCRIPT>fld = new ChoiceField(frm,"ContentType","Content Type","LSWA Document"); fld.format = "Dropdown"; fld.AddChoice("LSWA Document", "");fld.AddChoice("Folder", "");fld.IMEMode="";fld.BuildUI();</SCRIPT><SPAN class="ms-formdescription"></SPAN></TD></TR><TR style="display:none"><TH nowrap valign=top class="ms-formlabel"><nobr>Committee&nbsp;Document<font color=red></font></nobr></TH><TD class="ms-formbody"><SCRIPT>fld = new ChoiceField(frm,"Committee_x0020_Document","Committee Document",""); fld.format = "Dropdown"; fld.AddChoice("Agendas", "");fld.AddChoice("Minutes", "");fld.AddChoice("Actions", "");fld.AddChoice("Administration", "");fld.IMEMode="";fld.BuildUI();</SCRIPT><SPAN class="ms-formdescription"></SPAN></TD></TR><TR style="display:none"><TH nowrap valign=top class="ms-formlabel"><nobr>Document&nbsp;Type<font color=red></font></nobr></TH><TD class="ms-formbody"><SCRIPT>fld = new ChoiceField(frm,"Document_x0020_Types","Document Type",""); fld.format = "Dropdown"; fld.AddChoice("Incoming Correspondence", "");fld.AddChoice("Outgoing Correspondence", "");fld.AddChoice("Internal Document", "");fld.AddChoice("Fax", "");fld.AddChoice("E-mail", "");fld.AddChoice("Memo", "");fld.IMEMode="";fld.BuildUI();</SCRIPT><SPAN class="ms-formdescription"></SPAN></TD></TR><TR style="display:none"><TH nowrap valign=top class="ms-formlabel"><nobr>Year<font color=red></font></nobr></TH><TD class="ms-formbody"><SCRIPT>fld = new ChoiceField(frm,"Year","Year",""); fld.format = "Dropdown"; fld.AddChoice("2000", "");fld.AddChoice("2001", "");fld.AddChoice("2002", "");fld.AddChoice("2003", "");fld.AddChoice("2004", "");fld.AddChoice("2005", "");fld.AddChoice("2006", "");fld.AddChoice("2007", "");fld.AddChoice("2008", "");fld.AddChoice("2009", "");fld.AddChoice("2010", "");fld.IMEMode="";fld.BuildUI();</SCRIPT><SPAN class="ms-formdescription"></SPAN></TD></TR></TABLE>

<script type="text/javascript">

_g_tp_rgctNames = new Array;

</script><SCRIPT>var _tp_rgctfld = null;var _tp_ctfld = null;var _g_tp_rgcts = new Array;</SCRIPT>

<script type="text/javascript">

_g_tp_rgctNames.push("LSWA Document");

</script>

            <SCRIPT>

            _tp_rgctfld = new Array;

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="ContentType";

                _tp_ctfld.fRequired = BoolFromString("");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="SelectFilename";

                _tp_ctfld.fRequired = BoolFromString("");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="FileLeafRef";

                _tp_ctfld.fRequired = BoolFromString("TRUE");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Created";

                _tp_ctfld.fRequired = BoolFromString("");

                _tp_ctfld.fHidden = BoolFromString("TRUE");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Title";

                _tp_ctfld.fRequired = BoolFromString("FALSE");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("FALSE", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("TRUE", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Modified";

                _tp_ctfld.fRequired = BoolFromString("");

                _tp_ctfld.fHidden = BoolFromString("TRUE");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Modified_x0020_By";

                _tp_ctfld.fRequired = BoolFromString("");

                _tp_ctfld.fHidden = BoolFromString("FALSE");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Created_x0020_By";

                _tp_ctfld.fRequired = BoolFromString("");

                _tp_ctfld.fHidden = BoolFromString("FALSE");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Comment1";

                _tp_ctfld.fRequired = BoolFromString("");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Corro_x0020_Status";

                _tp_ctfld.fRequired = BoolFromString("");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Committee_x0020_Document";

                _tp_ctfld.fRequired = BoolFromString("FALSE");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Document_x0020_Types";

                _tp_ctfld.fRequired = BoolFromString("FALSE");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="Year";

                _tp_ctfld.fRequired = BoolFromString("FALSE");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("");

                _tp_ctfld.stDisplay ="";

                    _tp_rgctfld.push(_tp_ctfld);

                _tp_ctfld = new Object(null);

                _tp_ctfld.stName="MoveDocu";

                _tp_ctfld.fRequired = BoolFromString("FALSE");

                _tp_ctfld.fHidden = BoolFromString("");

                _tp_ctfld.fShowInNewForm = BoolFromString2("", true);

                _tp_ctfld.fShowInEditForm = BoolFromString2("", true);

                _tp_ctfld.fReadOnly = BoolFromString("TRUE");

                _tp_ctfld.stDisplay ="Move Document";

                    _tp_rgctfld.push(_tp_ctfld);

            _g_tp_rgcts.push(_tp_rgctfld);

            </SCRIPT>

<script type="text/javascript">

_g_tp_rgctNames.push("Folder");

</script>

            <SCRIPT>

            _tp_rgctfld = new Array;

            _g_tp_rgcts.push(_tp_rgctfld);

            </SCRIPT>

<script type="text/javascript">

function _FixMpCt2Flds()

{

    var frm = frmCurrent;

    var rgn1 = frm.rgctNames;

    var rgn2 = _g_tp_rgctNames;

    var rgctflds = _g_tp_rgcts;

    if (rgctflds.length < rgn1.length)

    {

        var rgnew = new Array;

        var i;

        var j = 0;

        for (i = 0; i < rgn1.length; i++)

        {

            var n1 = rgn1[i];

            var n2 = rgn2[j];

            if (n1 != n2)

            {

                rgnew.push(new Array);

            }

            else

            {

                rgnew.push(rgctflds[j]);

                j++;

            }

        }

        _g_tp_rgcts = rgnew;

    }

}

_FixMpCt2Flds();

</script>

</SPAN></form>

<SCRIPT>

</SCRIPT>

        <!-- FooterBanner closes the TD, TR, TABLE, BODY, And HTML regions opened above -->

 

</td></tr></table></PlaceHolder></TD></TR>

</TABLE>

</BODY>

</HTML>

                      

No Tags

I’ve been busy on a number of fronts and some of the fruits of that work will appear soon enough, but I thought that I would pop up to let you know about a forthcoming book written by Brett Lonsdale and Nick Swan on a SharePoint component that has until now, been seriously under-represented in the plethora of SharePoint books out there in the marketplace.

The Business Data Catalog is one of those SharePoint components that is easy enough to understand conceptually, but then will scare the utter crap out of you when you delve into the guts of its XML based complexity.  At least that was my experience the first time I toyed with it in early 2007. Luckily for me, my ass was saved by a tool that had just been released as a public beta called BDC MetaMan. I downloaded this tool and within around 15 minutes I used it to set up a BDC connection to Microsoft’s Systems Management Server v4 to pull software package details into a SharePoint list and felt very proud of myself indeed. 

Fast forward to mid 2009 and BDC MetaMan has come a hell of a long way, as have its creators. Nick and Brett are about as world-authoritative as you can possibly get on the BDC and if you wish to become a Jedi in the dark arts of the BDC “force” then you now have your official bible. This book is absolutely crammed with detail and the expertise of the authors in this feature shines throughout.

The book is split up across 11 chapters and although it is not explicitly stated by the authors, seems to be made of 3 broad parts. Chapter 1 introduces the BDC, how it is architected (web parts, BDC column, BDC Search, and integration with User Profile import and the SDK). Also covered is the range of data sources, an introduction to Application Definition Files (ADF) and how it all integrates into the Shared Service Provider model.

Once the intro chapter is done with, Brett and Nick don’t waste too much time in diving deep. 

Chapters 2 and 3 deal with the structure of BDC Application Definition (ADF) files, and follows up with the complex world of how authentication plays out with the BDC. Chapter 2 delves far more into the ADF files than I ever wished to tread, but Nick and Brett somehow manage to describe a long, boring XML file in a logical, easy to follow manner and there was a lot of stuff that I learned here that I had simply missed from trawling MSDN articles. The authentication chapter is covered in excellent detail in Chapter 3 and goes way beyond the usual NTLM/Kerberos double-hop stuff. Authentication in the Microsoft world has become very complex these days, and there are various options and trade-offs. This chapter covers all of this and more, brilliant stuff.

After the deep dive of ADF and authentication, we surface a little from the previous two chapters into what I think really, is part 2 of this book. That is, several chapters that deal with how you leverage the BDC once you have connected to a line of business application. Chapter 4 introduces the built-in web parts that come with the BDC, shows how they are used and how they can be modified either using SharePoint Designer or tweaking XSL styles directly. Chapter 5 explores the BDC column type, how it can be used in the Office document information panel, in SharePoint Designer workflows, as well as its limitations. Chapter 6 explains how to leverage the BDC for allowing SharePoint to crawl your back-end line of business data and present it in search results. In addition to this, chapter 6 has a lot to offer just from the point of view of customising the search experience, whether using BDC or not. Finally, Chapter 7 examines how the BDC can be utilised to add data into user profiles that is leveraged via audience targeting.

Next we dive back into “real programmer” territory and what I think makes part 3 of this book. Chapter 8 delves deep into the BDC object model, for those times when the out of the box stuff just won’t quite cut it for you. The example used to demonstrate this object model is a web service that exposes BDC data via several methods. Chapter 9 then covers the creation of a custom web part that is in effect, an Ajax version of the out of the box “Business Data List web part” that refreshes data every few seconds without requiring a page load. Chapter 10 is particularly interesting because it examines how the BDC is used in conjunction with another oft overlooked suite of technologies known as “Office Business Applications”. The combination of BDC and OBA offer many interesting capabilities and among the examples, there are examples of Excel and Word leveraging the BDC as well as creating custom task panes, custom ribbons and the like. Finally, chapter 11 deals with using the BDC to write data back to the line of business applications and finishes with a great example of using InfoPath to submit data to a line of business application via a webservice that calls the BDC. That is hellishly cool in a nerdy developer kind of a way.

Phew! First up, *man* these guys are smart! I have to say this is the hardest SharePoint book that I have reviewed. It is obviously aimed at developers but it has so much to offer beyond the BDC. The content is very technical at times and obviously low-level. That, itself, is not the problem. Conversely, complex topics are handled really well and everything is extremely logically organised and flows well. The book is simply very, very comprehensive! There is plenty of meat for developers to sink their teeth into and this book will keep you going for a long time.

The preface of the book states that it has been written for an audience of “Microsoft SharePoint 2007 Information Workers and Developers who need to learn how to use, customize and create solutions using the Business Data Catalog”. I would agree with this, but I hope that information workers do not get put off by chapter 2 (and to some extent, chapter 3). This book dives deep straight off the bat and it is actually the middle chapters that offer the sort of insights that information workers will find the most useful.

So, if you think that the BDC deserves more than one single chapter towards the back of a SharePoint book, then this is your answer. As well as becoming an expert on the BDC, It will open your eyes to many possibilities beyond it.

Thanks for reading

Paul Culmsee

                      

No Tags

Hi all

If you were going to slot me into a little stereotype box, then you would slot me into the “IT pro” side of the fence. My coding is okay, but my real vein of expertise lay in infrastructure and over my career, I developed what I think is a reasonable troubleshooting instinct.

I’ve also worked with developers for the whole of my career and have the scars to prove it. The thing about developers is that they have this in-built reflex that until yesterday I did not have a word for. Then it came to me.

All developers have a little Russell Crowe inside of them!

Why do I think this? Am I suggesting that developers are handsome, rugged types who melt your heart with their piercing eyes?  Oh, please. Allow me to explain with a simple mythical conversation with Mr Crowe. Let’s pretend you are a movie director.

You: “Hey Russell, we need to do another take, your dialogue wasn’t quite right.”

Russell: “Yes it was.”

You: “No seriously, I think if you had a look you’ll find that you missed a word or two.”

Russell: “Completely impossible. You are obviously an amateur and have no idea about acting.”

You: “I’ve directed twenty films and…”

BAM!!!  (Flying telephone hits you in the head at high speed, knocking you unconscious.)

Russell: (2 days later). Okay, so there was a minor issue with my dialogue, but the script was bad to begin with”

This exchange is somewhat representative of how programmers can occasionally be when it comes to troubleshooting. I remember one case where I was the “Cisco guy” who had problems with a developer who was so utterly fixated on “the network” being the cause of problems with his media streaming application. This created the classic “dev vs infrastructure guy” showdown, which we all know is usually won by the person whose home turf the battle is fought on. Therefore, the developer blaming “the network” and then going up against the “Cisco guy” is like Microsoft trying to win search market share off Google. The battle is so one sided it’s almost cruel to participate – but you feel it is your duty to put the little upstarts in their place anyway.

I have won the majority of such battles, not because I am any good, but because the developers have thrown the metaphorical phone at me before I’ve finished asking them if they would like a coffee. As a result of their inner Russell Crowe hurling the phone so quickly, their aim is way off, and the phone usually misses me, bounces off the wall and takes out their boss or some other authoritative figure.

So, they cop some heat and sulk in the corner for awhile, but do developers learn from this? Hell, no! The reason why this is so, is because little Russell doesn’t like to lose, and when he re-emerges, he causes temporary amnesia of all previous battles. Of course, his opponent remembers all, and the next battle is even more cruel that the previous one and the outcome is assured.

So, yesterday, my friend and colleague did his first “Russell Crowe” for some time. He hit a problem, and misinterpreted the cause and went down a path that led him to a very tunnel-vision view of what was wrong and what the solution was. He described the problem he was having to me and it didn’t feel “right”, but he was pretty insistent he was on the right path. So, I asked Twitter and got back a couple of suggestions and as soon as I put one to him… BAM!! Russell Crowe appeared and threw a phone at me.

“Well, they are obviously amateurs and haven’t a clue about the SharePoint SDK” was the gist of the response.

One of the respondents was Bjorn Furuknap, who I can assure you is *not* an amateur .

Anyway a few minutes later we found a different way to troubleshoot which pretty quickly pinpointed the problem. My colleague was very contrite and good natured about it as I teased him mercilessly. I later mentioned to Bjorn that I had just dodged a metaphorical flying phone and he said this wonderful quote which I think sums it up.

“Of course. He’s a developer. We’re all like that. It is always some else’s fault!”

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

                      

No Tags