It’s email integration captain, but not as we know it (problems with incoming email handling on SharePoint 2010)

Send to Kindle

Hi everyone. This is my last post for 2010, and I am going out on a troubleshooting note. See you all next year with lots of new content and cool stuff!

I had some interesting experiences recently with SharePoint 2010, specifically the Content Organiser feature and leveraging it with incoming email. I thought they might be worth sharing, but first I need to set some context via the use-case where this started.

Where would we be without the photocopier?

image

Most organisations large or small, have one of those multifunction photocopier/scanner/fax/coffee maker gizmos (okay so maybe not the coffee maker). You know the types – they are large, noisy and the paper feeders frequently jam, where the tech guy who comes to fix it has to be on site so often that he’s considered a staff member. They usually have a document feeder, can scan to PDF and email it straight through to you. If you have a really fancy-schmancy one, it might even OCR the content for you so the resultant PDF is not an image but text based.

While all that is good, the real benefit of these devices is more subtle. Employees like to congregate nearby for a little bit of office gossip, and to quietly bitch to each-other about how much their boss or co-workers annoy them. The conversations around the photocopier are usually some of the most insightful and valuable conversations you might have at times.

So I believe that anything we can do to encourage photocopier conversations is a good thing. For some organisations, this is about as close as it gets to cross departmental collaboration! Smile with tongue out. If we also leverage the fact that these devices offer this “scan to PDF and email” function at the press of a button, then SharePoint has a nice story to tell here – especially with SharePoint 2010 and the Content Organiser feature.

The premise: Content Organiser coolness

I will spend a few moments to introduce the Content Organiser feature for readers who have not seen much of SharePoint 2010. If you know all about this feature, skip to the next section.

For those of you who may not be aware, SharePoint 2010 has an interesting new feature called the Content Organiser. The Content Organizer feature is quite a powerful document routing solution that makes it easier to store documents consistently, according to administrator defined rules that can copy or move a document from one place in SharePoint to another place. I will get to the rules in a minute, but the content organiser feature is important for several reasons;

  • It makes the saving of documents easier because users do not necessarily have to worry about knowing the destination when uploading new content.
  • It is a highly flexible method for routing documents between sites and site collections around the SharePoint farm.
  • It underpins a solid compliance and records management file plan capability.

So before anything else can be done, we need to turn it on. The Content Organiser feature has to be activated on each site for the functionality to be enabled. In other words, it is a site scoped feature. Below is an illustration of the feature to activate.

image

Once the Content Organiser feature is activated, SharePoint 2010 makes several changes to the site configuration.

  • It creates a new document library called the Drop-off Library.
  • It creates a new custom list called Content Organizer Rules.
  • It adds two new Site Administration links in the Site Settings page to manage the Content Organiser for the site.

image

The description for the feature says “Create metadata based rules…” and it these Content Organiser rules that allow you to automatically route documents from the newly created Drop-off Library to some other location. It is important to know that the Drop-off Library is fixed – it is the first point of call for files that need to be moved or copied somewhere. Consider the drop-off library like a bellboy of a hotel. You give him your luggage and he will ensure it gets to its correct location (and unlike a bellboy you don’t need to tip).

So if the drop-off library is the starting point, where can documents be routed to? The location can be;

  • A document library and/or a folder within a document library on the site.
  • The Drop-off library of another site, which allows inter and intra site collection routing.

The Content Organiser rules are managed from Site Administration which is accessed via Site Settings. New rules are added in the same manner as adding new items to any SharePoint list. In the following example, we will create a rule where a content types of invoice will be routed to a document library called Finance.

image

The conditions section of the rule allows for multiple conditions to be defined to determine matching content and then, where that content should be routed to. The properties available are any columns assigned to the content type being routed. In the example below, we have added two conditions that have to be satisfied before the rule will fire.

image

Let’s take a closer look at this beast known as the Drop-off library. This is a special document library is added to the site upon feature activation, imaginatively called the Drop Off Library. As stated earlier, this library is really a temporary staging area for items that do not have all required metadata to satisfy any routing rules.

The sequence of events for the Content Organiser is;

  1. Documents with the correct content type, metadata, and matching rules are automatically routed to the final library and folder.
  2. Documents that lack the amount of metadata required to match a rule or that are missing required metadata are retained in the "Drop-Off Library" so that the user can enter metadata to satisfy a rule.
  3. After a user has edited a staging document with the appropriate metadata required to match a rule, the document is automatically routed to the target library and folder.

As an example, if we assume that a Content Organiser role will route any document with “Finance” in its name to document library called Finance, the behaviour will be as follows:

  • If the file uploaded has the word “finance” in its name, SharePoint indicates that the document has been successfully routed
  • If the file uploaded does not have the word “finance” in its name, SharePoint will indicate to the user that the content organiser has placed it into the Drop-off library.

Now, before you rush off and start to mess with the content organiser, I’d better tell you about a couple of caveats

  1. The Content Organizer will only work on content types that are, or derive from, the Document content type. So it does not work for automatically organizing large lists.
  2. When uploading documents via Windows explorer view, Content Organiser rules are ignored and the document will not be redirected to the Drop-off library. (through the browser if a document is uploaded to a destination library, SharePoint will move it to the Drop-off library for classification)
  3. There is a limit of six conditions per rule. After six conditions are added, the "Add new condition" link disappears.
  4. If you wish to route the document to another site, the Content Organizer feature has to be installed on that site for the Drop-off library to be created as that is the destination. Additionally, you need to add the configuration information in Central Administration by adding the destination to the list of send to connections for the web application (that is beyond the scope of this article, but easy enough to do).
  5. Ruven Gotz tells me there are also some potential risks around the fact that you can route files to any destination even if you do not have permission and potentially overwrite content. As Scott says, the content organizer will move content to the new location whether or not the contributing user has access to the destination location.

Content Organiser and email integration

So if we go back to where we started with our fancy photocopier. SharePoint has offered email integration on document libraries since the 2007 version. In effect, we give each document library or list an email address, set up a few parameters and any attachment will be in effect, uploaded to a document library.

On a semi-related note, my company actually developed a version of the content organiser for SharePoint 2007 that allowed the routing of documents based on business rules. Contact us if you want to now more.

Given the routing capabilities of the content organiser in SharePoint 2010, one would think that by email-enabling the Drop-off library that is created when you activate the content organiser feature, that we can have all scanned correspondence end up in the Drop-off library, ready for classification by an administrator and routed in accordance to specified routing rules.

Sounds logical enough – so logical in fact that I gave it a try.

Problem 1: Race condition?

Email enabling a document library is pretty easy, provided you have set up incoming email in SharePoint Central Administration first. In this case, I set up the incoming email on the Drop-off Library with the settings below: Note that I specified not to overwrite attachments with the same name.

image

I then programmed the photocopier to use this email address as a profile. That way, a user would scan incoming correspondence, then choose this profile as the destination. The photocopier would scan to PDF and then email those PDF’s to the Drop-off library. The problem was – not all of the scans arrived.

We noticed that individual scans (ie one document at a time), would work fine, but for some reason, bulk scans would not. Typically, if a user scanned say, 10 items, only 6 of them would make it to the document library. A trawl through the diagnostic logs was therefore required. Luckily, SharePoint 2010 has been built upon PowerShell, and there is a PowerShell command to get at the diagnostic logs. I have become a huge fan of PowerShell just from this one command, as it has eliminated the need for me to install additional tools to view logs on SP2010 boxes. Taking a punt, I assumed if there was to be an error message, it would have the word “E-mail” in it. So I issued the following PowerShell command:

Get-SPLogEvent | Where-Object ( $_.message -like "*E-Mail*" ) | Out-GridView

This will return any logs in a graphical format (the gridview) as shown below. Immediately I saw warning messages, telling that an error occurred while attempting to create an attachment for an item sent via email.

image

This was clearly related to my issue, so I adjusted the PowerShell script to be a little more specific so I can see the full message

Get-SPLogEvent | Where-Object { $_.message -like "*create an att*" } | Select-Object -Property message | Out-GridView

The email was sent to the list “Drop Off Library”, and the error was: the file DropOffLibrary/<filename> has been modified by SHAREPOINT\SYSTEM on <date>

image

Problem 2: A poorly named feature?

This time, I saw that it claims that the attachment was modified by SHAREPOINT/system. Hmm – that sort of error message is very similar to race conditions seen with SharePoint Designer workflows. Thus, my first thought was that email enabling the Drop-off library was possibly unsupported. I figured that the Drop-off library likely used event-receivers, workflow or scheduled tasks to do the document routing that might get in the way with processing incoming email attachments.

My suspicion was further given weight when I recalled that there was another feature to do with content organiser that I could activate: Email Integration with Content Organiser. According to the description, it enable a site’s content organizer to accept and organize email messages. Cool – this seemed logical enough, so I activated it.

image

Upon activating this feature, a small change is made to the Content Organiser Settings page, found under Site Settings. An e-mail address was assigned, and a link was provided to configure the organizers incoming email  settings as shown below:

image

This is where things started to get interesting. As expected, I was taken to the incoming email settings screen, but instead of it being the drop-off library, it was a hidden list called Submitted E-mail Records. At the time, this suggested that my initial conclusion that incoming email on the drop-off library was unsupported was correct. After all, why else would incoming email be redirected to another list instead of the drop-off library? I couldn’t think of any other logical explanation.

image

At the time, I searched the web to see if anybody else had mentioned the Submitted E-mail Records hidden list and problems with incoming emails. I was then surprised to learn that this hidden list was there in SharePoint 2007, but I’d never seen it before because Records Management in 2007 is crappy and I never used it much.

Anyway, try as I might, I could never get items emailed to the Submitted E-mail Records list, to ever route to the Drop-off library. The photocopier would happily mail items to Submitted E-mail Records list, but they would stay there and never get processed. Grrr!

After turning up some debug logs to verbose, specifically:

  • Document management Server: Content organiser
  • SharePoint foundation: E-Mail

We saw the following error message: “Cannot resolve mailbox (null) to a valid user”. Additionally, my colleague Peter Chow used reflector to examine the underlying code behind this feature. From what we could tell, a method, GetOfficialFilePropsFromBody, in the class: Microsoft.Office.RecordsManagement.RecordsRepository.EmailRecordsHandler, was attempting to extract a mailbox value and getting a null.  Unfortunately for us, the code eventually lead to obfuscated classes so we were not able to dig any deeper.

At this point, with nothing on the net to guide us and twitter going quiet, I logged a support call with Microsoft. Pretty quickly, our issue was reproduced and escalated to the local team and then to the US. A few weeks later, we got the following answer:

The E-mail Integration with Content Organizer feature is a there for legacy Exchange 2007 deployments that can journal mail to SharePoint.  Such functionality is not available in Exchange 2010 and we do not (and never have) supported directly e-mailing a content organizer as the steps below show.  It just won’t work – that mail submitted to that list needs to be in a special format that Exchange 2007 can create (or some other system who has implemented that documented protocol). The users should consider leaving the mail in Exchange and using Exchange’s record management and compliance functionality or buying a third party add-in if they need to get mail into a SharePoint content library. http://technet.microsoft.com/en-us/sharepoint/ff598594.aspx has a list of partners.

Workaround : For this  scenario we found a workaround to enable incoming email on the “Drop Off library” and which in turn routed the emails/attachments to the destination library as per the rule. The envelope data or other metadata will NOT be carried along with the item to its final location with this workaround. So the customer should expect to lose sender, sent, received, et cetera

If the customer wants to check on how to get this working using Exchange 2007: http://msexchangeteam.com/archive/2006/11/21/431608.aspx

So the format is generated by the backend transport layer via a policy on a Managed Folder.  So even in 2007, you can’t email a content organizer.  But you can set up a journaling rule on a managed folder in Exchange to journal every item dropped into that folder to SharePoint.

We should not encourage users to email Content organizer since we have that has a legacy feature to support Exchange 2007. If the users wants email to be sent to SharePoint then use Email enabled Doc Lib.

So there you have it. Do not activate the Email Integration with Content Organiser feature!. It is a legacy designed specifically for Exchange 2007, yet the description for the feature in SharePoint makes no mention of this! If this was explicitly mentioned in the description of this feature, two weeks worth of needless troubleshooting and a long support call would have been avoided. Even my local (and excellent) Microsoft escalation team were not aware of this. Oh well, you live and learn.

Back to square one. (shut-up and apply the latest cumulative update)

So now that we confirmed that the Email Integration with Content Organiser feature was a giant red herring and never going to fly, we returned focus to why some attachments were not being correctly processed by the Drop-off Library. As it happened, the August cumulative update for SharePoint 2010 had a fix in it. This technet thread describes the issue in more detail. http://social.technet.microsoft.com/Forums/en-US/sharepoint2010setup/thread/896cbe2a-94c6-4a0f-9e4f-ea6b0fe3cc45/. There appeared to be a bug where even if you set the email-enable feature option called “Overwrite files with the same name”, to no, it was ignored and an exception was logged when attachments with the same file-name arrived. Turns out the photocopier file names were not 100% unique! It used a timestamp as part of the filename that was unique down to the minute, not the second.

So after a journey that included a needless support call, we applied the CU and what do you know, problem solved!

To conclude this rather long and rambling post, I feel kind of bad that I got so side tracked on the Email Integration with Content Organiser feature. That’s part of troubleshooting life I guess. The only consolation I can really take from it, is that it also fooled Microsoft engineers too. I guess this issue is yet one more of the many caveats that we all have to learn about the hard way.

 

Thanks for reading

Paul Culmsee

www.sevensigma.com.au

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle

SharePoint Analysts–Stop analysing!

Send to Kindle

Michal Pisarek wrote a nice write-up of what makes a good SharePoint Analyst. I feel I have something to offer here, given that I…

  1. am a cynical old bastard
  2. am an opinionated old bastard
  3. have had some opportunities that not too many SharePoint people have had
  4. wrote this post on a plane while suffering jetlag 🙂

I am going to argue that as a SharePoint ”analyst”, the worst thing you can do is act like an analyst.

I previously wrote about the identity crisis that Business Analysts have which became apparent to me when I spoke at a BA World Conference last year. The gist of my point in that post came from my observations of a panel discussion on role of a BA within Agile development. I noted that the whole discussion seemed based on an underlying presumption that the role of the BA was to translate between IT and “the business". Agile by its very nature, shifts the ground out from under this assumption which caused a bit of consternation on the part of the panel. My takeaway from this, was that a role does not define the person. The body of knowledge of your discipline (in this case BABOK) is not the one truth. When you think about it, the body of knowledge by definition, consists of the lowest common denominators of a discipline of knowledge – a starting point. This is because the skills that you have built are your skills. Sure, you can write about them, but that only conveys a small dimension of what your skills entail.

Essentially, if everybody followed the BOK as the source of the truth then all consultancies would look the same. (Thinking about it, now you know why all large and expensive management consulting firms tend to sound the same…hehehe 🙂 )

image

The point is that it is the knowledge that you have earned that makes you what you are and what customers want. Knowledge comes from making mistakes, not by confirming your rightness. Therefore, people who practise constant learning by trying things out, challenging their models of reality, always build a hugely valuable corpus of tacit knowledge that is locked up in their brain where knowledge is connected and related in ways that form insight. It is because of this unique insight, that customers rarely want a “Business Analyst”.  They want Joe or Bob because of what Joe or Bob, as individuals, bring to the table. (Kailash writes about this in more detail if I am leaving you scratching your head at this point. I am going to quote him in a reply to Richard Harbridge).

The point I would like to emphasise, however, is that best practices cannot capture tacit knowledge. Codified best practices (or standards) are therefore necessarily incomplete. One of the things standards don’t tell you is how you should “practice the practice”. Practices have to be customised or adapted to specific organisational contexts before they can be practiced. The process of customisation invariably involves the creation of tacit knowledge to fill in the missing bits. This is why I used the word “rediscover” to describe this process, rather than “customise” (or “adapt”) : the former gives a better sense for the magnitude and type of effort involved whereas the latter suggests that minor tinkering will do the job (which it won’t).

So, here is a tip. Don’t define your own wellbeing by a role title or adherence to a body of knowledge because, if you do, you will inevitable have an identity crisis. The world simply moves on and with it, the problems that people are trying to solve tend to wriggle out of the domain of any single BOK. Any attempts to resist this will eventually become dogma and then we have “memetic smackdowns” as I describe in this EUSP Post.

image

Okay, so now that I have finished a mini rant, I’d like to address the whole “analyst” thing. Don’t get me wrong – it’s not the title of the role I am going to talk about here, but the analyst paradigm itself. That is, the notion that my purpose in life is to take some requirements, go to a back room, spend time “processing” and “synthesising” those requirements, translating away between "IT and “the business”, before formulating a solution for the customer.

This does not work well with most SharePoint projects! Especially ones that seek to “improve collaboration”.

Many people fail to see why this is the case, yet given the huge gap between the SharePoint that Microsoft shows you versus what you see on the ground, you would think that it would be kind of obvious. For what it’s worth, understanding why the analyst paradigm is dangerous also neatly explains why IT is generically predisposed to struggling with SharePoint.

Consider the following projects:

  • Implementing Microsoft Exchange for a global organisation
  • Implementing Active Directory for unified and policy based user and resource management
  • Replacing PBX with a Voice over IP system
  • Upgrading your wide area network from site to site VPN’s to MPLS managed WAN
  • Determining support and escalation procedures for a SharePoint deployment
  • Replacing a Windows XP/Office 2003 desktop environment with Windows 7/Office 2010

In the above list, if you ask the question “Will users accept and adopt this?” the answer is a fairly clear yes. At the end of the day, no matter what version of Exchange is in, Microsoft Outlook is still Microsoft Outlook. What the user sees is an Inbox and so long as mail comes in and goes out – they are happy. All Active Directory is to a user is a username and password to get to their computer. Irrespective of how clever the routing is in a Voice over IP system or a WAN network, a phone is just a phone. You can plan for support and escalation procedures for SharePoint, but at the end of the day, people will use the SharePoint system because it is the right solution for them. Sure they might get pissed if they have bad technical support, but if the solution is crap then no amount of awesome support structure is going to change this.

This, my friends, is the realm of the analyst paradigm!

You see, if you ask me to deploy Active Directory or Exchange, I will ask you a bunch of questions about your number of sites, number of users, communications infrastructure and the like, and I know that users will use it. I actually don’t need to talk to them. Oh sure, someone might do some communications planning, but I don’t need to address user adoption because adoption is already there! People adapted to email, telephone and Microsoft Windows years ago.

Instead, I can go into a back room, read a bunch of whitepapers and design/deployment guides and blammo. Here is your solution and here is how much I think it will cost. Better still, I can use waterfall style of project methodology because as a specialist, I have done this before and have expertise. I can tell you what needs to be done in considerable detail and break it down into a plan.

What it all boils down to is that there is a clear relationship between cause and effect, characterised by answering YES to the “If I do this, will users accept and adopt this” test. But most IT departments (and most IT integrators) are going to default to the analyst paradigm because many of their other projects tend to be in the above category. This analysis based mode of project delivery is very much ingrained into the reptile brain of a lot of IT professionals through years of repetition of implementing these types of projects. After all, it’s just IT, right?

Also, note that many of these type of projects are characterised by being very technically complicated. Some are insanely so and require specialist technical expertise. This is the realm of the supremely skilled person who performs an analyst function with a body of knowledge behind them. The whole industry certification process is built on this fundamental underpinning and works really well with things like Cisco and Microsoft in areas like Exchange, Active Directory and the like.

Now, consider the following projects or initiatives (I am deliberately picking things where people may disagree with me so go with me, ok).

  • Coming up with an navigational structure for a SharePoint collaborative portal
  • Installing SharePoint to improve collaboration
  • Replacing a folder based file share with a metadata based document repository
  • Designing a new road layout for a local suburb
  • Putting in a new intranet
  • Putting in a records management system
  • Installing a new time tracking system

In all of these problems, it is difficult to answer yes to the “Will users accept and adopt this” question. Sure, you can go all big stick and say “We will force them to” but that fails the “accept” part of the test. Therein lies the danger for the analyst paradigm. You can go off to the “back room” and use your body of knowledge, read best practices and “analyse” until the cows come home. Until you deliver the solution, you will not know if it will be accepted.  The cause and effect relationship is not clear until after an action has been taken!

image

The reason you can’t confidently answer yes is that all of these projects above require adaption on the part of the target audience. Adaption leading to adoption is a hit and miss affair. While we might like to think that we are all rational, clearly we are not. If you want rational, think Dr Spock from Star Trek – and even he got mad sometimes! Office politics and organisational inertia stems from the irrational world. Butt covering by positioning for “blame avoidance” for decisions made by fear and anxiety are commonplace. If you work in a large organisation and listen carefully to a typical meeting, the logic and facts that are spoken out loud are rarely what matters compared to the complex non spoken interplays that go on underneath the words.

For what it’s worth, I added a non IT example of designing a new road layout for a suburb. A rational analyst might think that residents have to accept the solution by definition because, after all, once a road is in, that’s it. But what typically happens is that residents of that neighbourhood see the plan, petition, form lobby groups and harass the hell out of their local political representatives. If this is enough to make the politician edgy then they will vote the plan down before it ever happens. Sure it might have been a good plan, but it’s gone now and the chances of further buy-in are greatly reduced.

The pure analyst is often taking a rational approach to an irrational problem space. Sorry folks – this is simply asking for trouble. Like the politician who does what they think will keep them in power versus what is logical shows the rational world does not always get a good look in. A good example in SharePoint land, is the “metadata is good and folders are bad” thing. Any metadata fanboy with this mantra often find the hidden organisation will beg to differ 🙂

So, what do we do then?

image

The facilitator paradigm instead aims to elicit resolution of problems using dialogue between stakeholders by achieving a greater degree of shared perception of the problem situation. Unlike the analyst paradigm, there is no back-room approach as such. By definition, we need to collaborate to do this. (Fancy that, eh? Collaborating to deliver a collaborative platform – who would have thought!)

Now, I am not talking about facilitation in the classic sense with everybody sitting in a room in a circle and plays team building games. Some of the best project managers and business analysts I have met are facilitators without necessarily knowing it. What I mean by facilitation is that our starting point is to leverage the wisdom of the crowd, by creating an environment conducive to participants being able to surface the irrational as well as the rational. Only in this way, can we come to a shared understanding of a problem and what should be done about it.

I previously termed this the holding environment, and it really is. A great business analyst or project manager knows this instinctively, and uses many tools (as well as some coercion and sometimes their own ego surrendering) to bring it about.

If you take anything away from this post, remember this. Anytime you cannot confidently answer the “Will users accept and adopt this” question, it is highly likely that not all users see that there is even a problem yet. Therefore, expecting people to magically buy-in and adapt when they don’t recognise the problem is never going to fly. Like trying to get a Darwinist to accept that intelligent design should be taught in schools, someone who does not believe in it to begin with, is not going to easily buy into a solution that requires them to change their beliefs and therefore behaviour. 

Conclusion

When you think about it (rationally!) we usually look at SharePoint as an enabling technology that can address a legacy of poor collaboration and information management. Yet, how can the world of the analyst work out if their solution will create the very same legacy without all stakeholders being on the same page?

So remember, in a project where the cause and effect relationship is not clear, use the facilitator paradigm and stop being such a bloody analyst!

Thanks for reading. Comments most welcomed Smile.

Paul Culmsee

www.sevensigma.com.au

 

In 2011, I will be posting sections of my Governance and Information Master Class here. Much of the content consists of mental models, alternative frames of reference, pattern and practices as well as other tools and methods, specifically for the facilitator paradigm. This is my interest area and feel the analyst paradigm is already well represented in the SharePoint space.

 Digg  Facebook  StumbleUpon  Technorati  Deli.cio.us  Slashdot  Twitter  Sphinn  Mixx  Google  DZone 

No Tags

Send to Kindle