MemorableURL.com

Accessing Content from an iPad while I’m on the road

I know that this is off topic but I wanted to share it…

I love the iPad; I have worked out how to upload, edit and even project my PowerPoint presentations, I can access my Exchange Inbox and Calendar, take notes (I use NotesNMore which rocks) and edit Word documents. In fact, I am close to being able to travel to customer sites without my heavy old laptop. However, here's the rub, my laptop is the machine that I create all of my content on and naturally it is where all of my content resides.

If you want to get a file onto the iPad then you need to either upload it into a specific application using iTunes, email it to yourself and 'Open In’ or use an intermediary service like box.net (which our corporate policy expressly forbids).

I was getting into the habit of uploading the latest and greatest version of the PowerPoint presentations that I expected to use on to the iPad before I headed out the door. Alas, I still lug my laptop along with me because I live in fear that I'll be asked to present on a topic that I was not expecting and hence had not uploaded the right files to the iPad.

I have found a really nice way around this dilemma…

Quick disclaimer - I work for EMC, EMC owns Iomega, Iomega makes the iConnect that I use here. The only relevance that this has in real life is that I would not have bought the iConnect without the staff discount! They didn't give it to me or pay me to write this - that said, after the fact I'd be happy to take delivery of a new ScreenPlay Director HD MediaDirector as a thank you!!

High-level Overview

The bits and pieces

• I use an Iomega iConnect at home. This is a small device that you plug your USB external drives into and it makes the content on them available to any device on your network and also available when you are on the road via a secure web site
• I use Iomega's QuikProtect application (included with the iConnect) which I have configured to synchronize all the files on my Desktop and in My Documents to a folder on one of my iConnect drives. It does this in real time and stores 5 versions of each file.

How it Works

QuikProtect makes copies of the files on my laptop whenever it is docked in the office. Using their secure web site I can access any of those files from a browser. If I connect to the site from the iPad I can download files and then open them in the appropriate application on the iPad.

The Process (without screenshots)

• I start up Safari and browse to the secure MyIomega web site
• I navigate to the folder with my content and touch the file
• The file opens in Safari and the 'Open In…' buttons are highlighted
• I open the file in Keynote (or whichever application is appropriate) where it is converted and I can then edit and project it using the VGA output adaptor.

Known issues

• Using Safari on the iPad the scroll bars do not appear if the list of items is too long. You can use search to find the file if necessary but that’s not ideal.
  • This will be especially painful given that I have QuikProtect configured to save five versions of a document. I've logged this as a bug with Iomega.
• It would be nice if the navigation was in a tree view just to make it easier to find things.

The Process (with Screenshots)

• I start up Safari and browse to the secure MyIomega web site

 

• I search for/navigate to the folder with my content and touch the file

 

• The file opens in Safari and the 'Open In' buttons are highlighted

 

 

• I open the file in Keynote (or whichever application is appropriate) where it is converted and I can then edit and project it using the VGA output adaptor.

 

SharePoint, Documentum or a Hybrid Approach? Capgemini's Answer.

I attended a Capgemini session today at EMC World that looked at when to use SharePoint, when to use Documentum and when to consider a hybrid solution. Jon Ludwig, the speaker, did a really good job of walking through the strengths and weaknesses of each solution, the available options and how to combine the solutions to get a best of breed solution.

Jon opened with a slide that pretty much summed up the key differences in the two company’s approaches, (included with his permission). On the left he had SharePoint which started with a UI focus and worked down to a supported repository. On the right he had Documentum starting with a repository focus and moving down to the UI – he pointed out that the original version of Documentum didn’t even have a UI. He described Documentum’s UIs are being ‘power user’ focused which is probably a fair representation.

 
He then used the slide below to show the functionality in SharePoint 2007, SharePoint 2010 and Documentum from a UI perspective.

This was interesting in itself but when you look at it next to the same comparison from a repository functionality view you really get to the meat of Jon’s key points. SharePoint is certainly making huge gains in the space of UI functionality but there’s still plenty of room at the back-end for add-ons from traditional ECM vendors.

One interesting side note was that Jon suggested that as soon as SharePoint adds specific functionality to SharePoint it then becomes a commodity which other vendors move away from. He quoted the fact that when SharePoint added enhanced WCM to SharePoint in 2010, Documentum partnered with Fatwire to move out of the WCM area and into ‘Web Experience Management’.  I’d suggest that this is the only the case when SharePoint gets close to an acceptable level of functionality – take workflow for example, I don’t see vendors moving away from real workflow/BPM/BPA…quite the contrary.

Jon danced around the issues of SharePoint’s ability to scale; he stayed on the fairly safe ground by talking about the fact that you would not want to ingest tens of thousands of images, heterogeneous data types or large objects into SharePoint.

Records Management

Jon spoke at some length about SharePoint 2010’s RM capabilities. He pointed out that SharePoint 2010 has most of the bits that you need to do records management but that SharePoint assumes that everything that is a record can be moved into SharePoint. This assumes that SharePoint can handle the scale and the types of objects that would need to flow in. He pointed out that SharePoint was not designed to be an enterprise scale repository’ in fact he specifically says “Microsoft will likely never offer full repository functionality; it’s not their focus.”

He went on to talk about federated RM as a potential solution.  With federated RM you can leave SharePoint content in SharePoint, leave file system content on the file system and manage it all using your file plan in Documentum. His story here was that you could get content into Documentum when it makes sense and the federate content that is not there. Documentum owns the file plan and the records management features but the content can stay in SharePoint (or elsewhere).

When talking about acting as a trusted advisor he made one interesting comment, he said that customers should not just do a feature/function comparison between SharePoint and the traditional ECM system and act on that alone, they need to also consider what it would cost to move everything into SharePoint. He also advised that he liked federated because he felt that you should give your records admins just one tool to use, keep it simple.

Jon was definitely a fan of using a hybrid solution for RM, not picking either one or the other solutions.

In general I think that Jon really grasped the issue at hand and had real, solid and pragmatic advice for customers. He did not bash SharePoint or Documentum, he pointed out the weaknesses and the strengths of both systems and seemed to focus on why the right solution at the right time would give you more than the sum of the parts.  

Jon can be contacted at [email protected].

The Evolution of Content Externalization

It is EMC World this week and as usual we are announcing a multitude of new/updated products. Two of them are relevant to my usual subject matter and thinking about them raised one interesting observation about the commoditization of certain aspects of data externalization from SharePoint.

The two products are ‘EMC SourceOne for Microsoft SharePoint’ and the next release of ‘Documentum Repository Services for SharePoint’. The former product is a product aimed at removing content from SharePoint and placing it in an archive; for operational reasons and for long-term preservation of SharePoint content that is no longer accessed actively from SharePoint. The latter I’ve mentioned before but it is basically a way of connecting SharePoint to Documentum in a way that is invisible to the SharePoint end user.

I’m not known for gratuitously marketing my employer’s products so you might wonder why I’m bringing this up in my Blog. Almost exactly a year ago I wrote a set of articles that focused on externalizing unstructured content from SQL Server. I concluded that there were four primary ways to handle externalized content:

1. Dump the unstructured content on to the local file system from SQL Server - this gets the content out of SharePoint and relieves any SQL Server bloat related issues but you do not get any additional value out of the practice.
2. Externalize all content types and store them in an archive alongside your other archived content - as well as solving the SQL bloat issues this adds significant value with common policy management, long term archiving and retention/disposition/litigation support on the archived content.
3. Store unstructured content in your ECM system alongside your existing content – fixes SQL bloat; provides policy enforcement, long term archiving and compliance management plus the ability to re-purpose and reuse the content. Also gives you a natural integration point in to other enterprise information systems
4. Put the content in an ECM system and use SharePoint as a portal – an excellent solution for accessing existing ECM content and processes but doesn’t really leverage SharePoint’s native capabilities.

One year on, where are we? Consider a brief analysis of each of the four options today…

1.    There are so many solutions for simple file system externalization that some vendors are giving these away for free. The premise is that rather than storing the unstructured BLOBs (Binary Large OBjects) in SQL Server you use Microsoft supported APIs (EBS/RBS) to store the BLOBs on a local file server. This reduces DB bloat and helps smooth out the IO flow.

2.    Storing your content in an archive works well for inactive content but is typically too invasive for most active content. The reason is that in this model the content is either moved to the archive and no longer a ‘first class’ SharePoint citizen or a copy is made in the archive for compliance purposes.

3.    If you need the power of a traditional ECM system and want your SharePoint content to play nicely with all of the other corporate assets then this works well. If you just want the content out of SQL Server then options #1 & #2 are probably more cost effective.

4.    As long as we have traditional ECM systems then this option will be around. Although this option can give you access to the power of the traditional ECM system you typically lose a lot of the rich functionality of native SharePoint so this is always not for everyone’s data.

One year ago you could find Microsoft partners touting solutions modeled around any of these options. However, the thing that I noticed about the SourceOne archive product and the new version of Repository services was that they both include BLOB externalization to the file system as a basic piece of their architecture; in fact I’d argue that in both cases this capability is being undersold because it is just a byproduct of the actual capability of the product.

Both products can externalize content to a file system and then optionally perform their core operations on the content. The SourceOne product externalizes content to the file system and then based on rules it can archive the same content out to a common repository to be managed alongside archive email and archive file system data, (the latter is to be released later this year I believe). In the repository services case the product has always externalized content to a staging area before it was ingested into Documentum but in the new release you can write more granular rules to leave some content in that staging area and only have specific content moved into Documentum.

I wonder which one of the remaining three will become just a product feature in another year…or maybe a new option will emerge. Perhaps the real question will be ‘when will Microsoft fill this architectural gap?’ in the future. Contrary to hype RBS does not do this in the core product; RBS simply adds another area where the externalization can occur. In fact, for solutions that need to do ‘intelligent’ management of the underlying content RBS is a much less elegant solution. More on that in a later post perhaps!

Managing Social Data: Is SharePoint the Answer?

I was approached late last year by ARMA to write an article for their Information Management publication. During a lapse of good judgment I suggested that a topic that everyone is interested in but no one is talking about is how to apply compliance controls to ‘semi-structured data’.  Semi-structured data being those things like Blogs, Wikis, activity feeds, calendar items, etc. that are stored as structured data but look very similar to unstructured data when being consumed. 

Here's the intro to the article and the cover of the magazine. Before you ask, no that's not me hiding behind the question mark.

Managing Social Data:
Is SharePoint the Answer?

Records management (RM) professionals have been challenged to manage electronic data for some time. Their efforts have tended to focus on unstructured data, such as documents, scanned images, and spreadsheets. Recently, structured data held in databases has also come under close scrutiny, especially during discovery. However, there is a third category of data that tends to be overlooked. Read on for more or download the PDF version

Although users of all semi-structured data providers need to take heed, I pointed out that SharePoint was an opportunity for companies to look at a single point to apply these controls. If you are using SharePoint for all of your ‘Web 2.0’-type content then why not look to SharePoint to apply the compliance controls that you need.

Interestingly, when I was 80% of the way through writing the document I reached the conclusion that all I was going to provide you my dear readers was the bad news; I was not unearthing much in the way of solutions to these issues. Luckily, I was able to get some time from some of the SharePoint product team and they were able to add some of the color around the capabilities of SharePoint 2010 - thanks to all concerned. I’d also like to thank the team from Gimmal who provided some of the use cases and real-life background data.

The article is long…maybe too long but it should be worth a read if you care about SharePoint, compliance or have chronic insomnia. This is a topic which will only get worse before we catch up...

Main article: http://content.arma.org/IMM/March-April2010/IMM0310managingsocialdata.aspx

PDF Version: http://content.arma.org/IMM/Libraries/March-April_2010_PDFs/IMM_0310_managing_social_data.sflb.ashx

Note: My email address is quoted incorrectly in the magazine, it is actually [email protected] (with an underscore not a period). My fault not ARMA's.

 

SharePoint 2010 Content Management

I was asked to write an article about SharePoint 2010's ECM capabilities after the SharePoint conference for SharePointPro Connections magazine. It took a while for them to publish it and given the huge amount of quality SharePoint-related information that they have I thought it might get lost so here's a copy.

I tried to stay agnostic - I know that the SharePoint guys will think that I am SharePoint bashing and the traditional ECM guys might complain that I am being too soft on Microsoft but I think that this is a fair representation of the changes. As always - comments are appreciated...

Head over to http://www.sharepointproconnections.com/ and check out all of their other articles.

---

When SharePoint 2007 first started being widely adopted, the question of whether it could be used to replace existing ECM solutions was often asked. The general consensus was that SharePoint 2007 was not necessarily the best choice for the corporate ECM backbone. However, when SharePoint 2007 is coupled correctly with a traditional ECM solution the two systems complement each other extremely well.
With the imminent arrival of SharePoint 2010 that same question is bound to arise – do we need SharePoint 2010 and our traditional ECM solutions? Let's walk through the content management-centric features of SharePoint 2010, see what works, and look for where there is still room for improvement.

SharePoint 2010’s ECM Focus
Not surprisingly SharePoint 2010 builds out in a lot of the areas that SharePoint 2007 saw success; enhancing developer-related functionality, building communities, integrating search, supporting business analytics and the ubiquitous Office integrations.

The Microsoft SharePoint development team also addressed areas related to content management. This article highlights the key content management-related features in SharePoint 2010 and questions how well they might address enterprise requirements.

Library Services: Managing Content in Large Scale Deployments
SharePoint 2010 does address some issues related to how to handle large volumes of data – the removal of some list size limitations and support for remote BLOB storage providers in SQL Server for example. However, rather than focusing primarily on the back-end scalability of the system it seems that Microsoft took a more user-centric approach to this challenge.

It would appear that they asked the question “If I have huge volumes of data how might users interact with that data?” Many of the features highlighted in this article relate to tagging, finding and interacting with specific content which supports this assertion.

Metadata Enhancements
One of the single biggest challenges in enterprise content management has always been mapping content with relevant and accurate metadata. If you accurately tag content in your ECM system there is a reasonable chance that you’ll be able to find it again later.

If you don’t, then good luck wading through hundreds of thousands of documents looking for the one you wanted. SharePoint 2010 has focused a lot of effort in this one area. They have introduced Enterprise Managed Metadata services, location-based metadata defaults, unique document IDs and content ratings.

Enterprise Managed Metadata Services
The Enterprise Managed Metadata (EMM) service allows you to create one or more centralized libraries containing content types, terms and managed keywords. These can be published to sites across your farms and also accessed via web services by third party applications.

For example, you can push object types out to all sites across your farms where they can then be either used directly or inherited from locally as base types to create custom types. Note that keywords and terms don’t use the push model as they are used on-demand so they’re always current.

Included in the managed metadata are taxonomies (a formally managed hierarchy of terms) and folksonomies (unmanaged user defined terms). Interestingly, users can not only create folksonomy terms but they can also have those terms pushed out to the owner of a specific taxonomy for potential inclusion in the managed list.

If I were a corporate librarian, I might find this pathologically annoying or amazingly useful – I’m not sure which. If you fill out a field that is related to a taxonomy you get type-ahead assistance which not only makes it easier to enter values but it also cuts down on the number of erroneous errors that one tends to get in a folksonomy-based system.

Location-Based Metadata Defaults
This feature allows you to automatically set default values for metadata when an object is uploaded to a folder or a document set. For example, all documents in the ‘HR Contracts’ folder could have the ‘Department’ column set to ‘HR’ and the ‘Document Type’ set to ‘Contract’. These rules cascade down the folder structure which helps too.

Given that SharePoint 2010 is promoting a ‘navigate using metadata’ paradigm rather than a folder model this feature is very useful. For now, users could continue to use folders as their ‘filing system’ but behind the scenes the content is being discretely tagged.

Unique Document IDs
The idea that SharePoint 2007 did not have unique document IDs seems beyond comprehension. How was SharePoint able to track which object was which without this fundamental identifier? It turns out that SharePoint 2007 did use unique identifiers but only internally.

SharePoint 2010 externalizes the use of unique IDs. Although you can enter these IDs directly in to the UI and bring up a document they are better suited to being used either programmatically or embedded in hypertext links; here’s an example of one just so you understand why: 7K3W6YVEA2YC-17-1. You can define naming schemes so a document can be identified even if it moves between site collections.

One thing missing from the unique IDs is that each version of the document does not have its own ID. Ideally you would want to be able to address an individual document in the version stack not just the entire stack.

Content Ratings
With this feature, each document in a list has a number of stars shown alongside it and users can rate the content. Later you can sort the documents by popularity. At first sight, this feature looks a bit like "a gimmick looking for a home" rather than a true enterprise content management feature, but after using it I realized that it is actually quite useful. Assuming that the people you share content with are like-minded, the democratic approach to deciding which document is most appropriate is amazingly accurate.

Document Sets
Document Sets should be something to get excited about, but alas they are not quite everything that they should be in this first iteration. The general concept of a document set is that it allows you to group a set of documents together and treat them as a single object.

They share a single set of metadata, can be versioned as a group, downloaded as a single Zip file and routed through a workflow as a group. The set also includes a Welcome Page that users see when they select the set. This page allows you to add some context to the group of documents. So what’s missing?

* A document can only be in one document set at a time. The concept of related documents is often used to represent a compound document, in this example it would be reasonable to have a common piece of content shared across many the document sets – a standard set of terms and conditions for example. 
* Documents do not have an explicit order within the set. Often the order of the component parts of the document in a collection is critical to whether the end result makes sense. You could order the documents implicitly by keeping the object names ordered alphabetically but that’s a little 1970’s for a product with 2010 in its name. 
* If you perform a search and find a document you have no way of knowing that the document belongs to a document set. From a contextual and compliance perspective this might be undesirable because the document’s inclusion in the document set is almost certainly relevant. 
* The ability to take a snapshot of the specific versions of documents in a document set and protect that snapshot is missing. This model allows you to freeze a copy of the documents as a record but then continue to update the documents in the live set and is often used in records management.

Advanced Content Routing
In SharePoint 2007, content routing was primarily a feature of Record Centers; SharePoint 2010 extends this functionality to all sites. With content routing you establish rules that determine where a document will be routed to based on the metadata of that document.

This feature allows you to create "drop off" areas in your sites; content can be dropped in to these locations and the rules will then file the content away in a predetermined location. Assuming your users tag their objects correctly this feature can save a lot of mistakes.

UI Changes
We are probably all aware of the new ribbon-oriented UI in SharePoint 2010 but there are some other more subtle UI changes hidden away in SharePoint 2010 which directly affect how you will interact with unstructured data.

These changes include Metadata Driven Navigation, advanced targeting, better business data access, RSS feed monitoring, key performance indicators, summary links and searches. Let’s consider the first two of these as they are the most relevant to this article.

Metadata Driven Navigation
If there’s one feature of SharePoint 2010 that encapsulates the approach that Microsoft have taken to large scale ECM deployments it is Metadata Driven Navigation. Being able to store 50 million documents in a single SharePoint list is very impressive but unless your users can find the ‘one in 50 million’ document that they need then you’re not really being successful.

Metadata Driven Navigation allows managed metadata fields to appear in tree-view controls on the left-hand side of the navigation pane. Users can then select values for any field to constrain the results. For example, "show me only final versions of contracts."

Behind the scenes you are really adding constraints to the search predicate and seeing the results in real-time. I’ve heard people refer to this as "zero typing search," which makes sense. You can elect to add these managed fields to the index which seems like it would be a good idea.

So why is this feature so useful? Two primary reasons, firstly it allows you to remove huge swaths of irrelevant documents and hone in on the ones that you need and secondly it is familiar – most online shopping sites use this approach.

Advanced Targeting
Along the theme of getting access to content quickly and easily, SharePoint 2010 introduces the ability to customize page views based on membership of audiences. Within Central Admin you can set up rules which specify who will be a member of specific audiences.

You can then specify which Web Parts, items and links will be visible on a page based on those audiences. This allows you to better target specific content to specific users within your environment.

Offline Support
SharePoint 2010 includes a rich desktop client called SharePoint Workspace, (a replacement for the old Groove product). There are a few ways of looking at this client; its primary use case is to allow you to work offline by caching a copy of documents and data to your local machine.

Workspace can cache an entire site’s content including custom lists, line of business data and even InfoPath forms. If you edit content when you are offline Workspace will push those changes back up to the originating site when you re-connect.

Workspace can also be viewed in a couple of other ways. It gives an alternative view of the SharePoint site’s content even when online; some users might prefer this non-Web interface. It could also be utilized as a solution for accessing your SharePoint sites in environments that have connectivity challenges. The synchronization of content in both directions is asynchronous so you can make changes to documents and then let the systems synchronize those changes in their own sweet time.

Office Web Applications
Microsoft is making a lot of noise around the new web–based versions of Word, PowerPoint, Excel and OneNote. For sure, these new applications fall into the unstructured data camp but right now the best application that Microsoft seems able to come up with is to interact with the Office documents “when at a shared terminal at a conference or at a coffee shop” . I’m sure that there are a multitude of better applications – if you know of a real one, contact me at my blog.

SharePoint List Item Limitation
In SharePoint 2010 you can have up to 50 million items in a SharePoint list. This is a really bad idea; seriously you should probably look for a new system architect when the number of items in a list is six digits long, ever mind eight!

That said, your lists will grow and SharePoint includes some interesting ways of limiting how large a search query can get so you don’t have users taking down the system by entering too broadly bounded queries. You can limit how many objects in an end user’s query results and you can then allow admins and processes to bypass that limit. You can also allow the limit to only be broached within certain times of the week.

Not Creating Unstructured Content in the First Place
Most of us tend to open Word when we want to type something, Excel when we want to calculate something and PowerPoint 10 minutes before we are about to present something. When one looks at many of the Web 2.0 features of SharePoint 2010 you realize that perhaps we should give more thought to our choice of tools.

SharePoint 2010 has broad support for the wiki paradigm; it has native blog management, activity feeds and discussion boards. I wonder how often we should be using these (semi) structured tools instead of just defaulting to creating a monolithic piece of content. Just a thought…

Your Homework
There are a plethora (well quite a few) other content-related features that are not covered in this article. Some of them are worth further investigation. I suggest taking a look at SharePoint 2010’s record management, the new web content management capabilities, all of the Search features (especially ‘refiners’ which allow you to quickly filter result sets), Business Connectivity Services, RSS feed monitoring, key performance indicators, and summary links.

Managing Semi-structured data in SharePoint and what’s new for ECM fans in SharePoint 2010

So, it may seem like I have forgotten how to write but in reality I’ve been writing, just not here! I have two papers complete which I think my usual readers will be interested in. One is for Information Management Magazine (ARMA’s publication) and the other for SharePointPro Connections Magazine. I need to wait until they are published before I can post links to, quote them or discuss them in detail but here’s a brief synopsis of each of them just to get you all excited:

Information Management (ARMA)

As well as publishing IM magazine ARMA represents information management professionals and records managers. They asked me to write an article about how records managers could manage the new wave of social data types – Blogs, Wikis, IM, Activity feeds, polls, discussion threads, etc. Given that SharePoint purports to manage most of these they also asked that the article would have a SharePoint focus.

This one started out well, I defined the data types and the issues at hand. I initially called this type of data ‘semi-structured’ but it turned out that the DB world already had dibs on that phrase so after talking to Microsoft I coined/stole the phrase ‘social data’. Technically what I analyzed was data that was structured at the back-end but had an associated presentation layer that makes it look unstructured to an end user.

It all fell apart when I failed to find anyone with an ‘off the shelf’ solution to the problem. Thankfully I had some great conversations with the SharePoint product team and it turns out that SharePoint 2010 does include some basic capabilities. I stitched that together with some of the custom solutions that I discovered and it all came together rather well.

SharePointPro Connections Magazine

This article is a summary of what’s new in SharePoint 2010 from an ECM perspective – the article highlights the key content management-related features and questions how well they might address enterprise requirements.  I cover most of the new ECM-related features and dig down into some of them where I feel there’s either a misunderstanding wrt what the feature really does or where I think Microsoft missed the mark. IMHO this is not as interesting as the ARMA article which digs into new territory but it is a good primer for SharePoint 2010.

These types of articles frequently get me into trouble because SharePoint fans think that I am bashing SharePoint and my Documentum colleagues think that I am glorifying SharePoint. My solution is to take a truly agnostic view of the factors and try to be honest. I am not known for my tact but this does seem to be working so I’ll stick with it. I’ve also taken to talking to insiders at Microsoft to make sure that what I am saying is at least fair – maybe not palatable but fair.

I’ll post links to the articles as they get published…

Externalizing BLOB Storage in SharePoint 2010

OK, so this entry should be called SharePoint Conference Day Three but to be honest I only attended one session – the rest of the day was taken up with meetings with Microsoft and Partners.

So as a rather sad indictment of the state of my life, the best session of the week for me was "Externalizing BLOB Storage in SharePoint 2010". I was absolutely enthralled with this session. Srini Acharya and Burzin Patel from the Microsoft SQL server advisory team did a very nice job of addressing the concept of RBS and how it fits in to the new SharePoint 2010 architecture.

They started with an overview of the general RBS concepts and also a demo of RBS using Microsoft’s own free file stream provider that just dumps the BLOBs on to a local file server. To be clear, RBS on its own without a provider will not do anything, Microsoft have the aforementioned file stream provider and vendors are being encouraged to develop their own. During the Q&A session they made the point that customers should not write their own providers and that vendors should work with Microsoft on their offerings.

Burzin mentioned that "BLOBs typically account for 60-70% of total content" in the SharePoint content database. This is a lower number than I’ve heard quoted previously although most of the numbers you see quoted are from vendors trying to sell externalization solutions so maybe they cannot be trusted ;-)

EBS vs. RBS

It seems obvious that the team have been making an effort to bring SharePoint awareness in to the RBS layer. They have added an ‘RBS Maintainer’ to sync certain operations between SharePoint and RBS. I think that the name of this component needs some Microsoft attention BTW.

They presented a table outlining the key advantages of RBS over EBS.

• RBS has Managed interface whereas EBS’s is unmanaged.
• The RBS BLOB store scope is per Farm, EBS is per Content DB.
• SharePoint 2010 adds the Configurable Maintainer to sync operations from SharePoint down to RBS.
• RBS also has a PowerShell-based UI within SharePoint, this was pretty rudimentary but again it shows a commitment to the integration.
• RBS supports multiple providers. I think that you will be able to register multiple providers for a farm and then select which ones are used and when – I don;t understand the scope of that decision though, I’d love to be able to enable a provider for site A and a different one for site B, (or even by document library perhaps).
• Best of all…the RBS implementation includes PowerShell-based migration between content stores. Translation? You can move BLOBs from SQL to any RBS provider, from RBS provider to RBS provider and from EBS to RBS… They are looking at a ‘shallow migration’ where the BLOB would stay where it is but the management of it would be migrated (only for already externalized BLOBs). This is a huge thing – one of the senior engineering leads was sitting by me during the session and when this was announced he turned around and gave me a huge grin…given that I’d met with him to discuss RBS earlier in the day it looks like he’d intentionally kept this nugget from me…you know who you are and I will get revenge!!

Issues According to Microsoft

I have to take care when talking about the architectural challenges with storing BLOBs in SQL because I am not in the business of criticizing SharePoint (anymore) so the next section was a blessing for me. So here it is…Microsoft’s assessment of the issues with storing BLOBs in SQL…not mine, Microsoft’s…taken verbatim from their slide!!

• Optimal Capital Expense
  • Trade cost effective BLOB storage for expensive SQL Storage
  • Ability to group/store BLOB separate from Metadata
• Optimal Operational Expense
  • Storage management beyond SQL
  • Facilitates cost effective/optimal backup/DR policies for BLOB vs. Metadata
• Take advantage of advanced storage features provided by BLOB store vendors
  • Expunge, multiple storage locations, immutable writes
  • Guaranteed retention, guaranteed deletion,
• Hierarchical Storage Management
  • SQL and layers of BLOB stores offer more savings in CapEx and OpEx
  • Efficient access patterns.

That’s Microsoft’s list – unedited. 

They then did a re-cap of where we are with RBS and SharePoint.

• RBS still needs a storage vendor to supply a provider and a BLOB store. OOTB RBS doesn’t do anything without a provider.
• RBS includes a RBS Maintainer and a Provider Library for SharePoint 2010. The maintainer synchronizes BLOB delete/orphan operations between SharePoint and RBS. The library is just that, a list of registered providers.
• All BLOBs in an active content DB will be externalized – the level of selectivity is still binary i.e. all or nothing.

How does it work? Defines and exposes 3 views for interaction"

• Application View
  • Interacts with the SharePoint WFE, Provider Library, SQL DB
  • Implemented by SharePoint 2010 - transparent to the user
• Administrator View
  • PowerShell commandlets - calls set of SPs, function as in SQL
  • Installation, configuration, provisioning, RBS maintainer etc.
• Provider View
  • Defines an interface that should be implemented but the BLOB storage provider.

The jumped in to a really nicely done demo

• Went to a RBS enabled site
• Opened a PowerShell window
• Showed that the file stream provider was disabled
• Showed a SQL select on the WSS_Content table. There were 73 results but they all had RbsId of null so none of the objects were externalized.
• Imported an image
• Re-ran the query and saw 74 rows – still nothing with a RbsId.
• Enabled the RBS provider
• Added another image
• Re-ran the query
• 75 rows but one of the rows now has an RbsId
• Went to the local file system on the C drive
• Document was on the file system with an object name of the GUID (he opened it to show that it was the same file).

It didn’t look like much but for those customers struggling with SQL bloat this was manna from heaven.

Next they went on to describe the RBS Maintainer module. This is the piece that creates an element of awareness between SharePoint and RBS. Here are the key points:

• It tracks the deletions in SharePoint and pushes them to the BLOB store. Interestingly it includes the ability to set up rules to delay the delete from the BLOB store to make it easier to manage restores – if the BLOB is still in the external store when you restore SQL then you are probably in good shape.
• Does the clean-up of BLOBs for failed transactions - orphaned BLOBs. This and the line above are basically garbage collection management.
• Can be on the DB server or separate box
• Can be scheduled to run periodically – I guess it is a batch job rather than an inline process.

They went on to discuss How to Backup works. I think they oversimplified this section – my advice would be to think carefully about your chosen backup method. Here was their advice:

• Backup:
  • Start SQL Backup first
  • Then backup BLOB store
• Restore
  • First restore BLOBs
  • Then Restore SQL

The team showed some performance data that I found questionable – it showed that externalization of data did not have a negative effect on the performance of SPP. I suspect that this was all they wanted to show but IMHO their data sample contained files that were too small and I’m guessing there were not suffering from the fragmentation issues that I mentioned in my earlier Blog post. I’m confident that if they used a more realistic sample they’d see an improvement in performance (reads and writes) especially directly to an equivalent file system.

There were a large number of follow-up questions in the session. Many of them were about whether RBS would make it easier to get existing file system data in to SharePoint… I’ll reserve my comments on that for a later entry I think.

Again, Srini and Burzin did a stellar job explaining this abstract concept to a packed audience.

SharePoint Conference Day Two

Today was a bit more fragmented because I had some Microsoft meetings and some EMC partner meetings but I got to some good sessions regardless…

Attended the EMC consulting best practices session "EMC: Good, better, best: Understanding Enterprise SharePoint Deployment Best Practices". The EMC team acted out a scenario that represented a typical customer/consulting engagement. This worked well and the audience seemed engaged as they moved through their demo. The size of the crowd suggests that there are a lot of people out there who are looking for more information and best practices. Not a surprise given the changes in 2010.

My next session was Revolutionizing the SAP user experience… with SharePoint. Truthfully, most of this session was way above my head as I am new to the SAP world but the rest of the audience seemed to be enthralled! The heart of the session seemed to be that if you've chosen to use SAP then SharePoint has a place in exposing the SAP systems to end users. They showed examples of taking 7 SAP screens and creating a single SharePoint-based view with only a subset of data available.

The most interesting session of the day for sure was the Document Management Deep Dive with Ryan Duguid. The following are notes from that session...

High-level Messages

• SharePoint is all about "Traditional ECM" and collaboration brought together.
• Primary tag line - "ECM for the Masses"

Ryan showed a graph that shows how Traditional ECM is typically aimed at small number of types of content in a typical organization but not aimed at the larger number of non-controlled types of content. SharePoint is focused on that latter class of content types and on the breadth of content.

A bit of random marketing in the deck...

ECM Tenets: 3 'E's

• Easy to Use
• Everyone Participates
• Enterprise Ready

The Importance of Disposition

Ryan spent a considerable length of time talking about disposition. FWIW I agree entirely with this position. He had a graph that showed that starting today, with a 40% annual increase in data you will have 45TB of content in 10 years. If you dispose of just 10% of content each year you'll drop that to 25TB…dispose of 20% per year and it goes to 10TB…30% gets you 4TB. This shows how important it is to have a real disposition policy.

What's added in 2010 for ECM?

• They added unique document IDs to all objects - this seems like such a fundamental component that I do not understand how they managed previously without it - maybe it was hidden. The doc ID is optional in a site, they can be reset across all the site which is very odd and you can provide a prefix to ensure uniqueness, (which suggests that maybe they are only unique within a site or perhaps farm - this needs checking out).
• Ratings (click on a star - 1 through 5 to say how lovely something is) on any object in the site is being sold as a a big deal although this is something a bit alien to me. I could see it being popular in many cases…I'd just never given it any thought. The more I think about it the more I like this concept…
• Tagging - this one I get. Being able to create tags, apply them to an object and manage them. This is how navigation is going to be driven moving forwards. The model that I see SharePoint moving towards is one where the physical location of an object is there for the application of policies and security but not for navigation. You can optionally include the folder location as a navigation tag so it can also be a navigation factor but that’s incidental in this model.
• Drag and drop from Windows explorer into the multi file upload component – useful.
• Multi-select operations – again, great to see this and hard to imagine how SharePoint got away without having it.
• The ability to view compliance details of an object from the drop down menu. See if there's any retention, disposition, etc. applied to that document. Nice feature, very useful for admins, records managers and end users.
• Send To - used to be able to send to one record center, now you can reroute to multiple record centers or even just to another site, (I believe that all the functions previously associated with record centers can now be applied to any site). Ryan showed how you would set up a new record center. Looks like the decision to Copy, Move or Move and leave Stub seems to be per record center.
• Can apply retention policies to folders not just content types.
  • Within a folder you can define whether the objects declared as records are treated differently then other objects in the folder. This explains how it is they can simply select an object and click 'declare as record' without answering any other questions. The rules to determine the behavior once something is tagged as a record are defined in this way.

• Folders are a way of applying security, retention, disposition rules, etc on a location centric basis. I suspect that Microsoft are going to push faceted search/navigation as the way forward and folders as being a back end mechanism. This makes a lot of sense so long as the tagging is done correctly…a big risk perhaps?

Managed Metadata Service

• Supports type-ahead when adding a tag to an item - this is fully qualified so if you type a term that exists in many places in the taxonomy it will allow you to see the one that is in the right location.
• Want to support a folksonomy maturing in to the managed taxonomy. For example, you can send a message to the taxonomy owner to suggest new terms all from within SharePoint 2010
• Ryan showed how easy it was to create new taxonomy values…of course creating an agreed-upon taxonomy in the first place is the real challenge!!
• Note that the Managed Metadata Service can be shared across sites – this is a huge deal.
• It also supports multi language support so you can switch to a new language and see localized tags, (they don’t translate terms but allow you to enter multilingual tags.
• The hierarchy is "Groups, term sets, terms" this allows you to set rules across the hierarchy including ownership of the terms.
• Seems like there's been a lot of thought put in to this feature. Especially around making it easy to move things around and turn on/off the individual terms.

Navigation by metadata is probably the most significant feature that I've seen in a long time. I was going to say 'if it works' but if data continues to grow as predicted then maybe it has to.

One very important point is that Microsoft believe that in order to get users to invest in the effort of tagging content in the first place then they need to see a 3x return on that investment. Faceted navigation is that return…spend the time tagging today and find the content in whatever way you prefer tomorrow.

Document Sets

• Like an intelligent folder, they wanted it to look familiar so they make it look like a folder
• The set can have a landing screen which looks like a wiki - more than just a relationship object, it allows for collaboration around a collection of documents.
• It looks like a document can only live in one Document Set which might be a real problem. This needs validating.
• Ryan described it as a "Foundation for document assembly"

Content Type Syndication

Share objects, metadata, workflow, between farms.

Ryan demoed this - He opened the admin screen and defined where the content type hub was. He went to that site and provisioned a new content type just as you would in any farm/site. He then managed the publishing for that content type. He published it, waited for a sync job to run and then went back to a remote site where the new content type was available to be provisioned from the hub for local use.

Content Organizer

In site settings there are rules that can route content to specific locations. He set up a new rule that says for a specific content type for content that has a specific metadata value it would be moved to one location. It can provision a new folder based on the value of a specific piece of metadata. For compliance this would be a very compelling feature – especially if you can get metadata tagging done accurately at the right time.

SharePoint Conference Day One

I'm at the SharePoint 2009 conference in Vegas this week. Here are some notes from today – no conclusions, just some notes…I'll try to post something later in the week and I'll continue to tweet all week - @chapmaa

The Keynote

You've no idea how many bad keynotes that I've sat through in my life but today's was not one of them. The keynotes ran for 2:45 without a break but the presenters did such a good job that it did not feel long at all. So how did they do that? They had some passionate people on the stage, (most notable was Tom Rizzo), and they told the message in a way that the audience wanted to hear it not in the way that marketing wanted to present it. Within the first 30 minutes Rizzo was writing code in a live demo! They didn't just say what 2010 will do they showed what it will do and how. Without exception the team did an exceptional job.

As a side note, Jeff Teper called SharePoint "the ultimate Swiss army knife"…I think that’s a good description.

The recurring themes…

• Reduce costs…not sure how many times Ballmer mentioned reducing costs but it was a core message. Have fewer systems, fewer vendors, be more nimble. "Get more done with less" was the overriding message here.
• The cloud…they kept mentioning the cloud but they didn't give to much detail. This was perhaps the one area where it was higher-level than I'd have liked to have seen. Microsoft online was mentioned, the new web-based office tools were used heavily but no specifics about the actual cloud structure per se.
• Support for inter- and intranet support as well as support for the office online tools. These are two very different things but they got a little merged in the keynote.
• Streamlined deployment and debugging…a bit techie for me this but I could see the value of being able to easily deploy solutions.

Post keynote I attended intro sessions and ECM sessions…

The Content Management Messages

I'd say that in general ECM is being played down as just part of a much bigger play. For example, Ryan Duguid pointed out that you can tag, annotate and place legal holds on anything in SharePoint, not just documents. I was watching out for ECM comments for obvious reasons, here are some highlights:

• Overall ECM 2010 Changes
  • Scale ECM up and out - a general message and pretty vague
  • Taxonomy, workflow and document sets have been added
  • DAM (later qualified by Ryan as being not being full blown DAM but being media management for images and video including intelligent streaming direct from SharePoint)
  • More flexible governance (policy and RM support)
    • Declare as record - I need to talk to Ryan about this one because it looked like it just 'made the object immutable'  not actually declared it as a record
    • Manage records in place
    • Route to records center and leave a stub behind
      • I loath and detest this as an approach but Ryan addressed this by saying that Microsoft were giving customers a choice - IMHO this means that they can do it properly or they can stub ;-)
  • Increase in number of objects that can be managed by SharePoint
  • Master metadata management (see below - this one deserves its own section)
  • Document Set support…create a relationship between documents and manage them as a single document.
  • Offline support - (see below - this gets its own notes too)
  • Extension of routing table usage
    • Can be used anywhere on any content type and across sites
    • Can create a 'content organizer' as a drop zone
      • Users import content in to the content organizer and the rules will move the content in to the correct location. This seems to include rules that will look at the contents of the object, (Ryan showed moving anything with a social security number to a specific location).

Metadata Driven Navigation

SharePoint 2010 has a lot of focus on metadata taxonomy management. This still looks like it is not entirely fully baked yet but the basics are there and it is flexible. It looks like it scales across SharePoint sites/farms but also allows external systems to access the taxonomy. This is a good strategy for a few reasons:

• Strategically, if you own a company's metadata you become the information hub for all the information-centric systems
• If you can get users to tag content correctly at a logical time then you can remove the workload from them later and use rules to apply complex operations on the content further on in the information lifecycle.
• Most importantly navigating content based on metadata is much more natural, (you search and then activate filters to remove noise - think about how eBay let's you filter in real time).
• If the only thing that SharePoint 2010 brought was this faceted navigation then it would be worth the effort to upgrade for that alone – IMHO.

Offline Support

Microsoft talked about "SharePoint Workspace" a lot. This allows you to synchronize content from SharePoint to your desktop to:

1. Work offline
2. Use this as a way of working in low-bandwidth locations

The second use case above is a very interesting usable model and in some ways addresses one of SharePoint's enterprise limitations.

I’ll make an effort to learn more about this because it is potentially very interesting as a Microsoft partner.

Managing content on the file system

This is another one where I need to do some extra research but in a nutshell the file system has the capability to store metadata natively and then content can be pushed up in to a specific SharePoint site based on rules. As the content hits the SharePoint site the rules in a content organizer can further file away the content. It can leave stubs behind BTW, I’ll reserve my comments until I see how good the native file system is at masking these as being virtualized.

Here are my raw notes from Ryan’s talk – as I say, I’ll be working through this one a bit this week.

• File server resource manager. Windows Server 2008 R2
  • Apply metadata properties to content on the file server
  • Can have rules running on the file server
  • Can look inside content for patterns too.
  • Can auto upload content from the file server to SharePoint automatically.
  • They leave links behind that point to the SharePoint location.
    • On ingestion content is further redirected to a SharePoint folder location

 

I’d say that day was was very good – 7,400 attendees, almost perfect organization and a great vibe.

SharePoint Conference and New EMC Products

Next week is the annual SharePoint love fest – this year it is in Vegas. Unfortunately, although I’ll be at the conference my understanding is that I will not be able to Blog or Twitter on what I see/hear…something to do with ‘What happens in Vegas, stays in Vegas”. Seems like it makes the entire trip a bit of a waste of time ;-)

Seriously though, I will try to post some highlights although they may be Twitters rather than full blown blog entries. It looks like a huge proportion of the week is given over to SharePoint 2010 which is not great surprise I guess. You can follow me @chapmaa if you have nothing better to do.

The big news for me next week also includes the release of two new SharePoint-related products. I don’t normally Blog about my role at EMC and our specific technologies but this is a particularly big event for us and not surprisingly the products deal with some of the specific ‘opportunities for excellence’ that I’ve discussed in my Blog.

For those of you that are interested, here’s what we will be announcing at the conference:

• ‘My Documentum for Microsoft SharePoint’ is a SharePoint emulator for Documentum users. It allows end users to interact with content in a Documentum system without leaving the familiar look-and-feel of SharePoint. The key feature of this solution is that it is very faithful to the native SharePoint look and feel…in fact it is about as close as we can make it without breaching any legal agreements that we have with our friends in Redmond.
• Slightly more interesting from a technology perspective is ‘Documentum Repository Services for Microsoft SharePoint’[1]. This product makes use of EBS and will support RBS moving forward. The solution makes Documentum into a back-end storage and ECM system for SharePoint. This is especially useful for companies that want SharePoint’s ECM to play nicely with their broader corporate ECM backbone. It is a solution that leverages the concepts that I’ve rambled on about for the last 18 months or so – creating a model where content is stored in a back-end repository rather than SQL Server but in such a way that it is 100% transparent to SharePoint. Apart from just being damn cool this solution really unites two core enterprise solutions in a way that takes no value away from either system. I’m very proud of the fact that we developed the entire product in 13 months which is amazing for a full-blown enterprise strength solution. It has been in limited release for 6 months and is now available to all and sundry.

If you care you can come to the EMC booth and see more…I’m attending the conference as an attendee and will be attending sessions but we will have people much more personable and presentable manning the booth.

If you’ll be there then have fun, if you are not there then congratulations on having a life.

---

[1] You have to include ‘Microsoft’ before SharePoint in product names making this a sweet 55 characters long!