This column continues last month's discussion of evaluating enterprise content management, "Vendor Evaluations Part 1: Functional Requirements." At the end of the column, I recommended an in-house evaluation as the best method to understand how well an enterprise content management (ECM) application would work in your organization. Now I will outline how to design and implement such an evaluation.

The evaluation plan should include at least six core areas: vendor assessment, product functionality, architecture, security, integration and ease of use. The importance of each of these will vary with your requirements and should be weighted accordingly.

Vendor assessment should include checking customer references with similar functional requirements. Do not assume others in your industry will have the same needs ­ they may be tackling different problems. Similarly, it is not necessary to require a reference from your industry if you are working on basic content and document management issues. Do get industry references if you are looking into categorization, information extraction or other language-dependent operations. Performance of these tools can vary widely depending on the nature of the content. For example, identifying protein names is a particular challenge in the pharmaceutical industry.

In addition to the usual financial considerations (initial license cost, maintenance fees and consulting rates), assess the vendor's viability. There are many start-ups in the enterprise search and categorization markets, and not all will survive. We saw significant consolidation last year with well-known and respected companies such as Quiver, Inktomi and Semio being acquired. Other big names in the market have scaled back or continue to lose money. Picking a long-term success is not a day at the races, but we should understand a vendor's position in the market and past performance to minimize the chance of surprises.

Test product functionality against your specific requirements. This means installing the product at your site if you will host the system yourself or running it off site if you will use an application service provider. Define a test plan that measures the effectiveness of the key features you expect. We all understand that search and categorization tools are not mind readers, and they never give us exactly what we want. Thus, it is important to determine how close they come to the ideal situation. Will results in hand press the vendor's technical staff to describe what is required to improve results? For example, can you specify that particular links appear in the search results list for specified terms? What is required to develop a gateway to your document management system to improve performance? Compile a test data set that reflects the full range of content you will use in production. If you will be indexing content in document management systems, network files systems, customer relationship management (CRM) applications or external Web sites, be sure they are included in the test.

The test plan should be divided into categories, such as content process, search, categorization, personalization and security. Within each category, list individual test elements, such as ability to index PDF files and effectiveness of access controls in a content repository. Weight each test element according to its relative importance, and use the combined score to calculate the category score. In turn, weight each category and combine the results to derive the final score for a product. This approach allows flexibility in developing test criteria without skewing the results to categories with many test elements.

During the evaluation, assess architectural compatibility and security factors. Key considerations are how the application will integrate with your portal and whether it will work with an existing or planned single sign-on system. Because most enterprise content management systems require access to distributed content, evaluating security mechanisms can be the most time-consuming part of the evaluation. Be sure to understand how distributed security mechanisms work. For example, are access control lists synchronized between repositories and a centralized search system or does one system delegate security services to another? One approach is not necessarily better than the other, but the implications should be understood. When synchronized access control lists (ACLs) are used, for example, changes in one system are not reflected in the other until they are synchronized again. This will influence the frequency of those operations.

Integration and ease of use are especially important when integrating an enterprise content management system in a portal. Often ECM systems provide services across a range of applications. During evaluation, determine the level of effort required to integrate these services. Does the vendor provide an application programming interface (API) such as a JSP or ASP interface suitable for your environment? Include both end users and developer needs in ease-of-use considerations.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access