REVIEWER: Dean Hoff, consultant with Dean Hoff Consulting.

BACKGROUND: Located in Minneapolis, Minnesota, Dean Hoff Consulting helps clients develop and implement analytic strategies. This includes building business intelligence applications utilizing software and other tools, then showing clients how the analytics can interact with their current processes.


PROBLEM SOLVED: While working with a major direct marketing agency, we ran into a problem when developing predictive models. Part of the process required us to apply model scores to a large demographic file containing more than 100 million records. We developed the models and wanted to implement them, but in order to do this, we needed to be able to score the records. We went to the agency's internal resources to determine if they had the capability to accomplish this. It turned out that the size of the file was too large, so they told us to outsource this step. After discussing possible solutions with a third party, it was clear that the costs would have been prohibitive. I knew there had to be a way that we could do this internally. After conducting extensive research, we decided to test DMExpress, the high-performance data transformation product from Syncsort Incorporated. We found that DMExpress allowed us to calculate a model score as well as score the large number of records in a short amount of time.

PRODUCT FUNCTIONALITY: Using DMExpress, we are able to take a historical marketing file and append the demographic information. Basically, we're matching the marketing file with a large demographic file using the join functionality. Then we develop the model using the appended file, take the results of that model and score all of the demographic records we have.

STRENGTHS: One of the key benefits of DMExpress is its speed. Instead of taking days to run the aggregations, I'm able to complete the process in minutes with DMExpress. It also provided me with the functionality I needed to process data in other areas of the application as well. I initially integrated DMExpress into the application to just handle the scoring, but because of its join capability, I was able to use it to do the data append, too. With the aggregation component, I can actually do some profiling with the data. Because of this enhanced functionality, I'm using it beyond the initial intent.

WEAKNESSES: It would be beneficial to be able to trim multiple occurrences of one or more characters from the beginning and/or end of a text value. Also, it would help to have arithmetic functions so I could perform modulus, power and random number generation. From what I understand, these features have now been incorporated into the latest DMExpress release.

SELECTION CRITERIA: I used SyncSort a number of years ago for a different project and knew that it was one of the faster programs for sorting, so I started researching what Syncsort had done since that time. I read about DMExpress and thought it would be a good fit for the project.

DELIVERABLES: On a monthly basis, I download the flat files from an FTP site into my system. Using the Advanced Data Management component of DMExpress, I am able to perform the complex aggregations much faster than anticipated, taking minutes instead of days. Once the processing is completed, the agency has a breakdown of everyone in a geographical region who was contacted and if they ordered a product. The information can then be used to improve the response rate of a direct marketing campaign by reaching out to consumers who are more likely to make a purchase.

VENDOR SUPPORT: I contacted Syncsort's technical support with a couple of questions when setting up the application. They responded immediately and were very helpful.

DOCUMENTATION: DMExpress includes an online help feature that guides us through the development of tasks and jobs. It is very easy to use and meets our needs.

Syncsort Incorporated
50 Tice Boulevard
Woodcliff Lake, NJ 07677
(201) 930-8200

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access