Back in the day I was a UNIX Systems admin. One of the “joys” of being a sysadmin was chugging through log files. I looked for errors in applications. I looked for commands that people used to delete their entire account (rm –r ./*.* is a good one). I looked for the reason why particular databases brought down the operating system. One of the essentials in any sysadmin’s toolbox is tail <>. Looking at these nicely formatted, but rarely structured enough to be placed in a database table (log4j is a good example), log files was a daily occurrence with an active environment.

Although application and operating system log files still exist, the growth of the Internet of Things is starting to emerge. Everything from heart monitors for running, to brake sensors in cars to refrigerators will provide information on what a device is doing and/or doing for you. Much of this sensor information is similar to those UNIX log files from my past life. Using multi-structured formats that often don’t fit well in a relational data store, the information from sensors in machinery, GPS mapping tools and even Fitbits comes in some interesting and constantly changing formats. Figure 1 is an example from a weather station:

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access