The topic of real-time data processing emerged as one of the most talked about at the recent Strata & Hadoop World conference in San Jose, a fact confirmed by several exhibitors at the event.

Information Management spoke with Marcelline Saunders, director of sales engineering at Lucidworks for her take on why the strong interest in real-time data, and what other data management and analytics topics were top of mind for attendees.


Information Management: What are the most common themes that you heard from attendees and how do those themes align with what you expected?

Marcelline Saunders: There was a lot of talk about using data visualization to discover relationships between data points. Then, once those relationships are found, discussion moved to how to find value and intelligence in those relationships.

Security remains top of mind as data applications continue to involve more and more users inside an organization and require more fine-grained security from collection to application.


IM: What are the most common data management and analytics challenges that attendees are facing?

MS: As I talked with participants in sessions and attendees that visited our booth, one topic that kept coming up was the challenge of real-time processing. Developers are realizing that static datasets are a thing of the past and [the need to] constantly update incoming streams of information is the new normal.

Data applications must be designed to ingest and index these high-volume streams of information and include them in reports and search results immediately. ETL systems also have to be sophisticated enough to handle these high volume streams of information and keep up with the pace. As streams become more commonplace, there will be a need to more efficiently find the relationships within these fresh, ever-changing data sets.


IM: What are the most surprising things that you heard from attendees?

MS: I was surprised that Hadoop isn’t brought up much in conversation anymore. It’s considered a boring topic. Storage is done. Everyone is moving on to the next thing: “Now that I have all this data in one place – how do I get value out of it?”

In the opposite direction, data security is becoming even more of a concern. For example, how do you show the end user only the data and documents they are authorized to access? Security models are becoming more sophisticated to meet these needs.


IM: What does your company view as the top data issues or challenges in 2016?

MS: As mentioned above, security is a large concern along with visualization tools for finding trends and relationships in data sets to get the most value. The importance of data streams is just now starting to be recognized by companies of all sizes and developers have to be ready to create apps that index and offer access to data and events as they happen.

Another is universal data access. Everyone inside an organization must have access to the information they need to do their jobs. Data and search applications should run across one common platform so dev teams can quickly create the custom apps that different business units and teams need.


IM: How do these themes and challenges relate to your company’s market strategy this year?

MS: Much of our strategy this year is focused on expanding the idea of what search and data applications can do – getting beyond, “It’s just a search box.” Companies are realizing that there’s the need for different apps all over the organization and that this need isn’t going to go away, but rather that it’s only going to increase. So they’d better figure out how to put systems and practices in place that allow for quick design, building, and deployment of data apps.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access