Elasticsearch Couples Analytics to Big Data Search

Mike Vizard
Slide Show

Top Predictions for Big Data in 2014

One of the bigger challenges with developing any kind of Big Data application is being able to identify which data is actually relevant. One of the primary tools that developers have been using to accomplish that task is an open source enterprise search and indexing engine developed by Elasticsearch that, as of this week, is now being packaged up into a formal offering.

Elasticsearch CEO Shay Banon says version 1.0 of the company’s namesake search engine adds analytics capabilities that can be federated across multiple data sources, new snapshot and restore capabilities, the ability to aggregate queries across multiple data sources, and an alert function that identifies when any data relevant to a particular query has been added to the system.

Search engines in the context of the enterprise never really took off. But with the rise of Big Data, Banon says it’s clear that developers are going to need a search engine capability that can be easily invoked through an application programming interface (API).

viz20140214-01

As an open source project, Elasticsearch is already fairly widely used. As a company, Elasticsearch is now making available support and additional functionality that developers of enterprise applications typically require. For example, Elasticsearch just released Elasticsearch Marvel, the company’s first commercial product that helps system operators monitor Elasticsearch deployments in real time, based on ElasticSearch ELK stack, which combines Elasticsearch with Logstash, open source log management software, and Kibana, open source visualization software.

As more organizations look to embrace Hadoop as a foundational component of a larger Big Data strategy, it’s clear that an ability to index and search that data is going to be critical. While there’s no shortage of enterprise search options these days, Elasticsearch has already been downloaded six million times, which suggests the community that is developing around it is becoming large enough to support enterprise-class applications.



Add Comment      Leave a comment on this blog post
Feb 14, 2014 10:52 AM Michael Michael  says:
I am surprised that Elasticsearch and the ELK stack is not viewed as a threat to Splunk. Don't they do the same thing? Elastic search is free. Splunk's expensive. Yet Splunk's market cap is over $11 billion (including options, RSUs and the recent stock issue). I don't get it. Reply
Feb 14, 2014 10:59 AM Michael Michael  says:
Why isn't the ELK stack viewed as a threat to Splunk? It's free and does the same thing as Splunk, yet Splunk is valued at over $ 10 billion! Reply
Feb 14, 2014 11:21 AM Michael Michael  says:
Shouldn't ELK stack be a threat to Splunk? Reply
Feb 15, 2014 9:53 AM sunitha sunitha  says:
Nice article, Big data is the future and is playing key role in operations. Register for Apr 4-6 Big Data Bootcamp-Austin http://globalbigdataconference.com/32/big-data-bootcamp/event.html Reply
Feb 15, 2014 5:23 PM pj pj  says:
@Michael - not sure why you would type the same thing 3 times... It isnt really a threat - at the end of the day Splunk is very easy, Elastic search and the ELK stack are more involved. You pay for splunk but do get what you pay for. You can go the free route, but then you will pay in terms of needing additional resources etc etc. Plus, Splunk is a LOT more feature rich than ELK in terms of the reporting and functionality - there is no denying that. I am sure the ELK stack works for some, but it is playing catch up and Splunk is still ahead and will likely remain ahead. Reply
Apr 25, 2014 10:12 AM kamal kamal  says:
ELK is not in mature state like splunk , splunk having many features and easy to use .ELK required more resources so if we say splunk is expensive then ELK is also expensive as it requires more resources. Reply
Dec 2, 2014 6:22 PM Mike Mike  says:
We've had to deploy splunk at my current job and it has been a pita. We are 5months in and still do not have it setup the way we need it. So all that time we have been paying splunk big $ plus we have been paying big money on engineering hours for all the folks involved in this. So in my book Splunk is still more expensive. Besides with ELK stack you can pay for support from some vendors that will still be cheaper than Splunk. It also depends on what you want to do with your logs. I've setup systems that integrate with monitoring and job schedulers via API from Graylog2 and Logstash to nagios and jira for creating automated jira tickets. Reply

Post a comment

 

 

 

 


(Maximum characters: 1200). You have 1200 characters left.

 

 

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


 
Resource centers

Business Intelligence

Business performance information for strategic and operational decision-making

SOA

SOA uses interoperable services grouped around business processes to ease data integration

Data Warehousing

Data warehousing helps companies make sense of their operational data


Thanks for your registration, follow us on our social networks to keep up-to-date