The 7 Log Management Tools Java Developers Should Know

splunk vs. sumo logic vs. logstash vs. graylog vs. loggly vs. papertrails vs. splunk>storm

splunk, sumo logic, logstash, graylog, loggly, papertrails - did i miss someone? i’m pretty sure i did. logs are like fossil fuels - we’ve been wanting to get rid of them for the past 20 years, but we’re not quite there yet. well, if that's the case i want a bmw! to deal with the growth of log data a host of log management & analysis tools have been built over the last few years to help developers and operations make sense of the growing data. i thought it’d be interesting to look at our options and what are each tools’ selling point, from a developer’s standpoint .

splunk

as the biggest tool in this space, i decided to put splunk in a category of its own. that’s not to say it’s the best tool for what you need, but more to give credit to a product who essentially created a new category.

pros

splunk is probably the most feature rich solution in the space. it’s got hundreds of apps (i counted 537 ) to make sense of almost every format of log data, from security to business analytics to infrastructure monitoring. splunk’s search and charting tools are feature rich to the point that there’s probably no set of data you can’t get to through its ui or apis.

cons

splunk has two major cons. the first, that is more subjective, is that it’s an on-premise solution which means that setup costs in terms of money and complexity are high. to deploy in a high-scale environment you will need to install and configure a dedicated cluster. as a developer, it’s usually something you can't or don’t want to do as your first choice. splunk’s second con is that it’s expensive. to support a real-world application you’re looking at tens of thousands of dollars, which most likely means you’ll need sign offs from high-ups in your organization, and the process is going to be slow. if you’ve got a new app and you want something fast that you can quickly spin up and ramp as things progress - keep reading. some more enterprise log analyzers can be found here .

saas log analyzers

sumo logic

sumo was founded as a saas version of splunk, going so far as to imitate some of splunk’s features and visuals early on. having said that, sl has developed to a full fledged enterprise class log management solution.

pros

sl is chock-full of features to reduce, search and chart mass amounts of data. out of all the saas log analyzers, it’s probably the most feature rich. also, being a saas offering it inherently means setup and ongoing operation are easier. one of sumo logic’s main points of attraction is the ability to establish baselines and to actively notify you when key metrics change after an event such as a new version rollout or a breach attempt.

cons

this one is shared across all saas log analyzers, which is you need to get the data to the service to actually do something with it. this means that you’ll be looking at possible gbs (or more) uploaded from your servers. this can create issues on multiple fronts -

1. as a developer, if you're logging sensitive or pii you need to make sure it’s redacted.
2. there may be a lag between the time data is logged and the time it’s visible to to the service.
3. there’s additional overhead on your machines transmitting gbs of data, which really depends on your logging throughput.

sumo’s pricing is also not transparent , which means you might be looking at a buying process which is more complex than swiping your team’s credit card to get going.

loggly

loggly is also a robust log analyzer, focusing on simplicity and ease of use for a devops audience.

pros

whereas sumo logic has a strong enterprise and security focus, loggly is geared more towards helping devops find and fix operational problems. this makes it very developer-friendly. things like creating custom performance and devops dashboards are super-easy to do. pricing is also transparent, which makes start of use easier.

cons

don't expect loggly to scale into a full blown infrastructure, security or analytics solution. if you need forensics or infrastructure monitoring you’re in the wrong place. this is a tools mainly for devops to parse data coming from your app servers. anything beyond that you’ll have to build yourself.

papertrails

papertrails is a simple way to look and search through logs from multiple machines, in one consolidated easy-to-use interface. think of it like tailing your log in the cloud, and you won't be too far off.

pros

pt is what it is. a simple way to look at log files from multiple machines in a singular view in the cloud. the ux itself is very similar to looking at a log on your machine, and so are the search commands. it aims to do something simple and useful, and does it elegantly. it’s also very affordable .

cons

pt is mostly text based. looking for any advanced integrations, predictive or reporting capabilities? you're barking up the wrong tree.

splunk>storm

this is splunk’s little (some may say step) saas brother. it’s a pretty similar offering that’s hosted on splunk’s servers.

pros

storm lets you experiment with splunk without having to install the actual software on-premise, and contains much of the features available in the full version.

cons

this isn't really a commercial offering, and you're limited in the amount of data you can send. it seems to be more of an online limited version of splunk meant to help people test out the product without having to deploy first. a new service called splunk cloud is aimed at providing a full-blown splunk saas experience.

open source analyzers

logstash

logstash is an open source tool for collecting and managing log files. it’s part of an open-source stack which includes elasticsearch for indexing and searching through data and kibana for charting and visualizing data. together they form a powerful log management solution.

pros

being an open-source solution means you're inherently getting a lot of a control and a very good price. logstash uses three mature and powerful components, all heavily maintained, to create a very robust and extensible package. for an open-source solution it’s also very easy to install and start using. we use logstash and love it.

cons

as logstash is essentially a stack, it means you're dealing with three different products. that means that extensibility also becomes complex. logstash filters are written in ruby, kibana is pure javascript and elasticsearch has its own rest api as well as json templates. when you move to production, you’ll also need to separate the three into different machines, which adds to the complexity.

graylog2

a fairly new player in the space, gl2 is an open-source log analyzer backed by mongodb as well as elasticsearch (similar to logstash) for storing and searching through log errors. it’s mainly focused on helping developers detect and fix errors in their apps. also in this category you can find fluentd and kafka whose one of its main use-cases is also storing log data. phew, so many choices!

takipi for logs

while this post is not about takipi, i thought there’s one feature it has which you might find relevant to all of this. the biggest disadvantage in all log analyzers and log files in general, is that the right data has to be put there by you first. from a dev perspective, it means that if an exception isn’t logged, or the variable data you need to understand why it happened isn't there, no log file or analyzer in the world can help you. production debugging sucks. one of the things we’ve added to takipi is the ability to jump into a recorded debugging session straight from a log file error. this means that for every log error you can see the actual source code and variable values at the moment of error. you can learn more about it here . this is one post where i would love to hear from you guys about your experiences with some of the tools mentioned (and some that i didn’t). i’m sure there are things you would disagree with or would like to correct me on - so go ahead, the comment section is below and i would love to hear from you.

originally posted on takipi blog