Java has a 'Commanding Lead' with App Log Data

New Relic just released its 2022 State of Logs Report, which captured the data gathered from millions of applications within the New Relic observability platform to provide an in-depth look at the use and management of logs.

The publication of a report on log data stats is not the sexiest tech news to cross my desk, but amid the Sturm und Drang of the current landscape the report's authors offer some appealingly quotidian insights into an activity that is, let's face it, critical to every business in every industry.

"With proper management and practices in place, logs have the power to help software engineers optimize the performance of systems and operations, identify and resolve technical issues, better manage resources, and strengthen security," the report's authors observed.

The report is based on petabytes of data gathered from millions of applications within the New Relic observability platform. The data was drawn entirely from applications reporting to New Relic in July 2022 and August 2022. The company anonymized and "coarse-grained" the data to provide a general overview of how logs are used and managed. "Any detailed information that could help attackers and other malicious parties was deliberately not included in the report," the authors stated.

The stat that caught my eye, of course, was this one: "When examining popularity around languages, the data shows that 50% of all logs ingested by language agents comes from Java. Java has a commanding lead over .Net (26%), Ruby (21%), Node.js (2%), and Python (0.1%)."

But the report is packed with insights into other log-related activities. For example, the authors noted a 35% year-over-year increase in logging data in general, along with a concomitant need among engineers to have access to that data.

"As the volume of log files grows, a trend is emerging with software engineers wanting to have log data available in one place to speed up the time to detecting and responding to transactions, errors, and security incidents," the report's author stated. "The practice of centralized log management was created out of the frustration and time commitment felt from software engineers in examining thousands of log files across a number of sources to pinpoint and resolve incidents. Even for relatively small companies, managing multiple logging sources and tools becomes increasingly complex, creating information silos and data that is not always adequately parsed or accessible."

Notably, Fluent Bit, the open-source logging and metrics processor and forwarder, emerged as the most used open-source tool for logs. NGINX was the most common type of log. And Firehose will soon be the de facto log forwarder for AWS serverless users.

Much more in the report, which is well worth reading New Relic also published an annual "State of the Java Ecosystem." More on that report in my next post.

Posted by John K. Waters on October 16, 2022