Sending Mule Application logs to Splunk

What is Splunk?

Splunk is known for its capabilities in log management, security information and event management (SIEM), and data analytics.

Why there is a need to send logs to External Systems?

CloudHub has a specific log retention policy, the platform stores logs of up to 100 MB per app & per worker or for up to 30 days, whichever limit is hit first also MuleSoft doesn’t have any facility to analyze and generate reports out of the logs so in order to retain application logs for longer time and more defined alerting and analyzing capabilities, these logs are being sent to external systems such as Splunk.

We can send both On Prem and Cloudhub logs to Splunk.

In this use case, I have shared steps of creating an index in Splunk and sending application logs to Splunk index:

Create Splunk Enterprise trial account and download Splunk.

Go to Settings> Data Inputs as shown below.

Click on HTTP Event Collector as shown below:

Click on New Token:

Enter Name and click Next:

Click select and select log4j and click on Review as shown below:

Click Submit

Now under Data Inputs> Http Event Collector, created token is visible as shown below with Token Value, which will be needed in Mule application:

I have created a simple flow for POC as shown below:

Now add, below dependency to application’s POM.xml:

<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-core</artifactId>
  <version>2.17.0</version>
</dependency>
<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-api</artifactId>
  <version>2.17.0</version>
</dependency>
<dependency>
  <groupId>com.splunk.logging</groupId>
  <artifactId>splunk-library-javalogging</artifactId>
  <version>1.11.4</version>
</dependency>

Add below repository to POM.xml:

        <repository>
        <id>splunk-artifactory</id>
        <name>Splunk Releases</name>
        <url>https://splunk.jfrog.io/splunk/ext-releases-local</url>
        </repository>

Add cofiguration, appender and Asynclogger and highlighted below, we will use index token value created by us earlier under property value:

<?xml version="1.0" encoding="utf-8"?>
<Configuration packages="com.mulesoft.ch.logging.appender,com.splunk.logging.org" status="INFO">

<Appenders>
<Http name="Splunk" url="http://localhost:8088/services/collector/raw" >
<Property name="Token" value="Splunk 5ed04a2c-8221-45e9-9708-7d7b710b0fd7"> </Property>
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n"></PatternLayout>
</Http>

</Appenders>

 <Loggers>
  <!-- Http Logger shows wire traffic on DEBUG -->
  <!--AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger" level="DEBUG"/-->
  <AsyncLogger name="org.mule.service.http" level="WARN"/>
  <AsyncLogger name="org.mule.extension.http" level="WARN"/>
  <!-- Mule logger -->
   <AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>
   <AsyncRoot level="INFO">
   <AppenderRef ref="splunk" />
   </AsyncRoot>
   </Loggers>
</Configuration>

Now Run your application and in Splunk click “Search & Reporting”

Under search : search using index reference as index=”Muleapplog”