20250102 ‐ elastic - cywongg/2025 GitHub Wiki

---

### **1. Exporting Spring Batch Job Run Status to Elasticsearch**

Spring Batch provides metadata about job executions, including the job's status, start time, end time, etc. Here's how you can export this directly to Elasticsearch:

#### **Steps:**
1. **Leverage Spring Batch Listeners**: Use `JobExecutionListener` or `StepExecutionListener` to capture job and step status events and send them to Elasticsearch.

2. **Add Elasticsearch Dependency**:
   Add the Elasticsearch client library to your project. For example, using Maven:
   ```xml
   <dependency>
       <groupId>org.springframework.data</groupId>
       <artifactId>spring-data-elasticsearch</artifactId>
       <version>4.5.0</version> <!-- Adjust the version as needed -->
   </dependency>

3. Implement Job Listener: Create a custom listener to extract job status and send it to Elasticsearch.

   import org.springframework.batch.core.JobExecution;
   import org.springframework.batch.core.listener.JobExecutionListenerSupport;
   import org.springframework.stereotype.Component;
   import org.elasticsearch.client.RestHighLevelClient;
   import org.elasticsearch.action.index.IndexRequest;
   import org.elasticsearch.common.xcontent.XContentType;
   
   @Component
   public class ElasticsearchJobListener extends JobExecutionListenerSupport {
   
       private final RestHighLevelClient elasticsearchClient;
   
       public ElasticsearchJobListener(RestHighLevelClient elasticsearchClient) {
           this.elasticsearchClient = elasticsearchClient;
       }
   
       @Override
       public void afterJob(JobExecution jobExecution) {
           try {
               IndexRequest request = new IndexRequest("spring_batch_jobs")
                   .source("{\"jobName\":\"" + jobExecution.getJobInstance().getJobName() +
                               "\",\"status\":\"" + jobExecution.getStatus() + 
                               "\",\"startTime\":\"" + jobExecution.getStartTime() +
                               "\",\"endTime\":\"" + jobExecution.getEndTime() + "\"}",
                           XContentType.JSON);
               elasticsearchClient.index(request, RequestOptions.DEFAULT);
           } catch (Exception e) {
               e.printStackTrace();
           }
       }
   }

4. Configure Elasticsearch: Configure the RestHighLevelClient bean in your Spring configuration.

   @Configuration
   public class ElasticsearchConfig {
   
       @Bean
       public RestHighLevelClient elasticsearchClient() {
           return RestClient.builder(new HttpHost("localhost", 9200, "http")).build();
       }
   }

5. Create a Dashboard in Kibana: Once the data is indexed in Elasticsearch (e.g., under the spring_batch_jobs index), you can create visualizations and dashboards in Kibana to monitor job statuses.

Kubernetes exposes batch job statuses via its API, which can be forwarded to Elasticsearch for visualization.

1. Enable Kubernetes API Access: Use the Kubernetes client library or kubectl commands to fetch job statuses. For example:

   • Using the Kubernetes CLI:

     kubectl get jobs -o json

   • Using the Kubernetes Java Client:

     import io.kubernetes.client.openapi.ApiClient;
     import io.kubernetes.client.openapi.apis.BatchV1Api;
     import io.kubernetes.client.util.Config;
     
     public class KubernetesJobStatusExporter {
     
         public static void main(String[] args) throws Exception {
             ApiClient client = Config.defaultClient();
             BatchV1Api batchApi = new BatchV1Api(client);
     
             var jobs = batchApi.listJobForAllNamespaces(null, null, null, null, null, null, null, null, null, null);
             jobs.getItems().forEach(job -> {
                 System.out.println("Job Name: " + job.getMetadata().getName() +
                                    ", Status: " + job.getStatus().getSucceeded());
             });
         }
     }

2. Index Kubernetes Job Status in Elasticsearch: Write a script or service to fetch job statuses periodically and send them to Elasticsearch. For example, transform the kubectl get jobs -o json output into JSON and index it using a REST client.

   Example using Python and elasticsearch-py:

   import subprocess
   import json
   from elasticsearch import Elasticsearch
   
   def get_kube_jobs():
       result = subprocess.run(['kubectl', 'get', 'jobs', '-o', 'json'], stdout=subprocess.PIPE)
       return json.loads(result.stdout)
   
   def send_to_elasticsearch(jobs):
       es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
       for job in jobs['items']:
           doc = {
               'jobName': job['metadata']['name'],
               'namespace': job['metadata']['namespace'],
               'status': job['status']
           }
           es.index(index='kube_jobs', body=doc)
   
   jobs = get_kube_jobs()
   send_to_elasticsearch(jobs)

3. Create a Dashboard in Kibana: As with Spring Batch, use the data in Elasticsearch to build visualizations and dashboards in Kibana.

• If you are primarily working with Spring Batch: Use the Spring Batch listener approach as it directly integrates with your batch jobs without requiring external tools or significant changes.
• If you need to monitor Kubernetes jobs in general: Use the Kubernetes API or kubectl approach, which provides a broader view of all Kubernetes jobs, including non-Spring Batch jobs.

By integrating both into Elasticsearch, you can create a unified monitoring system for both Spring Batch and Kubernetes jobs. This minimizes effort while maximizing visibility.