ELK Stack for Centralized Logging — Stop Hunting Logs, Start Debugging Fast

Your server is a locked cabinet. When something breaks — a 500 error, a memory leak, an attack — you don't have time to dig through log files scattered across three machines. Every minute lost searching is a customer lost. At Meteora Web, we've seen too many companies waste hours on manual debugging. With the ELK Stack, logs become a central database, searchable and visualized in real time. Here's how it really works.

What is the ELK Stack and why centralize logs?

ELK stands for Elasticsearch, Logstash, Kibana. Three tools that work together to collect, process and visualize logs and metrics from any source: servers, applications, databases, network devices.

The problem: logs in scattered files, different formats per application, no temporal correlation. An error in a PHP service might depend on a MySQL timeout, but if each log sits on a different machine, you won't see the connection.

The solution: a single pipeline that collects everything into Elasticsearch (search and storage engine), transforms it with Logstash (data processor) and makes it visible in Kibana (analysis interface). Result: debugging in seconds, not hours.

A real case we handled

A client had a WooCommerce site crashing every evening. No visible logs. We configured Filebeat to send PHP and Apache logs to a central Logstash. Within minutes we saw a pattern: allowed memory size exhausted at 10 PM. A heavy cron job. Fixed in one hour. Without ELK it would have been a treasure hunt.

How does Elasticsearch handle log storage and search?

Elasticsearch is a NoSQL database built on Apache Lucene. It is not a relational database: no rigid schemas. JSON documents arrive and are indexed automatically. For logs it's perfect: each log line becomes a document with dynamic fields (@timestamp, level, message, host, etc.).

Real performance: Elasticsearch can handle hundreds of millions of documents with sub-second full-text searches. We have used it to index over 500 GB of logs per day on a small 3-node cluster.

Basic mapping for a log index

You don't have to define mappings manually, but for optimization an explicit mapping for numeric or geo fields prevents runtime errors. Example mapping for a response_time field in milliseconds:

{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "response_time": { "type": "integer" },
      "status": { "type": "keyword" },
      "message": { "type": "text" }
    }
  }
}

Apply it via a PUT request to /index_name. Without mapping, Elasticsearch assigns default types — it works, but for fields like IP or status you often need keyword instead of text.

How does Logstash transform log data?

Logstash is the pipeline. It takes input from files, sockets, databases, transforms data with filters (grok, mutate, date) and sends it to Elasticsearch or other outputs.

Why use it? Raw logs are messy: different timestamp formats, mixed fields, multiline messages (stack traces). Logstash normalizes them. We've seen Apache logs with date 03/May/2025:12:00 become 2025-05-03T12:00:00 in seconds.

Basic Logstash configuration for Nginx

input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][log_type] == "nginx" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
    mutate {
      convert => ["response", "integer"]
      convert => ["bytes", "integer"]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "nginx-logs-%{+YYYY.MM.dd}"
  }
}

Warning: Logstash can become a bottleneck. We scale it by distributing load with Kafka or Redis as buffers. For huge volumes, consider Elastic Agent (the next-gen tool) that integrates Filebeat and Metricbeat.

How does Kibana visualize and analyze logs?

Kibana is the dashboard. It's not just for viewing logs: it does aggregated analysis, alerting, dashboards. Search for HTTP 5xx errors, group by URL, see average latency over time.

Concrete example: A client wanted to know how many Italian visitors they had and which pages caused the most crashes. With Kibana we created a dashboard showing a session map (based on geo-IP) and a histogram of error codes. Decision made: optimize error-prone pages before launching an Ads campaign.

Lucene query to find real-time errors

# In Kibana search field:
response:[500 TO 599] AND host:"web-01"

You can save these as Saved Searches and use them for alerts (with Watcher or ElastAlert).

What is a typical ELK pipeline configuration?

A complete centralized logging pipeline has four components:

1. Agents (Beats) — Filebeat, Metricbeat, Winlogbeat. Installed on each server, send logs to.
2. Logstash — Centralizes and transforms. You can have one or more in a cluster.
3. Elasticsearch — Stores and indexes. Single node for testing, cluster for production.
4. Kibana — Web interface to explore and build dashboards.

Our recommendation: for at least 3 servers, use a two-node Elasticsearch cluster and a dedicated Logstash. For high volume, add Kafka as a queue. Never run Logstash and Kibana on the same server as Elasticsearch in production — memory is precious.

Example docker-compose for development

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ports:
      - "9200:9200"
  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    environment:
      - LS_JAVA_OPTS=-Xms512m -Xmx512m
  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200

Note: in production always enable HTTPS and authentication. The base ELK images are secure by default from version 8.0.

What to do next

1. Install Filebeat on one of your production servers and send system logs (syslog, auth.log). Follow the official Elastic guide.
2. Create a temporary index on Elasticsearch with mapping for numeric fields (response time, status code).
3. Configure Logstash with a grok filter for your log format (Apache, Nginx, application logs).
4. Build a Kibana dashboard to visualize HTTP errors and slow responses.
5. Set up an alert (using built-in watcher or ElastAlert) for recurring errors — get notified as soon as they happen.
6. Read our parent guide on monitoring and observability to place ELK in the broader context.