Stop Paying for Hot Storage: The Ultimate Logrotate + LogLens Workflow

In modern DevOps, "Observability" often equates to "expensive monthly bills." Tools like Datadog, Splunk, or CloudWatch charge heavily for what is known as Hot Storage.

What is Hot Storage?

Hot storage refers to data that is indexed, kept in memory (RAM) or on high-speed SSDs, and ready to be queried instantly. When you send logs to the cloud, the provider parses every line, builds a massive index (like a book's index but for every word in your log), and stores it on expensive hardware.

The problem? 99% of your logs are write-only. You generate them, pay to index them, pay to store them, and then they are deleted 30 days later without a human ever looking at them. You are paying a premium for speed you rarely use.

This post outlines a "Cold Storage" workflow that feels like Hot Storage. We use logrotate (a standard Linux utility) to manage files efficiently, and LogLens to analyze them without the indexing tax.

The Architecture: Local, Compressed, and Queryable

Instead of shipping logs to a third party, we keep them local (or on cheap S3 storage) and process them in three stages:

1. Active Writing: The application writes to a live file (e.g., app.log).
2. Rotation & Compression: A background process renames the file and compresses it using Gzip to save space.
3. Analysis: We use LogLens to query these files directly, effectively treating compressed files as a database.

Step 1: Configure Logrotate for Efficiency

Logrotate is the standard "archivist" utility built into almost every Linux distribution. It runs automatically in the background to ensure your disk never fills up with stale data.

How it works:

Think of your log file like a bucket filling with water. When the bucket gets full (or a specific amount of time passes), Logrotate swaps it for an empty one and puts a lid on the full one.

Technically, it performs a "rename cascade":

• app.log.2.gz → Deleted (if retention limit is reached).
• app.log.1 → Compressed to app.log.2.gz.
• app.log → Renamed to app.log.1.
• A new, empty app.log is created.

Here is a battle-tested manual configuration. Create this file at /etc/logrotate.d/myapp:

/var/log/myapp/*.log {
    daily                     # Rotate files once every 24 hours
    size 100M                 # OR rotate if file exceeds 100MB
    missingok                 # Don't throw an error if the log file is missing
    rotate 10                 # Keep 10 archived files
    compress                  # Gzip the rotated files (saves ~90% disk space)
    delaycompress             # Leave yesterday's file uncompressed for 24h
    notifempty                # Don't rotate if the file is 0 bytes
    create 0640 www-data www-data # Permission for the new file
    sharedscripts
    postrotate
        # CRITICAL: Tell the app to reload its file handle
        systemctl reload myapp
    endscript
}

Understanding the Configuration:

• compress: This is the money-saver. Text logs compress extremely well (often 10:1 or 20:1 ratios). A 10GB log file becomes 500MB on disk, meaning you can keep months of history for pennies.
• delaycompress: This is a pro-tip. It keeps app.log.1 uncompressed. If you need to debug an issue from "just a few hours ago," you don't need to decompress anything.
• postrotate: This is the most important part. Linux applications write to inodes (file IDs), not filenames. If you rename app.log to app.log.1, your app will simply keep writing to app.log.1. The `postrotate` script sends a signal (usually SIGHUP) to the application, telling it: "Close your current file handle and open the new app.log we just created."

Step 2: The "Transparent" Search Workflow

Historically, searching compressed logs was painful. You had to use zcat (to decompress to stdout) pipe it into grep, and lose all your syntax highlighting. It was slow and cumbersome.

LogLens solves this with native streaming decompression.

When you run a command, LogLens detects the .gz extension. It streams the compressed bytes into memory, decompresses them on the fly, and runs your query logic before printing to the screen. No temporary files are created on disk.

Scenario: A user reports a payment error that happened "sometime yesterday."

# Search specifically in yesterday's compressed archive
# LogLens handles the Gzip decompression transparently
loglens search /var/log/myapp/app.log.2.gz "User checkout failed" --json

Or, use a Glob Pattern to search the live file AND 30 days of history simultaneously:

# '*' matches app.log, app.log.1, app.log.2.gz, etc.
loglens search /var/log/myapp/* "payment_gateway_timeout"

Step 3: Statistical Analysis on "Cold" Data

The main reason teams stick with expensive SaaS tools is for Aggregation. It's easy to ask Datadog "What is the average latency by endpoint?". Doing that with grep or awk in the terminal is difficult and error-prone.

With LogLens Pro, you can perform these aggregations locally. LogLens acts as an ad-hoc analytics engine. It reads the structured JSON (or Nginx format) from your compressed files and calculates statistics in memory.

# Calculate average latency for yesterday, grouped by API endpoint
# Reading directly from the compressed file on disk
loglens stats group-by /var/log/myapp/app.log.2.gz \
    --by endpoint \
    --avg latency_ms \
    --where 'status >= 200'

Step 4: Live Tailing with Logic

While logrotate handles history, loglens watch replaces the cloud "Live Tail" view. The standard Linux tail -f command dumps everything to the screen, which is unreadable on high-traffic servers.

LogLens adds a Query Layer on top of the tail. It parses every new line as it is written, evaluates your logic, and only prints matches.

# Watch live logs, but FILTER them before they hit your screen
loglens watch /var/log/myapp/app.log \
    --where 'level == "error" || latency_ms > 1000' \
    --highlight "user_id"

Smart Watching: If logrotate runs while you are watching, standard tools might stop receiving data. LogLens detects the file rotation event (via filesystem watchers) and automatically attaches to the new file handle, ensuring zero downtime in your monitoring.

The ROI Calculation

Let's compare the costs for a service generating 100GB of logs per day.

Conclusion: Simplify Your Stack

Complexity is the enemy of reliability. Cloud logging tools are fantastic for massive scale, but for many startups and mid-sized deployments, they are overkill.

By returning to the robustness of standard Linux logging (Logrotate) and augmenting it with the modern analysis capabilities of LogLens (Structured Querying, Stats, TUI), you build a logging pipeline that is cheaper, private, and completely under your control.