Docker 📅 2026-02-07

Docker Stop Command Exit Code 137: Troubleshooting SIGKILL and Container Shutdown Failures

The docker stop command, when confronted with unresponsive containers or insufficient resources, can abruptly terminate your processes with an exit code 137. This code signals a SIGKILL (signal 9) from the Linux kernel, often indicating an Out-Of-Memory (OOM) event or a failure to gracefully shut down within the default timeout. For SREs managing containerized deployments, diagnosing and resolving this issue is critical for system stability and performance.

🚨 Symptoms & Diagnosis¶

When a container exits with code 137, it means the process was forcefully terminated, likely due to a system-level intervention rather than an application-level graceful shutdown.

exited (137)
Exit Code 137
State: { "ExitCode": 137, "OOMKilled": false }

You might also observe these related log entries:

docker logs: Killed process <pid> (containerd) total-vm:<size>, anon-rss:<size>
dmesg: Out of memory: Killed process <pid> (<exec>) score <score> or sacrifice child

To quickly verify the exit code:

docker inspect <container> --format='{{.State.ExitCode}}'
# Expected output: 137

Root Cause: Exit code 137 typically arises from the Linux kernel's Out-Of-Memory (OOM) killer terminating a process due to host memory exhaustion, or a docker stop command timing out (default 10s) and triggering a SIGKILL after an initial SIGTERM fails to elicit a graceful shutdown.

🛠️ Solutions¶

Immediate Mitigation: Increase Memory & Force Graceful Stop¶

Immediate Mitigation: Increase Memory & Force Graceful Stop

This approach aims to quickly resolve recurring 137 errors by providing more memory to the Docker environment or the specific container, and extending the graceful shutdown period to prevent premature SIGKILLs.

Check Current Resource Usage: First, observe the memory and CPU usage of your containers to identify potential bottlenecks.
```
docker stats
```
Increase Docker Daemon/Host Memory Allocation: Depending on your Docker environment, increase the overall memory available to Docker.

Docker Desktop (Windows/macOS)Linux Host

Navigate to Preferences/Settings > Resources > Memory and increase the allocated RAM (e.g., to 4GB or more).

Resource Allocation Impact

Increasing Docker Desktop's memory allocation impacts your host machine's available RAM. Allocate judiciously to avoid host performance degradation.

For containers running directly on a Linux host, ensure the host itself has sufficient physical RAM. If your host is frequently hitting OOM conditions, you may need to add more physical RAM or reduce the total memory footprint of all running services.

Update Container Memory Limits and Extend Stop Timeout: Allocate more memory to the problematic container and give it more time to shut down gracefully.

# Update an existing container's memory limits
docker update --memory=2g --memory-swap=4g <container_id_or_name>

# Stop the container with an extended timeout (e.g., 30 seconds)
docker stop --time=30 <container_id_or_name>

# Restart the container
docker start <container_id_or_name>

Best Practice Fix: Tune Limits, Monitor OOM, Log Parsing¶

Best Practice Fix: Tune Limits, Monitor OOM, Log Parsing

For production environments, a robust strategy involves proactive resource limit configuration, kernel-level OOM monitoring, and thorough log analysis to prevent exit code 137 occurrences.

Inspect Failed Container State: Examine the container's final state for specific indicators of the shutdown failure. The OOMKilled flag might not always be true even if OOM was the cause, as the host OOM killer acts independently of Docker's cgroup-based OOM.
```
docker inspect <container_id_or_name> --format='{{json .State}}' | jq '.ExitCode, .OOMKilled'
```

Check Kernel OOM Logs: The most definitive way to confirm an OOM kill by the kernel is to check system logs. These commands require sudo privileges.

# Display the last 5 entries related to killed processes
sudo dmesg | grep -i 'killed process' | tail -5

# Filter journalctl for kernel messages related to OOM
sudo journalctl -k | grep -i oom

Configure Robust Resource Limits: Apply appropriate memory limits and reservations when creating containers. This prevents a single container from starving the host or other critical services.
Docker runDocker Compose (v3+)
docker run -d \ --memory=1g \ --memory-swap=2g \ --oom-kill-disable=false \ <image_name>
* --memory: Hard limit for memory. * --memory-swap: Total memory (RAM + swap). Setting swap higher than memory allows the container to use swap when RAM is full, preventing immediate OOM. * --oom-kill-disable=false: Explicitly enables the OOM killer for this container (default behavior). Set to true only for critical processes that must never be OOM-killed, even at the risk of crashing the host.
services: app: image: your-app-image deploy: resources: limits: memory: 1G reservations: memory: 512M # Guarantees this amount of RAM will be available
* limits.memory: The maximum amount of memory the container can use. * reservations.memory: The amount of memory guaranteed to the container. This helps prevent other containers from encroaching on essential resources.

Add Healthchecks to Applications: Implement application-level health checks (e.g., HTTP endpoints) to ensure the container is responsive. If docker stop times out, it might be because the application itself is frozen.

services:
  app:
    image: your-app-image
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 5s

Monitor Container Stats: Continuously monitor your container's resource usage to identify trends and potential memory leaks before they lead to OOM events.
```
# Watch Docker stats in real-time (updates every 5 seconds)
watch -n 5 'docker stats --no-stream'
```

🧩 Technical Context (Visualized)¶

The exit code 137 specifically signifies that a process was terminated by signal 9 (SIGKILL). This signal cannot be caught or ignored by the process, ensuring its immediate termination. In Docker's context, this most commonly occurs when: 1. The container process consumes too much memory, triggering the Linux kernel's Out-Of-Memory (OOM) killer to terminate it to protect the host system. 2. A docker stop command is issued, sending a SIGTERM (signal 15) to the container. If the application does not gracefully shut down within the default 10-second timeout, Docker escalates to a SIGKILL to force termination.

graph TD
    A[Docker Container Process Running] --> B{High Memory Usage?};
    B -- Yes --> C[Linux Kernel OOM Killer Activates];
    C -- "> D["SIGKILL (Signal 9) Sent"];
    D" --> E[Process Terminated];
    E --> F[Container Exits with Code 137];

    A --> G{docker stop Issued};
    G -- "Sends SIGTERM (Signal 15)" --> H["Container Graceful Shutdown Timeout (default 10s)"];
    H -- Timeout Exceeded --> I["SIGKILL (Signal 9) Sent"];
    I --> E;

✅ Verification¶

After applying the solutions, verify that your containers start, stop, and run without encountering exit code 137.

Check Exit Code After Stop/Start: Confirm that the container now exits cleanly (code 0) upon a controlled stop.

docker stop --time=30 <container_id_or_name> && docker start <container_id_or_name> && \
docker inspect <container_id_or_name> --format='{{.State.ExitCode}}' | grep -q 0 && \
echo 'Success: Container stopped gracefully.' || echo 'Failed: Container still exiting with non-zero code.'

Monitor Live Stats: Observe resource consumption to ensure containers stay within their allocated limits.
```
docker stats | grep <container_id_or_name>
```

Check Kernel Logs for OOM Events: Ensure no new OOM kill messages appear after your changes.

sudo dmesg | tail -1 | grep -v 'killed'
# Expected output should NOT contain "killed process"

📦 Prerequisites¶

To effectively troubleshoot and resolve Docker exit code 137, ensure you have: * Docker 20.10+ * docker-compose 2.0+ (if using Compose) * jq for JSON parsing (optional, but highly recommended) * root or sudo privileges for dmesg and journalctl commands. * Docker Desktop (if applicable) with sufficient host resources, or a Linux host with 8GB+ RAM.