Category:

Linux Administration

Linux Administration presents a collection of common issues and useful tips for Linux system administration. Whether you’re new to system administration or have been maintaining systems for some time, we hope these tips are helpful regardless of your background or choice in Linux distributions.

Command Line Interface (CLI)

Out of Memory OOM Killer issue Investigation

by blackMORE April 30, 2025

written by blackMORE

Wednesday morning our monitoring solution suddenly alerted me to a critical issue with our main web server. Attempting to access our WordPress website via browser returned database connection errors, indicating a problem with our hosting environment. When I logged into the server via SSH, I discovered it’s a Linux Out of Memory (OOM) Killer terminating our MariaDB database service due to memory resource constraints. Below is my investigation into this incident revealed valuable insights about memory allocation, resource management, and strategies to prevent unwanted OOM Killer interventions in WordPress environments. Also, the server is very poorly spected on purpose so see how much we can push on resources. You will there’s other posts I’ve done recently that kind of goes around resource monitoring, restarting services etc.

If you’ve been following our series on server management and optimisation, you’ll know we’ve been experimenting with deliberately under-resourced servers to explore the limits of performance tuning. Our recent articles have covered:

Today’s deep dive into OOM Killer management builds upon these previous explorations, showing how to maintain stability even when deliberately pushing resource boundaries.

Identifying OOM Killer Activity

After logging into our Linux server around 9:30 AM on 23 April 2025, my first troubleshooting step was examining system logs for clues about the database failure:

cd /var/log
grep -i "error\|fail\|killed" syslog | tail -20

The telltale signature of the OOM Killer immediately appeared:

systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
systemd[1]: mariadb.service: Failed with result 'oom-kill'.

These entries confirmed that MariaDB had been terminated by the OOM Killer mechanism, explaining our sudden database outage. But to implement proper solutions, I needed to understand what triggered the OOM Killer and how to prevent future incidents.

Investigating OOM Issues and Killer Behaviour

To comprehensively understand the events leading to OOM Killer activation, I extracted relevant syslog entries from around the incident time:

cat syslog | grep '2025-04-23T09' | egrep -v 'action-8-builtin'

This filtered log entries from 9 AM on 23 April, excluding noise from routine system tasks. The syslog revealed a clear sequence of memory allocation failures and resource constraints that triggered the OOM Killer.

Timeline of OOM Events and Killer Activation

9:23 AM: Early Warning Signs

The first warning signals appeared at 9:23 AM:

mariadbd[21566]: 2025-04-23 9:23:02 11749 [Warning] Aborted connection 11749 to db: 'wordpress_db' user: 'wordpress_user' host: 'localhost' (Got timeout reading communication packets)

Multiple database connection timeout errors occurred within a short timeframe. Then a critical memory management message appeared:

mariadbd[21566]: 2025-04-23 9:23:02 0 [Note] InnoDB: Memory pressure event freed 12409 pages

This indicated MariaDB’s InnoDB storage engine had detected memory pressure and was actively trying to reclaim memory by releasing its page cache—attempting to avoid triggering the OOM Killer.

System-wide memory issues soon followed:

systemd-journald[126]: Under memory pressure, flushing caches

This confirmed the entire system was experiencing significant memory pressure, forcing critical services to release cached memory to prevent OOM Killer activation.

9:24 AM: OOM Killer Activation and Process Termination

Just one minute later, the system reached a critical memory threshold triggering the OOM Killer:

kernel: php-fpm8.4 invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0

A PHP-FPM worker process attempted to allocate memory but failed due to insufficient resources, triggering the kernel’s OOM Killer mechanism. The kernel systematically examined all running processes to determine which to terminate.

After evaluating memory footprints, the OOM Killer selected its target:

kernel: Out of memory: Killed process 21566 (mariadbd) total-vm:2386824kB, anon-rss:380840kB, file-rss:8448kB, shmem-rss:0kB, UID:111 pgtables:1668kB oom_score_adj:0

The MariaDB database process was terminated because it consumed significant memory resources (approximately 380MB of physical RAM and 2.3GB of virtual memory). The OOM Killer determined this would free sufficient memory to maintain overall system stability and prevent a complete crash.

9:51 AM: Database Service Recovery

After investigating the OOM Killer event and implementing initial memory optimisations, I restarted the MariaDB service:

systemctl start mariadb.service

System logs confirmed successful recovery:

systemd[1]: Started mariadb.service - MariaDB 11.4.5 database server.

This restored database functionality to our WordPress site, but without addressing underlying memory issues, another OOM Killer intervention was inevitable.

Root Cause Analysis

The kernel memory information provided critical insights into our server’s memory allocation state:

Mem-Info:
active_anon:280016 inactive_anon:141866 isolated_anon:0
active_file:157 inactive_file:5 isolated_file:0
unevictable:8954 dirty:0 writeback:0
slab_reclaimable:9184 slab_unreclaimable:20897
mapped:27465 shmem:51178 pagetables:9186
free:18913 free_pcp:1985 free_cma:0
Free swap = 0kB
Total swap = 0kB

I immediately identified a critical configuration issue: Total swap = 0kB. Our server had no swap space configured whatsoever. This removed an essential safety buffer for memory allocation spikes, making the system vulnerable to OOM conditions.

Additionally, over 50 PHP-FPM worker processes were running simultaneously, each consuming significant memory resources. This excessive number explained the severe memory pressure leading to the OOM condition.

The pattern of database connection timeouts provided another clue – our WordPress application appeared to have connection handling inefficiencies, potentially causing connection leaks that increased memory consumption across the system.

Implementing Solutions to Prevent OOM Killer Events

Based on my analysis, I implemented several targeted changes to optimise memory management:

1. Adding Swap Space

The first and most critical task was configuring swap space to provide an essential memory allocation buffer:

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

I determined that 4GB was optimal for our WordPress server’s workload patterns. This swap configuration provides the kernel with additional virtual memory resources when physical RAM becomes constrained, helping prevent OOM conditions during traffic spikes.

2. Optimising PHP-FPM Memory Usage

Next, I discovered our PHP-FPM configuration used static process management with an excessive number of worker processes. I optimised the configuration in /etc/php/8.4/fpm/pool.d/www.conf:

; Before - Configuration that led to OOM conditions
pm = static
pm.max_children = 50

; After - Configuration to prevent OOM Killer activation
pm = dynamic
pm.max_children = 20
pm.start_servers = 5
pm.min_spare_servers = 3
pm.max_spare_servers = 10
pm.max_requests = 500

This change significantly reduced PHP memory consumption by decreasing maximum PHP-FPM workers from 50+ to a more efficient maximum of 20, while implementing dynamic scaling based on actual WordPress traffic. The pm.max_requests setting ensures workers are periodically recycled to prevent memory leaks from accumulating.

3. Tuning MariaDB Memory Allocation

I performed a comprehensive review of our MariaDB configuration and implemented memory optimisations in /etc/mysql/mariadb.conf.d/50-server.cnf:

[mysqld]
# Reduced InnoDB buffer pool size to prevent OOM issues
innodb_buffer_pool_size = 600M

# Optimised per-connection memory buffers
sort_buffer_size = 2M
read_buffer_size = 2M
join_buffer_size = 1M
key_buffer_size = 20M

# Disabled performance monitoring to reduce memory footprint
performance_schema = OFF

# Added connection limiting to prevent resource exhaustion
max_connections = 100

These targeted optimisations reduced the database server’s overall memory footprint while maintaining adequate performance for WordPress operations. The buffer pool size adjustment was particularly important since this represents the largest memory allocation in a typical MariaDB installation.

4. Implementing Memory Monitoring

To proactively identify memory issues before they trigger the OOM Killer, I implemented a comprehensive monitoring solution with a specialised bash script:

#!/bin/bash
# Memory monitoring script to detect potential OOM Killer conditions
FREE_MEM=$(free -m | grep 'Mem:' | awk '{print $4}')
FREE_SWAP=$(free -m | grep 'Swap:' | awk '{print $4}')
TOTAL_AVAILABLE=$((FREE_MEM + FREE_SWAP))

if [ $FREE_MEM -lt 200 ]; then
  echo "WARNING: Critical memory pressure detected - OOM Killer risk" | 
  echo "Free memory: $FREE_MEM MB" |
  echo "Free swap: $FREE_SWAP MB" |
  echo "Total available: $TOTAL_AVAILABLE MB" |
  echo "Top memory processes that could trigger the OOM Killer:" |
  ps aux --sort=-%mem | head -n 6 |
  /usr/bin/mail -s "OOM Risk Alert: Memory Low on $(hostname)" admin@example.com
fi

I deployed this monitoring script to run regularly via crontab:

# Check memory every 5 minutes to catch potential OOM Killer conditions
*/5 * * * * /usr/local/bin/memory-check.sh

This solution provides early warning of potential OOM situations, allowing proactive action before the OOM Killer is triggered.

5. Configuring OOM Killer Priority

To make our mission-critical MariaDB service less likely to be terminated by the OOM Killer, I modified its OOM score adjustment value using systemd configuration:

# Create directory for MariaDB service customisations
mkdir -p /etc/systemd/system/mariadb.service.d/

# Create OOM score adjustment configuration
cat > /etc/systemd/system/mariadb.service.d/oom.conf << 'EOF'
[Service]
# Lower value (-1000 to 1000 scale) makes process less likely to be terminated
OOMScoreAdjust=-900
EOF

# Reload systemd to apply changes
systemctl daemon-reload

This kernel configuration makes the MariaDB process much less likely to be selected by the OOM Killer algorithm. The OOMScoreAdjust value of -900 significantly reduces the likelihood of this critical service being terminated during OOM situations, directing the Killer to target less essential processes instead.

OOM Killer Management Key Lessons Learned

My investigation and resolution of this event taught me several crucial lessons about effective memory management:

Configure Proper Swap Space: Even on servers with substantial RAM, swap space provides an essential buffer for memory allocation spikes and helps prevent OOM conditions. Never run production servers without properly configured swap.
Monitor Database Connection Patterns: Database connection timeouts and aborted connections are early warning indicators of memory pressure and potential OOM conditions. These should trigger immediate investigation.
Optimise PHP-FPM Worker Processes: Running excessive PHP-FPM worker processes is a leading cause of memory exhaustion in WordPress environments. Right-sizing these workers based on available system memory is critical for preventing OOM Killer events.
Implement MariaDB Memory Tuning: Database servers require careful memory configuration tuning to balance performance requirements against overall system memory availability.
Deploy Proactive Memory Monitoring: Waiting for OOM Killer events to occur is not a viable strategy. Implementing proactive monitoring with appropriate alerting thresholds helps detect potential issues before the kernel invokes the OOM Killer.
Understand OOM Killer Selection Mechanisms: Developing a thorough understanding of how the OOM Killer selects processes for termination allows you to configure OOM score adjustments that protect your most critical services.

Results of Memory Optimisation Implementation

After implementing these memory management optimisations, our server’s resource utilisation patterns dramatically improved, and we have not experienced any additional OOM Killer events. The PHP-FPM worker configuration adjustments alone reduced our base memory consumption by approximately 30%, providing significantly more headroom for WordPress traffic spikes and background processes.

The addition of properly configured swap space has proven especially valuable during periodic high-traffic events, successfully preventing OOM Killer activation even during unexpected traffic surges. Our memory monitoring system has successfully identified several potential memory pressure incidents, allowing us to proactively investigate and resolve these issues before they triggered the OOM Killer mechanism.

The MariaDB memory optimisations have successfully balanced database performance with memory efficiency, maintaining good query response times while significantly reducing the database server’s memory footprint. WordPress page load times remain excellent despite the reduced memory allocation.

sudo systemctl status mariadb.service

● mariadb.service - MariaDB 11.4.5 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─limits.conf, migrated-from-my.cnf-settings.conf
Active: active (running) since Thu 2025-04-24 12:52:01 UTC; 5 days ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Main PID: 9284 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 24 (limit: 15050)
Memory: 770.4M (peak: 793.1M)
CPU: 1h 9min 48.291s
CGroup: /system.slice/mariadb.service
└─9284 /usr/sbin/mariadbd

Conclusion

This real-world investigation into OOM Killer events highlighted the critical importance of proper memory management for maintaining stable WordPress hosting environments. By thoroughly understanding how the OOM Killer works and implementing targeted optimisation measures, we’ve successfully created a more resilient and reliable server infrastructure that efficiently handles our WordPress workloads without memory-related service disruptions.

For anyone managing WordPress sites on Linux servers, I strongly recommend implementing these key memory optimisation strategies:

Configuring appropriate swap space as a memory pressure buffer
Optimising PHP-FPM worker process counts based on actual memory availability
Fine-tuning MariaDB/MySQL memory parameters for your specific workload
Implementing OOM score adjustments to protect mission-critical services
Deploying proactive memory monitoring with appropriate alerting thresholds
Regularly reviewing application code for memory leaks and inefficient resource usage

These memory management optimisations can help you prevent the unexpected service disruptions, potential data loss, and performance degradation that occur when the OOM Killer terminates essential processes. Proper memory configuration is not merely a performance consideration—it’s a fundamental requirement for maintaining reliable WordPress hosting operations.

April 30, 2025 0 comments

Kali Linux remote SSH - How to configure openSSH server - blackMORE Ops -7

Command Line Interface (CLI)

SSH Key Management Best Practices

by blackMORE April 25, 2025

written by blackMORE

Proper SSH key management is crucial for maintaining secure access to your systems while avoiding common pitfalls that could compromise your security or disrupt your workflow. This guide covers essential SSH key management best practices, including key generation, protection, rotation, and backup strategies. By implementing these recommendations, you’ll establish a robust security posture that balances convenience with strong protection of your sensitive systems and data.

Choosing the Right Key Type and Size

The foundation of effective SSH key management starts with generating the right type of key with appropriate parameters:

Key Types

Linux supports several SSH key types, each with different security characteristics:

RSA: The traditional option, still widely compatible but requires larger key sizes (minimum 3072 bits recommended)
Ed25519: Modern, offers excellent security with smaller key sizes and better performance
ECDSA: Provides a good balance between security and compatibility

Generate Strong Keys

For optimal security on Ubuntu systems, use these commands to generate keys:

# Ed25519 (recommended for most users)
ssh-keygen -t ed25519 -C "your_email@example.com"

# RSA (if compatibility with older systems is needed)
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

The -C flag adds a comment, typically your email address, which helps identify the key’s purpose later.

Protecting Your Private Keys

Private keys are the most sensitive component of your SSH authentication system:

Use Strong Passphrases

Always protect your private keys with strong passphrases:

# Generate a new key with passphrase protection
ssh-keygen -t ed25519 -C "your_email@example.com"

# Add a passphrase to an existing key
ssh-keygen -p -f ~/.ssh/id_ed25519

A strong passphrase should:

Be at least 12 characters long
Include a mix of uppercase, lowercase, numbers, and special characters
Not be used for any other accounts or systems

Set Proper File Permissions

SSH is sensitive to file permissions for security reasons. Set appropriate permissions:

# Set secure permissions on private key
chmod 600 ~/.ssh/id_ed25519

# Set permissions on public key
chmod 644 ~/.ssh/id_ed25519.pub

# Set permissions on .ssh directory
chmod 700 ~/.ssh

Incorrect permissions can lead to SSH refusing to use your keys.

Using SSH-Agent Effectively

SSH-Agent provides convenience without compromising security:

Start and Configure SSH-Agent

# Start the agent
eval $(ssh-agent -s)

# Add your key with a timeout (e.g., 4 hours)
ssh-add -t 4h ~/.ssh/id_ed25519

Adding a timeout ensures your keys aren’t held in memory indefinitely, reducing the risk if your session is compromised.

Configure SSH-Agent in Your Shell Profile

For persistent configuration, add these lines to your ~/.bashrc or ~/.zshrc:

# Start SSH agent if not already running
if [ -z "$SSH_AUTH_SOCK" ]; then
    eval $(ssh-agent -s)
    ssh-add ~/.ssh/id_ed25519 </dev/null
fi

Key Rotation Strategies

Regular key rotation is an essential security practice:

Implementing a Rotation Schedule

Personal keys: Rotate every 6-12 months
Automated system keys: Rotate every 3-6 months
High-security environments: Consider more frequent rotation

Process for Safe Key Rotation

Generate a new key pair:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new -C "your_email@example.com"

Add the new public key to servers while keeping the old one:

ssh-copy-id -i ~/.ssh/id_ed25519_new.pub username@remote-server

Test connectivity with the new key:

ssh -i ~/.ssh/id_ed25519_new username@remote-server

Once verified, replace the old key with the new one:

mv ~/.ssh/id_ed25519_new ~/.ssh/id_ed25519
mv ~/.ssh/id_ed25519_new.pub ~/.ssh/id_ed25519.pub

Remove the old public key from authorized_keys on remote servers.

Secure Backup Strategies

Losing access to your SSH keys can be catastrophic, making backups essential:

Encrypted Backups

Create encrypted backups of your SSH keys:

# Create an encrypted archive of your .ssh directory
tar -czf - ~/.ssh | gpg -c > ssh_keys_backup.tar.gz.gpg

Store this encrypted backup in a secure location, such as:

An encrypted USB drive kept in a safe location
A trusted password manager with file attachment capabilities
A secure cloud storage service with client-side encryption

Recovery Documentation

Create documentation for your key recovery process that includes:

Locations of backups
Encryption passwords (stored separately)
List of systems where keys are deployed
Steps to follow for key restoration

Managing Multiple Keys

As your infrastructure grows, you’ll likely need multiple keys for different purposes:

Organising Keys by Function

Create dedicated keys for different contexts:

Personal development work
Production system access
Automated backup processes
Different clients or projects

Name keys descriptively:

~/.ssh/id_ed25519_personal
~/.ssh/id_ed25519_production
~/.ssh/id_rsa_legacy_systems

Using SSH Config for Key Management

Configure ~/.ssh/config to specify which key to use for each host:

# Personal development server
Host dev-server
    HostName dev.example.com
    User developer
    IdentityFile ~/.ssh/id_ed25519_personal
    IdentitiesOnly yes

# Production servers
Host prod-*
    User admin
    IdentityFile ~/.ssh/id_ed25519_production
    IdentitiesOnly yes

# Legacy system with older SSH implementation
Host legacy
    HostName legacy.example.com
    User admin
    IdentityFile ~/.ssh/id_rsa_legacy
    IdentitiesOnly yes

The IdentitiesOnly yes setting ensures only the specified key is offered during authentication attempts.

Security Considerations for Converted Keys

If you’re using PuTTY keys converted for Ubuntu, additional considerations apply:

Validating Converted Keys

After converting PuTTY keys:

Verify file permissions are set correctly
Test with a simple SSH connection before using in automated processes
Consider generating fresh keys natively on Ubuntu for maximum security

For more information on converting PuTTY keys to use on Ubuntu, see our detailed guide on Converting PuTTY SSH Keys for Ubuntu.

Conclusion

Implementing these SSH key management best practices for Ubuntu systems creates a robust security foundation for your infrastructure. By generating strong keys, protecting them with passphrases and proper permissions, using SSH-agent effectively, implementing key rotation, maintaining secure backups, and properly managing multiple keys, you’ll significantly reduce the risk of unauthorized access while maintaining convenient authentication workflows. Remember that SSH key management is an ongoing process that requires regular attention to ensure continued security of your systems.

For specific issues related to SSH authentication with tools like rsync, refer to our troubleshooting guide on Troubleshooting rsync SSH Authentication Issues.

April 25, 2025 0 comments

CentOS

How to Automatically Restart Failed Services in Linux Using Monit

by blackMORE April 24, 2025

written by blackMORE

When running production servers, ensuring your critical services stay up and running is essential. Service failures can happen for various reasons — memory issues, unexpected crashes, or resource limitations. Fortunately, Linux offers reliable solutions to automatically monitor and restart failed services without requiring constant manual intervention.

Server reliability is crucial for maintaining uptime and providing consistent service to users. Whether you’re running web servers like Apache or Nginx, database services like MySQL/MariaDB, PHP-FPM, or custom applications, unexpected failures can significantly impact your operations.

This guide focuses on using Monit, a versatile monitoring utility with extensive service management capabilities, to automatically restart failed services. If you’re looking for an alternative approach using the built-in service manager, you might also be interested in using systemd for automatic restarts.

Which Method Should You Choose?

There are two main approaches to automatically restarting services in Linux: Monit (covered in this guide) and systemd (the built-in service manager). Here’s how they compare:

Feature	systemd	Monit
Installation required	✅ No (built-in)	❌ Yes
Email notifications	❌ Requires extra setup	✅ Built-in
Web interface	❌ No	✅ Yes
Resource monitoring	⚠️ Limited	✅ Extensive
Configuration complexity	✅ Simple	⚠️ Moderate
Modern distributions support	✅ Excellent	✅ Good
Legacy system support	❌ Limited	✅ Excellent

Choose Monit if:

You need advanced monitoring features
You want email notifications
You’re running an older Linux distribution
You need to monitor resource usage or network connectivity

Choose systemd if:

You’re running a modern Linux distribution
You need a simple, built-in solution
You’re primarily concerned with basic service restarts

For details on implementing the systemd approach, see our guide on How to Automatically Restart Failed Services in Linux Using systemd.

Using Monit for Service Monitoring and Recovery

Monit is a comprehensive monitoring utility that offers advanced features for service monitoring and automatic recovery. It’s particularly useful for:

Systems with or without systemd
Complex monitoring requirements
Email notifications on failures
Monitoring multiple aspects of a service (memory usage, CPU, connectivity)

Installing Monit

On Debian/Ubuntu:

sudo apt-get update
sudo apt-get install monit

On CentOS/RHEL:

# For CentOS 7/8 with EPEL repository
sudo yum install epel-release
sudo yum install monit

# For manual installation from source
cd /opt
wget http://www.tildeslash.com/monit/dist/monit-latest.tar.gz
tar -zxvf monit-latest.tar.gz
cd monit-*
./configure && make && make install

Configuring Monit

The main configuration file is typically located at /etc/monit/monitrc or /etc/monitrc. Make sure it’s readable only by root:

sudo chmod 600 /etc/monit/monitrc

Open the file for editing:

sudo vim /etc/monit/monitrc

Add these basic settings:

# Run checks every 2 minutes
set daemon 120

# Set logging
set logfile syslog facility log_daemon

# Optionally set up email alerts
set mailserver your.mail.server
set mail-format { 
    from: alerts@yourdomain.com
    subject: $SERVICE $EVENT at $DATE
    message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION
}

# Web interface (optional)
set httpd port 2812
    allow admin:strongpassword
    
# Include service-specific configurations
include /etc/monit/conf.d/*

The last line is critical – it tells Monit to include configuration files from the conf.d directory, which is where service-specific monitoring configurations should be placed in modern installations.

Monitoring Web Servers

Modern Monit installations use separate configuration files in the /etc/monit/conf.d/ directory for each service. This makes configurations more maintainable and easier to update.

Apache Configuration

Create a configuration file for Apache:

sudo vim /etc/monit/conf.d/apache2

Add this content:

# Apache web server monitoring
check process apache2 with pidfile /var/run/apache2/apache2.pid
    start program = "/etc/init.d/apache2 start"
    stop program = "/etc/init.d/apache2 stop"
    if failed host 127.0.0.1 port 80 protocol http then restart
    if 5 restarts within 5 cycles then timeout

Nginx Configuration

Create a configuration file for Nginx:

sudo vim /etc/monit/conf.d/nginx

Add this content:

# Nginx web server monitoring
check process nginx with pidfile /var/run/nginx.pid
    start program = "/etc/init.d/nginx start"
    stop program = "/etc/init.d/nginx stop"
    if failed host 127.0.0.1 port 80 protocol http then restart
    if 5 restarts within 5 cycles then timeout

PHP-FPM 8.3 Configuration

Create a configuration file for PHP-FPM 8.3:

sudo vim /etc/monit/conf.d/php-fpm-8.3

Add this content:

# PHP-FPM 8.3 monitoring
check process php8.3-fpm with pidfile /var/run/php/php8.3-fpm.pid
    start program = "/etc/init.d/php8.3-fpm start"
    stop program = "/etc/init.d/php8.3-fpm stop"
    if failed unixsocket /var/run/php/php8.3-fpm.sock then restart
    if failed port 9000 type TCP then restart
    if 5 restarts within 5 cycles then timeout

Monitoring Database Services

MariaDB Configuration

Create a configuration file for MariaDB:

sudo vim /etc/monit/conf.d/mariadb

Add this content:

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
    group database
    start program = "/etc/init.d/mariadb start"
    stop program = "/etc/init.d/mariadb stop"
    if failed host 127.0.0.1 port 3306 then restart
    if 5 restarts within 5 cycles then timeout

Custom Service Monitoring

For monitoring custom services, create separate configuration files in the conf.d directory:

sudo vim /etc/monit/conf.d/myservice

Add content like this:

check process myservice with pidfile /var/run/myservice.pid
    start program = "/etc/init.d/myservice start"
    stop program = "/etc/init.d/myservice stop"
    if failed port 8080 protocol http then restart
    if cpu usage > 95% for 3 cycles then restart
    if memory usage > 75% for 3 cycles then restart
    if 5 restarts within 5 cycles then timeout

Advanced Monitoring Options

Create more sophisticated monitoring rules in separate files:

sudo vim /etc/monit/conf.d/resources

# Monitor memory and CPU usage
check process resourcehog with pidfile /var/run/resourcehog.pid
    if cpu > 80% for 5 cycles then restart
    if memory usage > 200 MB for 5 cycles then restart
    
# Monitor response time
check host mywebsite.com with address mywebsite.com
    if failed port 80 protocol http 
       with timeout 5 seconds 
       with content = "Welcome"
    then alert
    
# Monitor disk space
check device rootfs with path /
    if space usage > 90% then alert

Starting and Testing Monit

Start the Monit service:

# On systemd-based systems
sudo systemctl start monit
sudo systemctl enable monit

# On older init.d systems
sudo service monit start

Verify Monit is running:

sudo monit status

Test the configuration:

sudo monit validate
sudo monit summary

Troubleshooting Monit

Syntax errors: Check for errors with sudo monit -t
Service not detected: Verify the PID file path is correct
Cannot start/stop service: Ensure Monit has proper permissions to execute init scripts
Path issues: Make sure all paths in your configuration file are absolute
Web interface not working: Check firewall settings and verify the httpd section is properly configured

Conclusion

Monit offers a powerful solution for automatically monitoring and restarting failed services in Linux environments. Its flexibility and extensive features make it particularly valuable for production systems where uptime is critical.

By implementing automatic service monitoring and restart capabilities with Monit, you can significantly improve your server reliability and reduce downtime. The added benefits of resource monitoring, email alerts, and web interface make it a comprehensive solution for server management.

Take the time to properly configure Monit for your critical services—it’s one of the most effective ways to improve your system’s reliability and save yourself from those middle-of-the-night emergency restarts.

April 24, 2025 0 comments

CentOS

How to Automatically Restart Failed Services in Linux Using systemd

by blackMORE April 24, 2025

written by blackMORE

When running production servers, ensuring your critical services stay up and running is essential. Service failures can happen for various reasons — memory issues, unexpected crashes, or resource limitations. Fortunately, modern Linux distributions include systemd, which offers built-in functionality to automatically restart failed services without requiring constant manual intervention.

Server reliability is crucial for maintaining uptime and providing consistent service to users. Learning how to automatically restart failed services using systemd is one of the most effective ways to improve system stability. Whether you’re running web servers like Apache or Nginx, database services like MySQL, or custom applications, unexpected failures can significantly impact your operations unless you configure systemd to handle recovery automatically.

This guide focuses on using systemd (the modern standard service manager in most current Linux distributions) to automatically restart failed services when they crash or terminate unexpectedly. We’ll walk through the process of configuring systemd to monitor and automatically restart services like MariaDB when they fail. If you’re looking for an alternative approach, you might also be interested in using Monit for service monitoring and automatic restarts.

Which Method Should You Choose?

There are two main approaches to automatically restarting services in Linux: systemd (covered in this guide) and Monit (a separate monitoring utility). Here’s how they compare:

Feature	systemd	Monit
Installation required	✅ No (built-in)	❌ Yes
Email notifications	❌ Requires extra setup	✅ Built-in
Web interface	❌ No	✅ Yes
Resource monitoring	⚠️ Limited	✅ Extensive
Configuration complexity	✅ Simple	⚠️ Moderate
Modern distributions support	✅ Excellent	✅ Good
Legacy system support	❌ Limited	✅ Excellent

Choose systemd if:

You’re running a modern Linux distribution
You need a simple, built-in solution
You’re primarily concerned with basic service restarts

Choose Monit if:

You need advanced monitoring features
You want email notifications
You’re running an older Linux distribution
You need to monitor resource usage or network connectivity

For details on implementing the Monit approach, see our guide on How to Automatically Restart Failed Services in Linux Using Monit.

Using systemd for Automatic Restarts

Systemd is the default init system and service manager in most modern Linux distributions (CentOS 7+, Ubuntu 16.04+, Debian 8+). It offers built-in functionality for monitoring and automatically restarting services without requiring additional software.

Finding Your Service Files

Before you can configure auto-restart, you need to locate the correct service file. The most reliable way to do this is by checking the service status:

sudo systemctl status mariadb.service

For example, with MariaDB you might see:

● mariadb.service - MariaDB 11.4.5 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: enabled)
   Drop-In: /etc/systemd/system/mariadb.service.d
            └─limits.conf, migrated-from-my.cnf-settings.conf
   Active: active (running) since Thu 2025-04-24 06:34:22 UTC; 6min ago

The important line is Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: enabled), which tells you exactly where the service file is located.

Configuring Auto-restart

Once you’ve identified your service file, you have two options:

Option 1: Create an Override

This approach is best when working with system-provided service files:

sudo systemctl edit mariadb.service

This opens an editor where you can add just the settings you want to override. Add these lines:

[Service]
Restart=always
RestartSec=3s

When you save and exit, systemd will automatically create a drop-in file in /etc/systemd/system/mariadb.service.d/ and reload the configuration. However, sometimes it doesn’t work and you need to edit the service file directly.

Option 2: Edit the Service File Directly

If you’re creating a custom service file or if you really must edit a file directly, use the path from the status output:

sudo vim /usr/lib/systemd/system/mariadb.service

Add these lines to the [Service] section:

Restart=always
RestartSec=3s

Understanding Restart Options

The Restart directive offers several options:

always: Restart the service regardless of the exit reason (most comprehensive)
on-failure: Restart only if the service exits with a non-zero status or crashes
on-abnormal: Restart if the service was terminated by a signal or timed out
on-watchdog: Restart only if the watchdog timeout is triggered
on-abort: Restart only if the service dumps core
on-success: Restart only if the service exits cleanly with code 0
no: Never restart the service (default behaviour)

Complete Example: MariaDB Service

Here’s a complete example of auto-restart configuration for MariaDB. First, find the service file:

sudo systemctl status mariadb.service

This will show something like:

● mariadb.service - MariaDB 11.4.5 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: enabled)

Now create an override:

sudo systemctl edit mariadb.service

Add these lines to the editor:

[Service]
Restart=on-failure
RestartSec=5s

When you save and exit, systemd will create the override file at /etc/systemd/system/mariadb.service.d/override.conf.

Verifying Your Configuration

After making your changes, you can verify the complete service configuration with:

systemctl cat mariadb.service

This will show both the original service file and any overrides you’ve created. You should see your Restart settings in the output.

Managing Your Service

After configuring auto-restart:

If you used systemctl edit, the changes are automatically applied. Otherwise, reload the systemd daemon:
```
sudo systemctl daemon-reload
```
Restart your service to apply the new configuration:
```
sudo systemctl restart mariadb.service
```
Enable the service to start automatically on boot:
```
sudo systemctl enable mariadb.service
```

Verifying Auto-restart Works

To confirm your service will restart automatically:

Check the service properties:

systemctl show -p Restart,RestartSec mariadb.service

Test the restart functionality:

sudo systemctl status mariadb.service  # Note the process ID
sudo kill -9 [process-id]              # Force kill the process
sudo systemctl status mariadb.service  # Verify it restarted automatically

Troubleshooting systemd Restarts

If you’re having issues with auto-restart:

Service doesn’t restart: Check journal logs with journalctl -u mariadb.service for errors
Continuous restart loops: If your service keeps failing and restarting, add these lines to limit restart attempts:
```
StartLimitIntervalSec=500StartLimitBurst=5
```
Permission issues: Ensure the service has proper permissions to restart

Conclusion

Automatic service restart capabilities are essential tools for maintaining high availability in Linux environments. Systemd offers a reliable, built-in solution that requires no additional software installation.

By implementing automatic restarts with systemd, you can significantly improve your server reliability and reduce downtime. No more manually restarting crashed services at 3 AM!

Take the time to properly configure automatic restarts for your critical services—it’s one of the simplest yet most effective ways to improve your system’s reliability.

April 24, 2025 0 comments

Desktop Managers

Kali Linux Without GUI

by blackMORE April 23, 2025

written by blackMORE

Kali Linux installations typically include a GUI by default but running Kali Linux without GUI maybe the only way to run it on a low spec hardware or to improve system performance. This guide explores how to use Kali Linux without GUI and why many advanced users prefer this streamlined approach.

Disabling the GUI on Kali Linux

To temporarily set up Kali Linux without GUI while keeping the graphical components installed:

Open a terminal window in your current Kali Linux installation.
Run this command to configure Kali Linux without GUI boot mode:

sudo systemctl set-default multi-user.target

Reboot your system to apply the Kali Linux without GUI setting:

sudo reboot

After reboot, your Kali Linux without GUI environment will present a simple login prompt without loading any desktop environment.

Re-enabling the GUI (If Needed)

If you’ve been running Kali Linux without GUI but need to temporarily access graphical tools:

From your Kali Linux without GUI terminal, run:

sudo systemctl set-default graphical.target

Reboot your Kali Linux system:

sudo reboot

Your Kali Linux will now boot with the GUI enabled again.

Completely Removing the GUI from Kali Linux

For a permanent Kali Linux without GUI setup that saves disk space:

Ensure you’re logged into your Kali Linux with administrator privileges.
Remove the Xfce desktop environment to fully commit to Kali Linux without GUI:

sudo apt purge --autoremove xfce4* kali-desktop-xfce libxfce4*

This command transforms your system into a pure Kali Linux without GUI installation by removing all desktop environment components.

Install GUI on Kali Linux

If you completely removed the GUI components from your Kali Linux without GUI setup and need to restore them:

In your Kali Linux without GUI environment, run:

sudo apt update
sudo apt install kali-desktop-xfce

Configure Kali Linux to boot with the GUI:

sudo systemctl set-default graphical.target

Reboot your Kali Linux system:

sudo reboot

Command line tools in Kali Linux without GUI

When running Kali Linux without GUI, you’ll need these essential command-line tools:

- Process management: Monitor your Kali Linux without GUI system with ps, top, btop, and htop
- Package management: Kali Linux without GUI software installation uses apt commands
- Terminal multiplexers: Enhance your Kali Linux without GUI experience with tmux or screen
- File management: In Kali Linux without GUI, use ls, cd, mv, cp, and rm
- Text editing: Kali Linux without GUI text editors include nano, vim, and emacs
- Network tools: Kali Linux without GUI network management uses ifconfig, ip, netstat, , and ss

For web access in your Kali Linux without GUI environment, install these text-based browsers:

Lynx: A lightweight browser for Kali Linux without GUI (sudo apt install lynx)
W3m: Another browser option for Kali Linux without GUI (sudo apt install w3m)
Links: A feature-rich browser for Kali Linux without GUI (sudo apt install links)

Conclusion

Just ensure the specific tools you need function properly in a Kali Linux without GUI environment before making the switch. I kind of like the option where I can just connect to my Kali remotely using Remote Desktop client or SSH depending on what I am doing.

That’s it.

April 23, 2025 0 comments

Newer Posts

Older Posts