How to Automatically Restart Failed Services in Linux Using Monit

161

When running production servers, ensuring your critical services stay up and running is essential. Service failures can happen for various reasons — memory issues, unexpected crashes, or resource limitations. Fortunately, Linux offers reliable solutions to automatically monitor and restart failed services without requiring constant manual intervention. This guide takes you through step by step instructions on howto install, configure and automatically restart failed services using Monit.

Server reliability is crucial for maintaining uptime and providing consistent service to users. Whether you’re running web servers like Apache or Nginx, database services like MySQL/MariaDB, PHP-FPM, or custom applications, unexpected failures can significantly impact your operations.

This guide focuses on using Monit, a versatile monitoring utility with extensive service management capabilities, to automatically restart failed services. If you’re looking for an alternative approach using the built-in service manager, you might also be interested in using systemd for automatic restarts.

Which Method Should You Choose?

There are two main approaches to automatically restarting services in Linux: Monit (covered in this guide) and systemd (the built-in service manager). Here’s how they compare:

Choose Monit if:

• You need advanced monitoring features
• You want email notifications
• You’re running an older Linux distribution
• You need to monitor resource usage or network connectivity

Choose systemd if:

• You’re running a modern Linux distribution
• You need a simple, built-in solution
• You’re primarily concerned with basic service restarts

For details on implementing the systemd approach, see our guide on How to Automatically Restart Failed Services in Linux Using systemd.

Using Monit for Service Monitoring and Recovery

Monit is a comprehensive monitoring utility that offers advanced features for service monitoring and automatic recovery. It’s particularly useful for:

• Systems with or without systemd
• Complex monitoring requirements
• Email notifications on failures
• Monitoring multiple aspects of a service (memory usage, CPU, connectivity)

Installing Monit

On Debian/Ubuntu:

sudo apt-get update
sudo apt-get install monit

On CentOS/RHEL:

# For CentOS 7/8 with EPEL repository
sudo yum install epel-release
sudo yum install monit

# For manual installation from source
cd /opt
wget http://www.tildeslash.com/monit/dist/monit-latest.tar.gz
tar -zxvf monit-latest.tar.gz
cd monit-*
./configure && make && make install

Configuring Monit

The main configuration file is typically located at /etc/monit/monitrc or /etc/monitrc. Make sure it’s readable only by root:

sudo chmod 600 /etc/monit/monitrc

Open the file for editing:

sudo vim /etc/monit/monitrc

Add these basic settings:

# Run checks every 2 minutes
set daemon 120

# Set logging
set logfile syslog facility log_daemon

# Optionally set up email alerts
set mailserver your.mail.server
set mail-format { 
    from: [email protected]
    subject: $SERVICE $EVENT at $DATE
    message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION
}

# Web interface (optional)
set httpd port 2812
    allow admin:strongpassword
    
# Include service-specific configurations
include /etc/monit/conf.d/*

The last line is critical – it tells Monit to include configuration files from the conf.d directory, which is where service-specific monitoring configurations should be placed in modern installations.

Monitoring Web Servers

Modern Monit installations use separate configuration files in the /etc/monit/conf.d/ directory for each service. This makes configurations more maintainable and easier to update.

Apache Configuration

Create a configuration file for Apache:

sudo vim /etc/monit/conf.d/apache2

Add this content:

# Apache web server monitoring
check process apache2 with pidfile /var/run/apache2/apache2.pid
    start program = "/etc/init.d/apache2 start"
    stop program = "/etc/init.d/apache2 stop"
    if failed host 127.0.0.1 port 80 protocol http then restart
    if 5 restarts within 5 cycles then timeout

Nginx Configuration

Create a configuration file for Nginx:

sudo vim /etc/monit/conf.d/nginx

Add this content:

# Nginx web server monitoring
check process nginx with pidfile /var/run/nginx.pid
    start program = "/etc/init.d/nginx start"
    stop program = "/etc/init.d/nginx stop"
    if failed host 127.0.0.1 port 80 protocol http then restart
    if 5 restarts within 5 cycles then timeout

PHP-FPM 8.3 Configuration

Create a configuration file for PHP-FPM 8.3:

sudo vim /etc/monit/conf.d/php-fpm-8.3

Add this content:

# PHP-FPM 8.3 monitoring
check process php8.3-fpm with pidfile /var/run/php/php8.3-fpm.pid
    start program = "/etc/init.d/php8.3-fpm start"
    stop program = "/etc/init.d/php8.3-fpm stop"
    if failed unixsocket /var/run/php/php8.3-fpm.sock then restart
    if failed port 9000 type TCP then restart
    if 5 restarts within 5 cycles then timeout

Monitoring Database Services

MariaDB Configuration

Create a configuration file for MariaDB:

sudo vim /etc/monit/conf.d/mariadb

Add this content:

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
    group database
    start program = "/etc/init.d/mariadb start"
    stop program = "/etc/init.d/mariadb stop"
    if failed host 127.0.0.1 port 3306 then restart
    if 5 restarts within 5 cycles then timeout

Custom Service Monitoring

For monitoring custom services, create separate configuration files in the conf.d directory:

sudo vim /etc/monit/conf.d/myservice

Add content like this:

check process myservice with pidfile /var/run/myservice.pid
    start program = "/etc/init.d/myservice start"
    stop program = "/etc/init.d/myservice stop"
    if failed port 8080 protocol http then restart
    if cpu usage > 95% for 3 cycles then restart
    if memory usage > 75% for 3 cycles then restart
    if 5 restarts within 5 cycles then timeout

Advanced Monitoring Options

Create more sophisticated monitoring rules in separate files:

sudo vim /etc/monit/conf.d/resources

# Monitor memory and CPU usage
check process resourcehog with pidfile /var/run/resourcehog.pid
    if cpu > 80% for 5 cycles then restart
    if memory usage > 200 MB for 5 cycles then restart
    
# Monitor response time
check host mywebsite.com with address mywebsite.com
    if failed port 80 protocol http 
       with timeout 5 seconds 
       with content = "Welcome"
    then alert
    
# Monitor disk space
check device rootfs with path /
    if space usage > 90% then alert

Starting and Testing Monit

Start the Monit service:

# On systemd-based systems
sudo systemctl start monit
sudo systemctl enable monit

# On older init.d systems
sudo service monit start

Verify Monit is running:

sudo monit status

Test the configuration:

sudo monit validate
sudo monit summary

Troubleshooting Monit

• Syntax errors: Check for errors with sudo monit -t
• Service not detected: Verify the PID file path is correct
• Cannot start/stop service: Ensure Monit has proper permissions to execute init scripts
• Path issues: Make sure all paths in your configuration file are absolute
• Web interface not working: Check firewall settings and verify the httpd section is properly configured

Conclusion

Monit offers a powerful solution for automatically monitoring and restarting failed services in Linux environments. Its flexibility and extensive features make it particularly valuable for production systems where uptime is critical.

By implementing automatic service monitoring and restart capabilities with Monit, you can significantly improve your server reliability and reduce downtime. The added benefits of resource monitoring, email alerts, and web interface make it a comprehensive solution for server management.

Take the time to properly configure Monit for your critical services—it’s one of the most effective ways to improve your system’s reliability and save yourself from those middle-of-the-night emergency restarts.