Troubleshooting high server loads on Linux servers
What causes high server loads?
Excessive usage of any of the following items can typically cause this issue:
- CPU
- memory (including swap)
- disk I/O
How can I check these items?
That depends whether you want to review their current resource usage, or historical resource usage. This tutorial will cover both.
if you wanted to view the load averages for your server from the 23rd of the month, you would run this command:
sar -q -f /var/log/sa/sa23
You do not need to specify the date when viewing the statistics for the current day. As such, this command would show the load average for today:
sar -q
Current CPU usage
top c
tip
hit "P" to sort by processes that are currently consuming the most CPU.
Historical CPU usage
Check the "%idle" column:
sar -p
Current memory usage
free -m
tip
run "top c" and hit "M" to see which processes are consuming the most memory.
Historical memory usage
This depends on the version of sar, which used to use '-r' to show %memused and %swpused (swap memory used), but later changed to '-S' to show %swpused.
Check "%memused" and "%swpused":
sar -r
Current disk I/O usage
iostat -x 1 10
Historial disk I/O usage
sar -d
The resources that are checked are as follows:
- 1 minute load average
- kilobytes of swap used
- kilobytes of memory usage
- packets per second inbound
- packets per second outbound
- number of processes
The subject line of the email will look something like this:
nodeN.example.com [L: 24] [P: 315] [Swap Use: 17% ] [pps in: 745 pps out: 844]
Before you use the script
You will need to add your email address to the "EMAIL" variable. For example:
EMAIL="email@example.com"
You would also likely want to adjust the following 5 items:
- MAX_LOAD
- MAX_SWAP_USED
- MAX_MEM_USED
- MAX_PPS_OUT
- MAX_PPS_IN
#!/bin/sh
export PATH=/bin:/usr/bin
##########################################################################
# #
# Email Shortcuts : #
# #
# L - the 1 minute load average #
# P - the number of processes in the process list #
# Swap Usage - the percentage of swap memory being used #
# pps in - packets per second inbound #
# pps out - packets per second outbound #
# #
##########################################################################
###############################################################################
# START USER CONFIGURABLE VARIABLES
###############################################################################
EMAIL="email@example.com"
# 1 minute load avg
MAX_LOAD=3
# kB
MAX_SWAP_USED=1000
# kB
MAX_MEM_USED=500000
# packets per second inbound
MAX_PPS_IN=2000
# packets per second outbound
MAX_PPS_OUT=2000
# max processes in the process list
MAX_PROCS=400
###############################################################################
# END USER CONFIGURABLE VARIABLES
###############################################################################
IFACE=`grep ETHDEV /etc/wwwacct.conf | awk '{print $2}'`
if [[ "$IFACE" =~ "venet" ]] ; then
IFACE=venet0
fi
IFACE=${IFACE}:
###############################################################################
# 1 min load avg
###############################################################################
ONE_MIN_LOADAVG=`cut -d . -f 1 /proc/loadavg`
echo "1 minute load avg: $ONE_MIN_LOADAVG"
###############################################################################
# swap used
###############################################################################
SWAP_TOTAL=`grep ^SwapTotal: /proc/meminfo | awk '{print $2}'`
SWAP_FREE=`grep ^SwapFree: /proc/meminfo | awk '{print $2}'`
let "SWAP_USED = (SWAP_TOTAL - SWAP_FREE)"
echo "Swap used: $SWAP_USED kB"
###############################################################################
# mem used
###############################################################################
MEM_TOTAL=`grep ^MemTotal: /proc/meminfo | awk '{print $2}'`
MEM_FREE=`grep ^MemFree: /proc/meminfo | awk '{print $2}'`
let "MEM_USED = (MEM_TOTAL - MEM_FREE)"
echo "Mem used: $MEM_USED kB"
###############################################################################
# packets received
###############################################################################
PACKETS_RX_1=`grep $IFACE /proc/net/dev | awk '{print $2}'`
sleep 2;
PACKETS_RX_2=`grep $IFACE /proc/net/dev | awk '{print $2}'`
let "PACKETS_RX = (PACKETS_RX_2 - PACKETS_RX_1) / 2"
echo "packets received (2 secs): $PACKETS_RX"
###############################################################################
# packets sent
###############################################################################
PACKETS_TX_1=`grep $IFACE /proc/net/dev | awk '{print $10}'`
sleep 2;
PACKETS_TX_2=`grep $IFACE /proc/net/dev | awk '{print $10}'`
let "PACKETS_TX = (PACKETS_TX_2 - PACKETS_TX_1) / 2"
echo "packets sent (2 secs): $PACKETS_TX"
let "SWAP_USED = SWAP_TOTAL - SWAP_FREE"
if [ ! "$SWAP_USED" == 0 ] ; then
PERCENTAGE_SWAP_USED=`echo $SWAP_USED / $SWAP_TOTAL | bc -l`
TOTAL_PERCENTAGE=`echo ${PERCENTAGE_SWAP_USED:1:2}%`
else
TOTAL_PERCENTAGE='0%'
fi
###############################################################################
# number of processes
###############################################################################
MAX_PROCS_CHECK=`ps ax | wc -l`
send_alert()
{
SUBJECTLINE="`hostname` [L: $ONE_MIN_LOADAVG] [P: $MAX_PROCS_CHECK] [Swap Use: $TOTAL_PERCENTAGE ] [pps in: $PACKETS_RX pps out: $PACKETS_TX]"
echo "$SUBJECTLINE"
ps auxwwwf | mail -s "$SUBJECTLINE" $EMAIL
exit
}
if [ $ONE_MIN_LOADAVG -gt $MAX_LOAD ] ; then send_alert
elif [ $SWAP_USED -gt $MAX_SWAP_USED ] ; then send_alert
elif [ $MEM_USED -gt $MAX_MEM_USED ] ; then send_alert
elif [ $PACKETS_RX -gt $MAX_PPS_IN ] ; then send_alert
elif [ $PACKETS_TX -gt $MAX_PPS_OUT ] ; then send_alert
elif [ $MAX_PROCS_CHECK -gt $MAX_PROCS ] ; then send_alert
fi
I hope you find this useful.