All yesterday I had multiple apache processes killing my linux (centos) server. I would kill them, and they would come back minutes later taking up all processor resources and crashing the machine. I was using tcpdump to watch packets come across on port 80, but this connection apparently didn't persist, so that did nothing for me. Then I found netstat:
[root@centos ~]# netstat -anp | sort -u
Active Internet connections (servers and established)
Active UNIX domain sockets (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
Proto RefCnt Flags Type State I-Node PID/Program name Path
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 2169/portmap
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2593/mysqld
tcp 0 0 0.0.0.0:696 0.0.0.0:* LISTEN 2210/rpc.statd
tcp 0 0 127.0.0.1:2207 0.0.0.0:* LISTEN 2447/python
tcp 0 0 127.0.0.1:2208 0.0.0.0:* LISTEN 2442/hpiod
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2629/sendmail: acce
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 2469/cupsd
tcp 0 0 192.168.5.3:3306 192.168.5.1:1358 ESTABLISHED 2593/mysqld
tcp 0 0 192.168.5.3:3306 192.168.5.1:1645 ESTABLISHED 2593/mysqld
tcp 0 0 192.168.5.3:33648 192.168.5.4:25 ESTABLISHED 1995/python
tcp 0 0 192.168.5.3:52169 192.168.5.2:3306 ESTABLISHED 1995/python
tcp 0 0 192.168.5.3:52177 192.168.5.2:3306 ESTABLISHED 1995/python
tcp 0 0 :::22 :::* LISTEN 2460/sshd
Turns out apache was choking on an unprintable character passed in via a XML document to a API that the machine hosts. Would have taken a long time to figure out where it was coming from without netstat.
Share on Twitter
Share on Facebook