High CPU utilization: postgres, snort_inline, afcd

Darin Marvin · #1 (**permalink**) 01-09-2009, 03:38 AM

I am running v7.305 in a virtual environment with Athlon X2 4800+ and 2 gigs RAM, using Ubuntu 64-bit as host. Guest OS is using 2 CPUs. Everything has been working fine until the last couple of weeks when I noticed surfing has been slow. I have trouble connecting to the backend, but when I do CPU is around 98%. RAM use is high (as expected). I ran 'top' today and here is the result (CPU, Mem, Command):

53 1.1 postgres
45 1.0 postgres
44 1.0 postgres
16 0.9 afcd
11 7.0 snort_inline

We only have two users and they just use the gateway for e-mail and Web surfing. No torrents or streaming HD video or anything else that would consume so many resources.

Any ideas?

Thx

Jack Daniel · #2 (**permalink**) 01-09-2009, 04:01 AM

Two suggestions:
Check the system messages log for postgres errors, you may have database issues (solutions are elsewhere in the forums depending on the specific problem).
And check the IM/P2P logs for signs of false positives or excessive/odd IM/P2P traffic. Disabling both IM and P2P, waiting a few minutes (really, a few minutes, not seconds- until AFCD drops off of the process list in top) and then re-enabling may clear AFCD's CPU use.

Darin Marvin · #3 (**permalink**) 01-10-2009, 09:41 PM

Jack,
Thanks for the advice. I do not use IM/P2P Classifier so the logs for that are empty.
The afcd and snort processes seem to have calmed down a lot, but instead of three (3) postgres processes running, today there are now five (5)! They are taking up 98-99% of the CPU and also that of my host OS.
I looked at the system messages log and do not see any postgres errors at all. There are just the usual dhcpc-sh connection checking every 60 seconds (fine) and cron running /usr/local/bin/reportcontrol.sh and /sbin/audld.plx occasionally. No real errors that I can see.
I will look through the forums for multiple postgres process spawning, but so far I have not seen anything like this issue.

Best Regards,
Darin

Jack Daniel · #4 (**permalink**) 01-10-2009, 09:44 PM

If IM and P2P are disabled there should not be any AFCD process running, does your system still show active an AFCD process?

Darin Marvin · #5 (**permalink**) 01-10-2009, 10:07 PM

Here is a quick update: I manually killed all five (5) postgres processes and CPU usage is back down to normal (about 5%). The system seems to be running fine for now. I wonder why the postgres (both username and command name) processes are spawned and then lock up? Perhaps they are getting stuck in a loop somewhere (bug?).

Any feedback on what else I could check would be appreciated. Perhaps there is a bug somewhere.

- Darin

Darin Marvin · #6 (**permalink**) 01-10-2009, 10:10 PM

Jack,
Sorry, my mistake: IM/P2P control is active. I am running Live Messenger on one PC on the network, but nothing is being logged by the system.
- Darin

Darin Marvin · #7 (**permalink**) 01-11-2009, 06:45 PM

Below you can see I do get two postgres system errors, and when I ran top again there was a postgres process running at 88% CPU. The log does not give any reindex hints at all, though. Is there something I can do besides killing the stranded processes every day?

Thx.

2009:01:11-00:15:00 (none) postgres[27424]: [3-1] LOG: unexpected EOF on client connection
2009:01:11-00:15:01 (none) /usr/sbin/cron[27426]: (root) CMD (nice -n19 /usr/local/bin/create_rrd_graphs.plx)
2009:01:11-00:15:01 (none) /usr/sbin/cron[27428]: (root) CMD ( /usr/local/bin/reportcontrol.sh)
2009:01:11-00:15:34 (none) dhcpc-sh: DHCP connection fine. Checking again in 60 seconds
2009:01:11-00:16:01 (none) /usr/sbin/cron[27472]: (root) CMD (/sbin/auisys.plx --nosys)
2009:01:11-00:16:35 (none) dhcpc-sh: DHCP connection fine. Checking again in 60 seconds
2009:01:11-00:17:01 (none) /usr/sbin/cron[27506]: (root) CMD ( nice -n19 /usr/local/bin/gen_inline_reporting_data.plx)
2009:01:11-00:17:35 (none) dhcpc-sh: DHCP connection fine. Checking again in 60 seconds
2009:01:11-00:18:35 (none) dhcpc-sh: DHCP connection fine. Checking again in 60 seconds
2009:01:11-00:19:01 (none) /usr/sbin/cron[27558]: (root) CMD (/usr/bin/sp_updater)
2009:01:11-03:15:00 (none) postgres[32414]: [3-1] LOG: unexpected EOF on client connection
2009:01:11-03:15:01 (none) /usr/sbin/cron[32416]: (root) CMD (nice -n19 /usr/local/bin/create_rrd_graphs.plx)
2009:01:11-03:15:01 (none) /usr/sbin/cron[32418]: (root) CMD ( /usr/local/bin/reportcontrol.sh)
2009:01:11-03:15:48 (none) dhcpc-sh: DHCP connection fine. Checking again in 60 seconds
2009:01:11-03:16:01 (none) /usr/sbin/cron[32467]: (root) CMD (/sbin/auisys.plx --nosys)
2009:01:11-03:16:48 (none) dhcpc-sh: DHCP connection fine. Checking again in 60 seconds
2009:01:11-03:17:01 (none) /usr/sbin/cron[32494]: (root) CMD ( nice -n19 /usr/local/bin/gen_inline_reporting_data.plx)

Jack Daniel · #8 (**permalink**) 01-12-2009, 04:05 AM

Don't worry about the "unexpected EOF" messages, they are safe to ignore- that is just a closed connection, note the line is marked "log", not "error".

Darin Marvin · #9 (**permalink**) 01-15-2009, 01:48 AM

Here is what the system log shows when I manually kill the postgres processes that are hogging the CPU:

2009:01:14-16:25:42 (none) postgres[6310]: [3-1] FATAL: terminating connection due to administrator command
2009:01:14-16:25:42 (none) postgres[6310]: [3-2] STATEMENT: select distinct flow_start_day from accounting where flow_start_day < $1
2009:01:14-16:25:45 (none) postgres[13409]: [3-1] FATAL: terminating connection due to administrator command
2009:01:14-16:25:45 (none) postgres[13409]: [3-2] STATEMENT: select distinct flow_start_day from accounting where flow_start_day < $1
2009:01:14-16:25:50 (none) postgres[20236]: [3-1] FATAL: terminating connection due to administrator command
2009:01:14-16:25:50 (none) postgres[20236]: [3-2] STATEMENT: select distinct flow_start_day from accounting where flow_start_day < $1

Ed.Kok · #10 (**permalink**) 03-04-2010, 07:25 AM

This morning saw the Astaro with high utilization and found the process 'postgres' at around 99%. The System Messages log shows:

Code:

2010:03:04-08:09:17 RM-HQ-AST postgres[27308]: [3-1] LOG: unexpected EOF on client connection
2010:03:04-08:10:01 RM-HQ-AST /usr/sbin/cron[27394]: (root) CMD ( /usr/local/bin/reportcontrol.sh)
2010:03:04-08:10:09 RM-HQ-AST postgres[27366]: [3-1] LOG: unexpected EOF on client connection
2010:03:04-08:11:00 RM-HQ-AST postgres[27428]: [3-1] LOG: unexpected EOF on client connection
2010:03:04-08:12:04 RM-HQ-AST postgres[27459]: [3-1] LOG: unexpected EOF on client connection
2010:03:04-08:13:24 RM-HQ-AST postgres[27543]: [3-1] LOG: unexpected EOF on client connection
2010:03:04-08:14:19 RM-HQ-AST postgres[27633]: [3-1] LOG: unexpected EOF on client connection

Any suggestions how to solve this? Just reboot, or kill the process?

Ed