HTTP Proxy Cache size

Billybob · #1 (**permalink**) 07-26-2008, 06:05 PM

Recently upgraded from astaro 6.xx to 7.21. Hardware P4 2GHZ core duo 250gb hdd, 2 gb ram.

I have been running fine for a while, logged into the console just to see how everything was running and noticed that the proxy cache is at 17M. Let the http proxy run for about a week and its still at 17M. Is this normal? In astaro 6.3x under var/chroot-squid the cache was a few hundered megs. Under 7.21 under var/chroot-http, the cache is only 17M, I guess by design. Was wondering if I am the only one seeing this

BarryG · #2 (**permalink**) 07-27-2008, 11:16 AM

How big is the 'storage' partition?

df -h

Barry

Billybob · #3 (**permalink**) 07-27-2008, 05:35 PM

/dev/disk/by-label/storage
75G 480M 74G 1% /var/storage

So no problem for space. Buth du -ch |grep total of /var/chroot-http/var/cache directory still yields 17M

Could you check your cache directory and see what the size is?
Thanks Barry

BarryG · #4 (**permalink**) 07-28-2008, 10:58 PM

Sorry, I don't use the http proxy on ASL.

I'm sure someone else could check.

Barry

William · #5 (**permalink**) 07-29-2008, 02:12 AM

Quote:

Originally Posted by Billybob

/dev/disk/by-label/storage
75G 480M 74G 1% /var/storage

So no problem for space. Buth du -ch |grep total of /var/chroot-http/var/cache directory still yields 17M

Could you check your cache directory and see what the size is?
Thanks Barry

Astaro went to a proprietary proxy with 7.x. Unfortunately this means we have no way to tweak it. Astaro has a firm predilection to use ram for caching. They'll push the machine to swap before they'll shift caching to disk. I would not htink your cache is going to get much bigger if your traffic patterns stay the way they are. Also keep in mind much more web traffic is generated on the fly and therefore isn't cachable.

svens · #6 (**permalink**) 07-29-2008, 06:40 AM

Hi

Quote:

Originally Posted by Billybob

Recently upgraded from astaro 6.xx to 7.21. Hardware P4 2GHZ core duo 250gb hdd, 2 gb ram.

I have been running fine for a while, logged into the console just to see how everything was running and noticed that the proxy cache is at 17M. Let the http proxy run for about a week and its still at 17M. Is this normal? In astaro 6.3x under var/chroot-squid the cache was a few hundered megs. Under 7.21 under var/chroot-http, the cache is only 17M, I guess by design. Was wondering if I am the only one seeing this

Unfortunately this is a embarassing bug i introduced with V7.200 of ASG: The maxcachesize setting in /var/chroot-http/etc/httpproxy.ini is interpreted as Bytes, instead of Megabytes. This issue should be fixed with 7.300 (and with the 7.300 Beta out now). The 17MB you're counting is mostly the space required for the cache directory layout.

Cheers,

Sven.

svens · #7 (**permalink**) 07-29-2008, 06:44 AM

Hi,

Quote:

Originally Posted by William

Astaro went to a proprietary proxy with 7.x. Unfortunately this means we have no way to tweak it. Astaro has a firm predilection to use ram for caching. They'll push the machine to swap before they'll shift caching to disk. I would not htink your cache is going to get much bigger if your traffic patterns stay the way they are. Also keep in mind much more web traffic is generated on the fly and therefore isn't cachable.

The HTTP Proxy doesn't cache web content in RAM - this is all done on disk. The only thing kept in RAM are:

- Categorization results
- DNS
- Cache index
and various other data structures

I've never seen an HTTP Proxy (> V7.200) causing the ASG to swap. If you have a box where the proxy eats all memory, send me a login (sschnelle@astaro.com).
I'm always interested in catching Bugs

Cheers,

Sven.

Billybob · #8 (**permalink**) 07-29-2008, 06:09 PM

Quote:

Originally Posted by William

Astaro went to a proprietary proxy with 7.x. Unfortunately this means we have no way to tweak it.

Yes, I read about that and find it a little faster than squid for general surfing purposes. By the way I followed your tweaking guide at Astaro's excessive swap usage and had my 6.xx system tweaked fairly nicely. Specially the echo 0 > /proc/sys/vm/swappiness part that I didn't think about using on my own.

Thanks sven for the explanation. I knew it had to be in the coding somewhere and glad that it will be fixed in the future. I guess I am old fashioned and like caching stuff even with fatter pipes to the internet at cheap prices.

William · #9 (**permalink**) 07-29-2008, 06:59 PM

Quote:

Originally Posted by svens

Hi,

The HTTP Proxy doesn't cache web content in RAM - this is all done on disk. The only thing kept in RAM are:

- Categorization results
- DNS
- Cache index
and various other data structures

I've never seen an HTTP Proxy (> V7.200) causing the ASG to swap. If you have a box where the proxy eats all memory, send me a login (sschnelle@astaro.com).
I'm always interested in catching Bugs

Cheers,

Sven.

I have posted about this on multiple occasions. Astaro's base squid and memory configurations pushed many machines into swap. I posted my tweaks for combating this. Astaro's excessive swap usage

squid by default(and therefore Astaro's base squid config) caches mostly to ram. That combined with a high base swappiness setting caused many 5-early 7.x machines to heavily swap. It's a proven tweak to a proven issue over a wide range of Astaro products.

svens · #10 (**permalink**) 07-29-2008, 08:29 PM

Hi William
,

Quote:

Originally Posted by William

I have posted about this on multiple occasions. Astaro's base squid and memory configurations pushed many machines into swap. I posted my tweaks for combating this. Astaro's excessive swap usage

squid by default(and therefore Astaro's base squid config) caches mostly to ram. That combined with a high base swappiness setting caused many 5-early 7.x machines to heavily swap. It's a proven tweak to a proven issue over a wide range of Astaro products.

sorry, maybe i didn't made myself clear - i was talking explicitly about ASG V 7.200 and above, which doesn't use squid any longer, and the overall memory
usage of the HTTP Proxy should be *much* better then the versions before 7.100.

Cheers,

Sven.