[ask]store_rewriter lusca

Diskusi tentang Ubuntu Server baik webserver, database server, samba server dan service lainnya serta jaringan menggunakan Sistem Operasi Ubuntu.
User avatar
rafdinal
Posts: 131
Joined: 22 Jun 2011, 20:01

[ask]store_rewriter lusca

Postby rafdinal » 05 Nov 2011, 11:35

cache log aku gann:

011/11/05 11:28:27| Ready to serve requests.
2011/11/05 11:28:27| WARNING: store_rewriter #1 (FD 7) exited
2011/11/05 11:28:27| WARNING: store_rewriter #2 (FD 8) exited
2011/11/05 11:28:27| Too few store_rewriter processes are running
FATAL: The store_rewriter helpers are crashing too rapidly, need help!
Squid Cache (Version LUSCA_HEAD-r14809): Terminated abnormally.
CPU Usage: 0.072 seconds = 0.012 user + 0.060 sys
Maximum Resident Size: 20672 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():


dimana salahnya gan?

makasih
User avatar
yudiarbi
Posts: 627
Joined: 22 Mar 2011, 09:31
Location: Probolinggo, Jawa Timur
Contact:

Re: [ask]store_rewriter lusca

Postby yudiarbi » 07 Nov 2011, 07:41

di bagian object sized in memory bos, itu memory penuh, jd gak mampu dia...
User avatar
nurri
Posts: 7
Joined: 07 Nov 2011, 13:28

Re: [ask]store_rewriter lusca

Postby nurri » 07 Nov 2011, 13:37

itu di memory proxy atau swap... mohon di jelasin mas warga baru ^_^
User avatar
yudiarbi
Posts: 627
Joined: 22 Mar 2011, 09:31
Location: Probolinggo, Jawa Timur
Contact:

Re: [ask]store_rewriter lusca

Postby yudiarbi » 07 Nov 2011, 14:31

itu di konfigurasi squidnya bro
User avatar
rafdinal
Posts: 131
Joined: 22 Jun 2011, 20:01

Re: [ask]store_rewriter lusca

Postby rafdinal » 07 Nov 2011, 17:29

@yudiarbi,,punya storeurl.pl bro,,kalau boleh di posting kesini...
User avatar
nurri
Posts: 7
Joined: 07 Nov 2011, 13:28

Re: [ask]store_rewriter lusca

Postby nurri » 08 Nov 2011, 19:31

#!/usr/bin/perl
# $Rev$
# by chudy_fernandez@yahoo.com
# Youtube updates at http://wiki.squid-cache.org/ConfigExamp ... Discussion
$|=1;
while () {
@X = split;
# $X[1] =~ s/&sig=.*//;
$x = $X[0] . " ";
$_ = $X[1];
$u = $X[1];

#photos-X.ak.fbcdn.net where X a-z
if (m/^http:\/\/photos-[a-z]?(.ak.fbcdn.net.*)/) {
print $x . "http://photos" . $1 . "\n";

#maps.google.com
} elsif (m/^http:\/\/(khm|mt)[0-9]?(.google.com.*)/) {
print $x . "http://" . $1 . $2 . "\n";

# compatibility for old cached get_video?video_id
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?(videoplayback\?id=.*?|video_id=.*?)\&(.*?)/) {
$z = $2; $z =~ s/video_id=/get_video?video_id=/;
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $z . "\n";

# youtube 1024p HD itag=37, 720p HD itag=22
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?\&(itag=37|itag=22).*?\&(id=[a-zA-Z0-9]*)/) {
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $2 . "&" . $3 . "\n";

# youtube 360p itag=34 ,480p itag=35 and others
} elsif (m/^http:\/\/([0-9.]{4}|.*\.youtube\.com|.*\.googlevideo\.com|.*\.video\.google\.com).*?\&(itag=[0-9]*).*?\&(id=[a-zA-Z0-9]*)/) {
print $x . "http://video-srv.youtube.com.SQUIDINTERNAL/" . $3 . "\n";

} elsif (m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {
print $x . "http://www.google-analytics.com/__utm.gif\n";

#Cache High Latency Ads
} elsif (m/^http:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|\.googlesyndication\.com|yieldmanager|cpxinteractive)(.*)/) {
$y = $3;$z = $2;
for ($y) {
s/pixel;.*/pixel/;
s/activity;.*/activity/;
s/(imgad[^&]*).*/\1/;
s/;ord=[?0-9]*//;
s/;&timestamp=[0-9]*//;
s/[&?]correlator=[0-9]*//;
s/&cookie=[^&]*//;
s/&ga_hid=[^&]*//;
s/&ga_vid=[^&]*//;
s/&ga_sid=[^&]*//;
# s/&prev_slotnames=[^&]*//
# s/&u_his=[^&]*//;
s/&dt=[^&]*//;
s/&dtd=[^&]*//;
s/&lmt=[^&]*//;
s/(&alternate_ad_url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&url=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&ref=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/(&cookie=http%3A%2F%2F[^(%2F)]*)[^&]*/\1/;
s/[;&?]ord=[?0-9]*//;
s/[;&]mpvid=[^&;]*//;
s/&xpc=[^&]*//;
# yieldmanager
s/\?clickTag=[^&]*//;
s/&u=[^&]*//;
s/&slotname=[^&]*//;
s/&page_slots=[^&]*//;
}
print $x . "http://" . $1 . $2 . $y . "\n";

#cache high latency ads
} elsif (m/^http:\/\/(.*?)\/(ads)\?(.*?)/) {
print $x . "http://" . $1 . "/" . $2 . "\n";

} elsif (m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
print $x . "http://" . $1 . "\n";

#cdn, varialble 1st path
} elsif (($u =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/[a-z0-9]{2,5}/cdn./;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#rapidshare
} elsif (($u =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {
print $x . "http://cdn." . $3 . "/SQUIDINTERNAL/" . $5 . "\n";

} elsif (($u =~ /maxporn/) && (m/^http:\/\/([^\/]*?)\/(.*?)\/([^\/]*?)(\?.*)?$/)) {
print $x . "http://" . $1 . "/SQUIDINTERNAL/" . $3 . "\n";

#domain/path/.*/path/filename
} elsif (($u =~ /fucktube/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?[^\/]*\/[^\/]*)\/(.*)\/([^\/]*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
@y = ($1,$2,$4,$5,$6);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "/" . $y[3] . "." . $y[4] . "\n";

#like porn hub variables url and center part of the path, filename etention 3 or 4 with or without ? at the end
} elsif (($u =~ /tube8|pornhub|xvideos/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?(\.[a-z]*)?)\.([a-z]*[0-9]?\.[^\/]{3}\/[a-z]*)(.*?)((\/[a-z]*)?(\/[^\/]*){4}\.[^\/\?]{3,4})(\?.*)?$/)) {
print $x . "http://cdn." . $4 . $6 . "\n";

#for yimg.com video
} elsif (m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {
print $x . "http://cdn.yimg.com//" . $3 . "\n";

#for yimg.com doubled
} elsif (m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {
print $x . "http://cdn.yimg.com/" . $3 . "\n";

#for yimg.com with &sig=
} elsif (m/^http:\/\/([^\.]*)\.yimg\.com\/(.*)/) {
@y = ($1,$2);
$y[0] =~ s/[a-z]+([0-9]+)?/cdn/;
$y[1] =~ s/&sig=.*//;
print $x . "http://" . $y[0] . ".yimg.com/" . $y[1] . "\n";

#youjizz. We use only domain and filename
} elsif (($u =~ /media[0-9]{1,5}\.youjizz/) && (m/^http:\/\/(.*?)(\.[^\.\-]*?\.[^\/]*)\/(.*)\/([^\/\?\&]*)\.([^\/\?\&]{3,4})(\?.*?)$/)) {
@y = ($1,$2,$4,$5);
$y[0] =~ s/(([a-zA-A]+[0-9]+(-[a-zA-Z])?$)|([^\.]*cdn[^\.]*)|([^\.]*cache[^\.]*))/cdn/;
print $x . "http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

#general purpose for cdn servers. add above your specific servers.
} elsif (m/^http:\/\/([0-9.]*?)\/\/(.*?)\.(.*)\?(.*?)/) {
print $x . "http://squid-cdn-url//" . $2 . "." . $3 . "\n";

# spicific extention
# } elsif (m/^http:\/\/(.*?)\.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|wmv|3gp|mp(4|3)|exe|msi|zip|on2|mar|swf).*?/) {
# @y = ($1,$2);
# $y[0] =~ s/((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
# print $x . "http://" . $y[0] . "." . $y[1] . "\n";

#generic http://variable.domain.com/path/filename. "ext" or "exte"
#http://cdn1-28.projectplaylist.com
#http://s1sdlod041.bcst.cdn.s1s.yimg.com
} elsif (m/^http:\/\/(.*?)(\.[^\.\-]*?\..*?)\/([^\?\&\=]*)\.([\w\d]{2,4})\??.*$/) {
@y = ($1,$2,$3,$4);
$y[0] =~ s/([a-z][0-9][a-z]dlod[\d]{3})|((cache|cdn)[-\d]*)|([a-zA-A]+-?[0-9]+(-[a-zA-Z]*)?)/cdn/;
print $x . "storeurl://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] . "\n";

# all that ends with ;
} elsif (m/^http:\/\/(.*?)\/(.*?)\;(.*)/) {
print $x . "http://" . $1 . "/" . $2 . "\n";

} else {
print $x . $_ . "\n";
}
}
User avatar
nurri
Posts: 7
Joined: 07 Nov 2011, 13:28

Re: [ask]store_rewriter lusca

Postby nurri » 08 Nov 2011, 19:34

mas yudiarbi tanya lagi... aq tidak bisa lihat log squid ku? apa yg salah
User avatar
yudiarbi
Posts: 627
Joined: 22 Mar 2011, 09:31
Location: Probolinggo, Jawa Timur
Contact:

Re: [ask]store_rewriter lusca

Postby yudiarbi » 09 Nov 2011, 12:43

iptablesnya gmn nurri?
User avatar
rafdinal
Posts: 131
Joined: 22 Jun 2011, 20:01

Re: [ask]store_rewriter lusca

Postby rafdinal » 10 Nov 2011, 12:26

@yudiarbi,,storeurl punya agan ini bisa gak HIT downlaod 4shared???
User avatar
sayed
Posts: 137
Joined: 21 Jul 2011, 16:19
Location: Lhokseumawe, Aceh
Contact:

Re: [ask]store_rewriter lusca

Postby sayed » 05 Feb 2013, 15:46

apakah ini udah jalan storeurl luscanya
mohon pencerahannya?

root@server:/etc/lusca# tail -f /var/log/lusca/store.log
1360053867.050 RELEASE -1 FFFFFFFF 0F5C0B27B9A99510BCF9994A97546847 304 1359557660 -1 1360162460 unknown -1/0 GET http://img2.blogblog.com/img/icon18_edit_allbkg.gif
1360053867.094 RELEASE -1 FFFFFFFF 67366FAB5C1857F6351E240BEC8146F8 304 1359557665 -1 1360162465 unknown -1/0 GET http://www.blogblog.com/1kt/awesomeinc/ ... _light.png
1360053867.317 RELEASE -1 FFFFFFFF 2D959F24BEF6BE6B70396B0F052CDCB3 304 1360053867 1359096420 1360053867 unknown -1/0 GET http://feedjit.com/flash/fj.swf
1360053867.488 RELEASE -1 FFFFFFFF 27ABB0D9BF5493E980BD7CD14E6FF393 200 1360053867 23349600 23349600 image/gif 43/43 GET http://e1.extreme-dm.com/s10.g?login=sp ... 50%2Cd.bmk
1360053867.511 RELEASE -1 FFFFFFFF DDE9599F4EB28A0B19448FB6ADE17047 304 1360053867 1077803767 1360140267 unknown -1/0 GET http://t1.extreme-dm.com/i.gif
1360053867.837 SWAPOUT 00 00000000 F68AD0F96B5B890867E0694D14EA8B15 200 1360053867 -1 1360053867 text/javascript -1/1755 GET http://feedjit.com/router/?ishd=0&w=tra ... 153&ww=160
1360053867.930 RELEASE -1 FFFFFFFF 4BCD39E58C8E6D3203B04FC6C3AF7074 200 1360053867 -1 1360053867 text/html 2638/2638 GET http://www.blogger.com/navbar.g?targetB ... rhj4w7GWvQ
1360053868.162 RELEASE -1 FFFFFFFF A479EB4C44767211622E0C77E79A89F2 304 1360053868 1359096412 1360053868 unknown -1/0 GET http://feedjit.com/wa/tf.html?r=521620496
1360053868.452 SWAPOUT 00 00000001 DC263F252FD5F4B17FCBB4B1DD0E7969 200 1359697967 1074714690 956144580 image/gif 35/35 GET http://www.google-analytics.com/__utm.g ... D(referral)%7Cutmcmd%3Dreferral%7Cutmcct%3D%2F2008%2F08%2Ffatal-urlrewriter-helpers-are-crashing.html%3B&utmu=D%7E
1360053887.364 RELEASE -1 FFFFFFFF 1FB3E7822AE30FF1A9F7ECCC2A05F24C 200 1360053885 -1 375007920 text/xml 88/88 POST http://ubuntu-indonesia.com/forums/ubbthreads.php
1360053908.645 RELEASE -1 FFFFFFFF B5A8762C62145B483BBCC9D925994ABB 0 -1 -1 -1 unknown -1/0 GET http://csi.gstatic.com/csi?v=3&s=blogge ... 95,ol.3268
1360053908.875 RELEASE -1 FFFFFFFF 4133AD2E02E094E654584AECA561DADC 304 1360053908 -1 1360053908 unknown -1/0 GET http://andrewpakpahan.blogspot.com/2008 ... shing.html
1360053909.068 RELEASE -1 FFFFFFFF 6F5E7131AC30357BD1ABE953EEF4A71B 304 1359557661 -1 1391093661 unknown -1/0 GET http://www.blogger.com/static/v1/widget ... bundle.css
1360053909.145 RELEASE -1 FFFFFFFF 9662B128062B55628124286444FAE02E 304 1359557660 -1 1391093660 unknown -1/0 GET http://www.blogger.com/static/v1/widget ... widgets.js
1360053909.254 RELEASE -1 FFFFFFFF 92C21DD46B19E6896843264DE1E9238B 200 1360053909 1360053909 1360055709 text/css 72/72 GET http://www.blogger.com/dyn-css/authoriz ... 266154ef10
1360053909.472 RELEASE -1 FFFFFFFF 9DC0CF326F1D09324E3AD42B63DEB9EE 304 1360011865 -1 1360098265 unknown -1/0 GET http://pagead2.googlesyndication.com/pa ... top_exp.js
1360053909.513 RELEASE -1 FFFFFFFF C319D6E74CE983EE1E4BA27D750053BB 304 1359557664 -1 1360162464 unknown -1/0 GET http://img1.blogblog.com/img/icon18_email.gif
1360053909.523 RELEASE -1 FFFFFFFF 52DD17B3DC1241266BEC59D629058A86 304 1359557660 -1 1360162460 unknown -1/0 GET http://img2.blogblog.com/img/icon18_edit_allbkg.gif
1360053909.531 RELEASE -1 FFFFFFFF 54021825E8A5359D55FF74D8B575103E 304 1359557662 -1 1360162462 unknown -1/0 GET http://img1.blogblog.com/img/icon18_wrench_allbkg.png
1360053909.545 RELEASE -1 FFFFFFFF 6A2680E8F466850FFAC27BC96CE8D083 304 1359557665 -1 1360162465 unknown -1/0 GET http://www.blogblog.com/1kt/awesomeinc/ ... _light.png
1360053909.568 RELEASE -1 FFFFFFFF 0DC65A1139FB757C3332397E3C3A059A 304 1360053909 1359096475 1360053909 unknown -1/0 GET http://feedjit.com/serve/?bc=FFFFFF&tc= ... 699&ww=160
1360053909.923 RELEASE -1 FFFFFFFF 844723F45EAC5FEC4B27474694B74C56 200 1360053909 -1 1360053909 text/html 2640/2640 GET http://www.blogger.com/navbar.g?targetB ... rhj4w7GWvQ
1360053910.045 RELEASE -1 FFFFFFFF 9E9D859888B9399D8B78E061A87E2120 304 1360053909 1359096488 1360053909 unknown -1/0 GET http://feedjit.com/js/inj_tf.css
1360053910.304 RELEASE -1 FFFFFFFF F95194CF093D3F06170ED17904BAD40C 304 1360053910 1359096420 1360053910 unknown -1/0 GET http://feedjit.com/flash/fj.swf
1360053910.428 RELEASE -1 FFFFFFFF 7B8C08046A8556B9AC9F167C985719FA 304 1360053910 1077803767 1360140310 unknown -1/0 GET http://t1.extreme-dm.com/i.gif
1360053910.452 RELEASE -1 FFFFFFFF 7AC679B0AB0A019EC2FAC7695F62A8AF 200 1360053910 23349600 23349600 image/gif 43/43 GET http://e1.extreme-dm.com/s10.g?login=sp ... 50%2Cd.bmk
1360053910.830 SWAPOUT 00 00000002 947682F6F01506D8818C31A8EF83C880 200 1360053910 -1 1360053910 text/javascript -1/1759 GET http://feedjit.com/router/?ishd=0&w=tra ... 793&ww=160
1360053911.146 RELEASE -1 FFFFFFFF 277552AED9E16AFB16C0A8B51327A541 304 1360053911 1359096412 1360053911 unknown -1/0 GET http://feedjit.com/wa/tf.html?r=521620496
1360053911.276 SWAPOUT 00 00000003 537A39D4F91B3F212866C981DDC196CC 200 1359697967 1074714690 956144580 image/gif 35/35 GET http://www.google-analytics.com/__utm.g ... D(referral)%7Cutmcmd%3Dreferral%7Cutmcct%3D%2F2008%2F08%2Ffatal-urlrewriter-helpers-are-crashing.html%3B&utmu=D%7E
User avatar
q_p
Posts: 3109
Joined: 14 Oct 2012, 13:01
Contact:

Re: [ask]store_rewriter lusca

Postby q_p » 05 Feb 2013, 17:33

Kayaknya sudah jalan itu mas. Bagaimana dengan access.log -nya mas ? Saya lebih prefer strore.log -nya di-disable saja u/ mengurangi beban squid/lusca.
User avatar
sayed
Posts: 137
Joined: 21 Jul 2011, 16:19
Location: Lhokseumawe, Aceh
Contact:

Re: [ask]store_rewriter lusca

Postby sayed » 07 Feb 2013, 02:17

access.log nya mas gada keluar tcp_hit
semua pada tcp_miss

kalo disable store.log bisa cuma kasi tanda pagar atau
cache_store_log /dev/null?
User avatar
q_p
Posts: 3109
Joined: 14 Oct 2012, 13:01
Contact:

Re: [ask]store_rewriter lusca

Postby q_p » 07 Feb 2013, 05:27

[quote=sayed]kalo disable store.log bisa cuma kasi tanda pagar atau
cache_store_log /dev/null?
yang

Code: Select all

cache_store_log /dev/null
Mas sayed ini ngikutin tut (refferensi) yang mana ? kalau ada link-nya, mungkin kita bisa bantu telusuri dari sana ... :)