World of *Nix

As i was checking WassUp stats of my blog, suddenly i found that my blog got 71 visits from some external sites (referrer) in last 24 hours. First my felling was “WOW, so may referrer”.

referrer spam

Later after i check some of those referring sites, i found these are just some sites, that using my site to get better position in site ranking. It’s call “referrer spam”. From wikipedia:

Referrer spam (also known as log spam or referrer bombing[1]) is a kind of spamdexing (spamming aimed at search engines). The technique involves making repeated web site requests using a fake referrer url that points to the site the spammer wishes to advertise.[2] Sites that publicize their access logs, including referrer statistics, will then end up linking to the spammer’s site, which will in turn be indexed by the search engines as they crawl the access logs.

This benefits the spammer because of the free link, and also gives the spammer’s site improved search engine ranking due to link-counting algorithms that search engines use.

Though these visits will not loss you any thing, but it will just waste your server resource. And also if you show some stats on your site like referrer, then your visitors will get some wrong links.

Ok lets see how we can stop them. I searched on net, but didn’t got any good solution. So here is my solution.

Just add these lines to your .htaccess file and smile.

Order Allow,Deny
Deny from 195.54.42.56
Deny from 89.149.244.217
Deny from 78.137.7.99
Deny from 94.142.134.155
Allow from all

Here i use ip to block/stop this abuse, because after some research i fount this is happening from just particular ips, means those ips running some scripts to do this. Its semi manual way (still looking for better way). Just get these ips from your access log or any 3rd party plugins or apps. then add “Deny from newip” before Allow from all .

That’s all . If you know any better option please share with us.

If you are using slow connection like me and trying to download the apples iPhone SDK , then you already know how much tough it is , because of apple only allow to download this file after login. And non of the download manager can give you the resume facility. so after trying 2 days with different downloaders i tried wget and got sucess 😀

Here i am discribing how to do it. First login to http://developer.apple.com/iphone (if you don’t have any account, just create one, it’s free). then use firefox extension to export (firebug+firecookie) cookie to cookie.txt . Then rest of the task simple. just copy download link from http://developer.apple.com/iphone and give the following command in the console.

$cd /path/to/cookie.txt

$ wget -ct 0 –load-cookies cookie.txt your_file_url

that’s all, now you can resume download without any problem. Here is the command i used to download to download it.

$ wget -ct 0 –server-response –load-cookies cookies.txt http://developer.apple.com/iphone/download.action?path=/iphone/iphone_sdk_3.0__final/iphone_sdk_3.0__leopard__9m2736__final.dmg

If this post help you, don’t forget to share it with others.

On the first post of this series we learned how to disable etags. in this post we will learn how to enable output compression.

Before enable this, let me introduce what deflate do. deflate is apache module, that main task is to compress all the output before serving or sending to client.

Nowadays most of the browser are enough expert to handle this. Here main technique is , apache will send the output in compressed format, after receiving this output your browser will uncompress it and render the formated output.

This tricks will really improve your site performance, because user will get your site with less time and bandwidth.

Let me give you an example: let’s think if you are using jquery javascript library that size is 94 Kilo bytes, but if you enable the deflate with default configuration, it will be only 14 Kilo bytes. unbelievable right ? believe me, it’s true. And we are using this on our most popular product http://www.somewhereinblog.net .

OK, now i am showing how you will do this:

first of all you need to enable deflate module. You know i am debian fan, so i am giving commands for debian linux, other linux/unix commands will almost same. as ubuntu is comes from debian, these commands will also work for ubuntu

Open a new terminal and give this command.
sudo a2enmod deflate

it will enable deflate. Wait, you are not finish yet. little work left.
create new file called deflate.conf in your apache conf.d folder. and paste the following code:
<Location />

# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems…
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don’t compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary

# Make sure proxies don’t deliver the wrong content
Header append Vary User-Agent env=!dont-vary

</Location>

save the file, and restart apache.

sudo /etc/init.d/apache2 restart

that’s all. now you can see the different.

**if you have not enabled header module of apache, please remove the line “Header append Vary User-Agent env=!dont-vary” from deflate.conf .

For more customization or option of deflate, please visit http://httpd.apache.org/docs/2.2/mod/mod_deflate.html

From last few months i am working to optimize our somewhereinblog.net server, because right now we are getting huge amount of hits everyday (1,00,000 + hits). In this series i will describe the tricks that i learned 🙂

On my first post of this searies i will talk about ETag. Before disable this, first we need to know what is ETags. From wikipedia:

An ETag (entity tag) is an HTTP response header returned by an HTTP/1.1 compliant web server used to determine change in content at a given URL. When a new HTTP response contains the same ETag as an older HTTP response, the contents are determined to be the same without further downloading. The header is useful for intermediary devices that perform caching, as well as for client web browsers that cache results. One method of generating the ETag is based on the last modified time of the file and the size of the file.

In certain situations, ETags may not improve the performance of a web application. For instance, some ETag generation schemes incorporate the file’s inode on the system. The file’s inode is unique to the file only on one specific machine. If a site has multiple servers, each with its own copy of the file (i.e. load balancing), then a user’s request for the same file may get served by a different machine. In that case, the inode will almost certainly be different and, if it is used to generate the ETag, it will cause the file to be re-downloaded.

So, this is not always helpfull for all type of sites. Always remember, if you are not using anything, you should not enable or kept enable on server.

Apache by default enable this ETags. We need to disable this to get better performance. Here i am describing the steps for debian and ubuntu linux.

We will take help from mod_header to disable ETags, so first we need to enable the mod_header (apache module)

sudo a2enmod header

sudo /etc/init.d/apache2 restart

this command will enable mod_header. Now open /etc/apache2/apache2.conf with your faviourate browser.

sudo nano /etc/apache2/apache2.conf

then paste the under given code to this apache2.conf file

Header unset ETag
FileETag None

now restart the apache2, and check if it working or not. If every thing OK, then you will not see the etags anymore 🙂

For better understanding i am giving you 2 screenshots.

Before ETag disable

After ETag disable:

If you don’t have root access to server, you still can do this from .htaccess , just copy that code to .htaccess instate of apache conf file.

Ask me if you have any question or problem about ETag.

Apt is the best friend for any deb dependend linux like debian , ubuntu etc.

You can speed up your apt by searching the nerestrepositery of apt (for your distro/linux)

I tested some tools to do this, APT-SPY shows the best performance 🙂

From manpage:

apt-spy is a program that tests a series of Debian archive mirrors for bandwith. It writes an /etc/apt/sources.list with the fastest server.

Run this command as su or root next thing will done by apt-spy, it will write a new /etc/apt/sources.list and the olde one will be backup with sources.list.bak

apt-spy -d unstable -n 5