User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /pages/thankyou.html

• You need a separate Disallow line for every URL prefix you want to exclude
• You may not have blank lines in a record because they are used to delimit multiple records
• Notice that before the Disallow command, you have the command: User-agent: *. The User-agent: part specifies which robot you want to block. Major known crawlers are: Googlebot (Google), Googlebot-Image (Google Image Search), Baiduspider (Baidu), Bingbot (Bing)
• One important thing to know if you are creating your own robots.txt file is that although the wildcard (*) is used in the User-agent line (meaning "any robot"), it is not allowed in the Disallow line.
• Regular expressions are not supported in either the User-agent or Disallow lines

• It is strongly recommended that you place your sitemap at the root directory of your website: http://yourwebsite.com/sitemap.xml But in some situations, you may want to produce different sitemaps for different paths on your site (e.g., security permission issues)
• Sitemaps should be no larger than 10MB (10,485,760 bytes) and can contain a maximum of 50,000 URLs. This means that if your site contains more than 50,000 URLs or your sitemap is bigger than 10MB, you must create multiple sitemap files and use a Sitemap index file
• All URLs listed in the sitemap must reside on the same host as the sitemap. For instance, if the sitemap is located at http://www.yourwebsite.com/sitemap.xml, it can't include URLs from http://subdomain.yourwebsite.com
• Once you have created your sitemap, let search engines know about it by submitting directly to them, pinging them, or adding the sitemap location to your robots.txt file
• Sitemaps can be compressed using gzip, reducing bandwidth consumption

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
 <loc>http://www.yourwebsite.com</loc>
 <lastmod>2013-01-01</lastmod>
 <changefreq>weekly</changefreq>
 <priority>0.9</priority>
</url>
<url>
 <loc>http://www.yourwebsite.com/articles/100</loc>
 <changefreq>weekly</changefreq>
</url>
<url>
 <loc>http://www.yourwebsite.com/articles/101</loc>
 <lastmod>2013-01-02</lastmod>
 <changefreq>weekly</changefreq>
</url>
<url>
 <loc>http://www.yourwebsite.com/articles/102</loc>
 <lastmod>2013-01-02T13:00:12+00:00</lastmod>
 <priority>0.5</priority>
</url>
</urlset>

<img src="image.png" alt="text_to_describe_your_image">

• Deflate is an option which comes automatically with the Apache server and which is simple to set up.
• GZIP on the other hand needs to be installed and requires a bit more work to install. However, GZIP does achieve a higher compression rate and therefore might be a better choice if your website uses pages which have a lot of images or large file sizes.

# compress text, html, javascript, css, xml:
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript

<IfModule mod_expires.c>
    ExpiresActive on
 
    ExpiresByType image/jpg "access plus 1 month"
    ExpiresByType image/jpeg "access plus 1 month"
    ExpiresByType image/gif "access plus 1 month"
    ExpiresByType image/png "access plus 1 month"
</IfModule>

RewriteCond %{HTTP_HOST} ^www\.example\.com$
RewriteRule ^/?$ "http\:\/\/example\.com\/" [R=301,L]

RewriteCond %{HTTP_HOST} !^www.example.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]