How to create Robot.txt file for SEO?

Robot.txt is a text file webmasters create to instruct web robots most often search engines how to crawl pages on their website. Robot.txt file also known as robots exclusion protocol (REP). It also tells web robots which pages to crawl and not to crawl. The REP also includes directives like meta robots, as well as page - sub directory, or site - wide instructions for how search engines should treat links (such as "follow" or "nofollow").

Let's say a search engine is about to visit a site. Before it visits the target page, it will check the robots.txt for instructions.

The Basic format of robot.txt file looks like:

	User – agent: [user-agent name] Disallow: [URL string not to be crawled]

Together, these two lines are considered a complete robots.text file.

	User – agent: * Disallow: /

The above code, is the actual skeleton of a robots.txt file.The asterisk after "user-agent" means that the robots.txt file applies to all web robots that visit the site. The slash after "Disallow" tells the robot to not visit any pages on the site.
You all might be wondering why would anyone want to stop web robots from visiting a site. This is where the secret to this SEO hack comes in.You might be having a lot of pages on your site, right? If a search engine crawls your site, it actually crawls all the pages of your website, it will take the search engine bot a while to crawl them, which can also have negative effects on your ranking. That's because Googlebot (Google's search engine bot ) has a crawl budget.

This is how Google explains:

1. Crawl rate limit

Which limits the maximum fetching rate for a given site. The Crawl rate can go up and down based on a couple of factors:

a) Crawl Health: if the site responds really quickly for a while the limit goes up, that is more connections can be used to crawl. If the site slows down or responds to errors, the limit goes down and Googlebot crawls less.

b) Limit set in Search Console: website owners can reduce Googlebot's crawling of their site.

2. Crawl Demand

Even if the crawl rate limit isn't reached, if there's no demand from indexing, there will be low activity from Googlebot. The two factors that play a significant role in determining crawl demand are:

a) Popularity: URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.

b) Staleness: our systems attempt to prevent URLs from becoming stale in the index.

Crawl Budget: The number of URLs Googlebot can and wants to crawl.

Finding your robots.txt file:

If you just want a quick look at your robots.txt file, or want to see for any site all you have to do is type the basic URL of the site into your browser's search bar (e.g. abc.com, example.com etc.). Then add/robots.txt onto the end.

Following things will happen:

1. You'll find a robots.txt file.
2. You'll find an empty file.
3. You'll get a 404 error.

Let's see Few examples of robot.txt in for www.abcxyz.com site.

Robots.txt file URL: www.abcxyz.com/robots.txt

	User – agent: * Disallow: /

Using the above syntax, would tell all web crawlers not to crawl any pages on www.abcxyz.com, including the homepage.

	User – agent: * Disallow:

Using the above syntax, would tell all web crawlers to crawl all pages on  www.abcxyz.com, including the homepage.

	User – agent: Googlebot Disallow: /abcwyz-subfolder/

Using the above syntax, would tell only Google's crawlers not to crawl any pages that contain the URL string www.abcxyz.com/abcxyz-subfolder/.

	User – agent: Bingbot Disallow: /abcwyz-subfolder/blocked-page.html

Using the above syntax, would tell only Bing's crawlers to avoid crawling the specific page at the URL string www.abcxyz.com/abcxyz-subfolder/blocked-page.

Technical Phrases:

1. User-agent: The specific web crawler to which you're giving crawl instructions (search engine).
2. Disallow:  The command used to tell a user-agent not to crawl particular URL. Only one "Disallow" line is allowed for each URL.
3. Allow: Only applicable for Googlebot. The command to tell Googlebot it can access a page or sub folder even though its parent page or sub folder may be disallowed.
4. Crawl-delay: How many milliseconds a crawler should wait before loading and crawling page content.
5. Sitemap: Used to call out the location of any XML sitemap(s) associated with this URL. Only supported by Google, Ask, Bing and Yahoo.

Some points to be noted:

• A robots.txt file must be placed in a website's top-level directory (web-crawling robots only look for the file in one specific place; the main directory(root domain or homepage).If a user agent visits  www.abcxyz.com/robots.txt and does not find a robots file there, it will assume the site does not have one and proceed with crawling everything on the page.

• The file must  be named as "robots.txt" as it is a case sensitive (not Robots.txt, robots.TXT, pr anywise)

• Each sub domain  on a root domain uses separate robots.txt files. This means that both blog.abcxyz.com and abcxyz.com should have their own robots.txt files.

• The /robots.txt file is a publicly available: just add/robots.txt to the end of any root domain to see that website's directives. This means that anyone can see what pages you do want to crawl or don't want to crawled.   
• It's generally a best practice to indicate the location of any sitemaps associated with this domain at the bottom of the robots.txt file. Example:

	User – agent: * Allow: /*.htmls Disallow: /*/data/* Sitemap: https://www.abcxyz.com/en-gb/sitemap.xml

Some common cases that justify Why do we need robots.txt?

• Preventing duplicate content from appearing in SERPs
• Keeping entire sections of a website private
• Keeping internal search engine results pages from showing up on a public SERP
• Preventing search engines from indexing certain files on your website
• Specifying the location of sitemap(s)
• Specifying a crawl delay in order to prevent your servers from being overloaded when crawlers load multiple pieces of content at once

In case, there are no areas on your site to which you want to control user-agent access, you may not need a robots.txt file at all.

How does robots.txt work?

1. Crawling the web to discover content
2. Indexing that content so that it can be served up to searched who are looking for information.

To Crawl sites, search engines follow links to get from one site to another, crawling across many links and website. This crawling is also known as "spidering".

After arriving at a website but before spidering it, the search crawler will look for a robots,txt file. If it finds one the crawler will read that first and then continue through the page. And any case, there is no robots.txt it will proceed the entire website.

That's all from my end...

If you have any queries, feel free to write in comments down below..

Stay tuned for more digital advertising!!

Thank You...

SEO Techniques -White Hat & Black Hat SEO

Search Engine Optimization (SEO) techniques are classified into two broad categories:

• Techniques that search engines recommend as part of good design referred to as White Hat SEO, and 
• Techniques that search engines do not approve and attempt to minimize the effect of referred to as Black Hat or spamdexing.

SEO Tactics are as follows in detail:

An SEO tactic, technique or method is considered as White Hat if it follows the followings :

• If it conforms to the search engine's guidelines.
• If it does not involves any deception.
• It ensures that the content a search engine indexes and subsequently ranks is the same content a user will see.
• It ensures that a web page content should have been created for the users and not just for the search engines.
• It ensures the good quality of the web pages.
• It ensures the useful content available on the web pages.
• Always follow a White Hat SEO tactic and don't try to fool your site visitors. Be honest and definitely you will get something more.
• Next chapter onward we will put light on White Hat SEO techniques. The White Hat SEO are very simple and can be done without investing much cost.

Black Hat or Spamdexing

An SEO tactic, technique or method is considered as Black Hat or Spamdexing if it follows the followings

• Try to improve rankings that are disapproved of by the search engines and / or involve deception.
• Redirecting users from a page that is built for search engines to one that is more human friendly.
• Redirecting users to a page that was different from the page  the Search Engine ranked.
• Serving one version of a page  to search engine spiders/bots and another version to human visitors. This is called Cloaking SEO tactic.
• Using hidden or invisible text or with the page background color, using a tiny font size or hiding them within the HTML code such as "no frame" sections.
• Repeating keywords in the meta tags, and using  keywords that are unrelated to the site's content. This called Meta tag stuffing.
• Calculated placement of keywords within a page to raise the keyword count, variety, and density of the page. This is called Keyword stuffing.
• Creating low - quality web pages that contain very little content but are instead stuffed with very similar keywords and phrases. These pages are called Doorway or Gateway Pages.
• Mirror websites by hosting multiple web sites all with conceptually similar content but using different URLs.
• Creating a rogue copy of a popular website which shows contents similar to the original to a web crawler, but redirects web surfers to unrelated or malicious web sites. This called Page hijacking.

Always be away to adopt any of the above Black hat tactic to improve the rank of your website. Search Engines are smart enough to identify all the above properties of your site and ultimately you are not going to get anything.

What is SEO Copywriting?

SEO Copywriting is the technique of writing the view able text on a web page in such a way that it reads well for the surfer, and also targets specific search terms. Its purpose is to rank highly in the search engines for the targeted search terms.

As well as the view able text, SEO Copywriting usually optimizes other on-page elements for the targeted search terms. These include the Title, Description and keywords tags, headings and alt text.

The idea behind SEO Copywriting is that search engines want genuine content pages and not additional pages (often called "doorway pages" ) that are created for the sole purpose of achieving high rankings.

What is Search Engine Rank?

When you search any keyword using a search engine then it displays thousands of results found in its database. A page ranking is measured by the position of web pages displayed in the search engine results. If search engine is putting your web page on first position then your web page rank will be number 1 and it will be assumed as with a high rank.

SEO is the process of designing and developing a web site to attend a high rank in search engine results.

What is On-page and Off- page SEO?

Conceptually, there are two ways of doing SEO:

• On - Page SEO: This includes providing good content, good keywords selection, putting keywords on correct places, giving appropriate title to every page etc.
• Off- Page SEO: This includes link building, increasing link polarity by submitting in open directories, search engines, link exchange etc.   

SEO Website Domain

• When you start thinking of doing a business through internet, first thing which you think about is your website domain name. Before you choose a domain name you should consider  the followings:
• Who would be your target audience?
• What you intend to sell to them. Is it tangible item or just text content.
• What will make your business idea unique or different than everything else that is already on the market?

Many people think it is important to have keywords in a domain. Keywords in the domain name are usually important, but it usually can be done while keeping the domain name short, memorable, and free of hyphens.

Using keywords in your domain name gives you a strong competitive advantage over your competitors. Having your keywords in your domain can increase click through rates on search engine listings and paid ads as well as make it easier to using your keywords in getting keyword rich descriptive inbound links.

Avoid buying long, confusing domain names. May people separate the words in their domain names using either dashes or hyphen. In the past the domain name itself was a significant ranking factor but now search engines have advanced it is not very significant factor anymore.

Keep two to three words in your domain name it will be more memorable. Some of the most memorable websites do a great job of branding by creating their own word. Few examples are eBay, Yahoo!, Expedia, Slashdot, Fark, Wikipedia, Google...

You should be able to say it over the telephone once and the other person should know how to spell it and they should be able to guess about what you sell.

Few Points:

Finally, you should be able to answer yourself for the following questions:

• Why do you want to build your website? Why should people buy from your site and not from other site? What makes you different from others?
• Who are your target audience and what you intend to sell to them?
• List 5-10 websites which you think, they are amazing. Now think why are they amazing?
• Create 5 different domain names. Make at least 1 of them funny. Tell them to a half  dozen people and see which ones are the most memorable. You will get more honest feedback of the people do not know you well.
• Buy your domain name which is more catchy, memorable and relevant to your business.

That's all from my end...

If you have any queries, feel free to write in comments down below..
Stay tuned for more digital advertising!!

Thank You...

      

How does Search Engine Optimization/SEO works?

SEO commonly known as Search Engine Optimization is the activity of optimizing Web pages or whole sites in order to make them more search engine friendly, thus getting higher positions in search results.

SEO is sometimes also called SEO copyrighting because most of the techniques that are used to promote sites in search engines deal with text.

• SEO stands for Search Engine Optimization.
• SEO is all about optimizing a website for Search Engines.
• SEO is the process of designing and developing a website to rank well in search engine results.
• SEO is to improve the volume and quality of traffic to a website from search engines.
• SEO is a subset of search engine marketing.
• SEO is the art of ranking in the search engines.
• SEO is marketing by understanding how search algorithms work and what human visitors might search. 

A comprehensive search engine optimization project is divided into four interrelated phases:

1. Pre-site activities - The research and planning activities undertaken before an  existing or new site or page is actually touched or built.

• Understanding your organization's online business strategy.
• Researching your market category, customers and competitors.
• Keyword research and selection.

2. On-Site activities - The activities involved in the content and design of web pages.

• Writing the title, description and keyword meta tags.
• Writing content - Body copy, titles, images tags, outbound links that reflect and enhance keywords.
• Building Internal Links - Helping the search engines navigate the site.
• Site design and construction - Ensuring the web pages utilities design and code that can be properly crawled and indexed by the search engines.

3. Off-site activities -Building a portfolio of quality inbound links to your website.

4. Post-site activities - Analyzing and responding to site traffic and user feedback once a website has been optimized. Effective SEO is a continuous feedback.

If you plan to do some basic SEO, it is essential that you understand how search engines work and which items are most important in SEO.

How Search Engine Works?

Search engines perform several activities in order to deliver search results:

• Crawling - is the process of fetching all the web pages lined to a website. This task is performed by a software, called a crawler or a spider (or Google bot, as is the case with google). 
• Indexing - is the process of creating index for all the fetched webpages and keeping them into a giant database from where it can later be retrieved. Essentially, the process of indexing is identifying the words and expressions that best describe the page and assigning the page to particular keywords.

Processing - when a search request comes, the search engine processes it. i.e. it compares the search string in the search request with the indexed pages in the database.

• Calculating Relevancy - Since it is likely that more than one pages contains the search string, so the search engine starts calculating the relevancy of each of the pages in its index to the search string.
• Retrieving Results - The last step in search engines' activities is retrieving the best matched results. Basically,  it is nothing more than simply displaying them in the browser. 

Search engines such as Google and Yahoo! often update their relevancy algorithm dozens of times per month. When you see changes in your rankings it is due to an algorithmic shift or something else outside of your control.

Although the basic principle of operation of all search engines is the same, the minor differences between their relevancy algorithms lead  to major changes in results relevancy.

That's all from my end...

If you have any queries, feel free to write in comments down below..
Stay tuned for more digital advertising!!

Thank You...