Sitemap and robots.txt for Small Business Sites in India

Set up an XML sitemap and robots.txt for your Indian small business website. Plain steps, Google's rules, Search Console checks and common mistakes.

By the DIG Marketing team. . 11 min read

Quick answer: A sitemap is a list of the pages you want Google to know about. A robots.txt file is a short text file at the root of your site that tells crawlers which parts they may crawl. Most small business sites need a sitemap only when they are large or hard to browse, and they need a robots.txt file that does not block anything important. Both take under an hour to check, and this guide shows how.

Many owners of shops, clinics, coaching centres and service firms in India have a website built by a freelancer or set up in a hosting dashboard, and they never see these two files. They only find out when a page does not show up in Google. This guide explains what each file does, what Google says about them, and a simple checklist you can run yourself or hand to your web person.

Do you need a sitemap at all?

Google's own Sitemaps report guidance is honest about this. If your site is small, meaning about 500 pages or fewer, and you can reach every page by following links from your homepage, you probably do not need a sitemap. In that case, the job is to make sure your menu and footer link to the important pages and that the homepage is indexed.

You are more likely to need one when you have many pages, new pages with few links pointing to them, or a site that you publish on often.

If you use a website builder such as Wix or Squarespace, the platform usually creates a sitemap for you. Check its help pages before you build one by hand. You may only need to submit the URL it already gives you.

What an XML sitemap is and what goes in it

An XML sitemap is a file, usually at yourdomain.in/sitemap.xml, that lists URLs in a standard format. Google's documentation says each sitemap file can hold up to 50,000 URLs and must stay under 50 MB uncompressed. If you have more, you split the list into several files and create one sitemap index file that points to them.

The rules that matter most for a small site are these:

  • List canonical URLs only. A canonical URL is the one version you want shown in search. If the same page opens at both http and https, or with and without www, pick one version and list only that one.
  • Use full addresses. Write https://yourdomain.in/services/ in full, not /services/ on its own.
  • Leave out pages that redirect or do not load. A redirected address should be replaced by its final destination. A missing page should be removed.
  • Leave out pages you have blocked from search on purpose. Thank-you pages, login pages and internal search result pages usually do not belong in the list.
  • Save the file as UTF-8. Hindi or other Indian language text in page titles will then display correctly.

A short, accurate list beats a long one full of old or duplicate addresses. Google treats the sitemap as a hint, so listing a page does not guarantee that it is crawled or indexed.

How to create the sitemap

You have three common routes.

  1. Let your platform create it. WordPress plugins, most e-commerce platforms and many hosted site builders can generate a sitemap automatically. Open the address in your browser and check that it lists your real pages.
  2. Generate it with a tool. A sitemap generator can crawl a custom site. Check the output by eye before you upload it.
  3. Write it by hand. For a few dozen pages, a plain text editor is enough.

Here is the shape of a minimal sitemap, with example addresses only:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
 <url>
 <loc>https://yourdomain.in/</loc>
 </url>
 <url>
 <loc>https://yourdomain.in/services/</loc>
 </url>
</urlset>

If you include lastmod, use the date of a real, meaningful content change. Google uses that date only when it is consistently accurate, and it ignores the priority and changefreq tags.

After you upload the file, open it in a browser. You should see the XML list, with no login screen and no warning text.

What robots.txt does and does not do

A robots.txt file lives at the root of the host, for example https://yourdomain.in/robots.txt. Each host has one file. Google's guidance says it is mainly used to manage crawler traffic and to stop crawling of pages you do not need crawled. It is not a way to keep a page out of Google.

This point catches many owners. If you block a page in robots.txt, Google may still show its address in results, just without a description, because it cannot read the page. To keep a page out of search, use a noindex instruction on the page itself, or protect the page with a password. The page must stay crawlable for Google to see the noindex tag, so do not block it in robots.txt at the same time.

A simple, safe file for a small business looks like this:

User-agent: *
Allow: /

Sitemap: https://yourdomain.in/sitemap.xml

That file lets every crawler visit the whole site and points it to the sitemap.

Three things to check in the file:

  • Do not block CSS or JavaScript folders. Google needs them to render your pages. Google's example shows a site that allows its scripts folder even when another rule blocks it.
  • Do not add a blanket Disallow: /. That single line stops crawlers from visiting the whole site. Some sites copy it from a test setup and forget to remove it.
  • Keep the file small. Google says it reads up to 500 KiB and ignores anything after that, so a short file is safest.

How to submit and check it in Search Console

You can give Google your sitemap in two ways. You can submit it in the Sitemaps report in Google Search Console, or you can add a Sitemap line to robots.txt. Google will find the sitemap from robots.txt the next time it reads the file. Submitting in Search Console also shows when Google fetched the file and what it found.

To submit it:

  1. Open Google Search Console and select your property. You need owner permission to use the Sitemaps report.
  2. Go to Indexing and then Sitemaps.
  3. Type the sitemap address, such as sitemap.xml, and press Submit.
  4. Check the status after a day or two. Google's statuses are Success, Couldn't fetch, and a status that reports problems in the file.

If the status is not Success, open the row for details. Google lists common causes: the file is blocked by robots.txt, the address returns a 404, the site has an unresolved manual action, or the file cannot be read. Fix the cause and submit the file again. Google keeps retrying a failed sitemap for a few days, then stops.

For robots.txt, Search Console has a report that shows the copy Google last fetched and any problems it found. Open it after each change.

Common mistakes to check before you publish

Run this checklist once, and again after any redesign or move to a new host:

  • Open yourdomain.in/robots.txt and yourdomain.in/sitemap.xml in a browser. Both should load.
  • Check that the sitemap lists only the version of the domain you use, usually https with or without www.
  • Remove any page that returns a 404 or redirects. Pages that are gone for good can be left as they are, but any page with a clear new home should redirect. The Search Console basics guide shows where to see these pages.
  • Make sure no important page carries a noindex tag. A staging copy of the site with noindex can be copied live by mistake.
  • Check that your homepage and main service pages are linked from the menu or footer. A sitemap does not replace links. Our guide on internal links for small business websites shows how to build them.
  • Confirm that the sitemap changes when you add or remove pages. A file that was generated once and never updated is a common reason for stale results.

A simple monthly routine

Sitemaps and robots.txt rarely need daily attention. A monthly check of about 15 minutes is enough for most small sites:

  1. Open the sitemap and count the URLs. Compare that with the number of pages you actually want in search. If the count has jumped, check for test or duplicate pages.
  2. Look at the Sitemaps report in Search Console for any new status other than Success.
  3. Open robots.txt and confirm it has not changed. Website updates and plugin changes sometimes add rules.
  4. Check the indexing report for pages marked as blocked by robots.txt or noindex. If you did not mean to block them, fix the cause.

Note the date and the count on your phone. Judge the trend over several months, because search results change slowly.

What to expect after you submit

A sitemap does not make a page rank higher. It helps Google find URLs and shows which versions you prefer. Pages still need useful content and links from other pages. Treat the sitemap as basic housekeeping, not a shortcut.

If you want a second pair of eyes on your site, a free check is a sensible first step. Our free tools and trials page lists what we can look at without a commitment. If you would like us to set up or review the files for you, the services page explains what is included. If something is blocking your pages and you are not sure what it is, you can also contact our support team with the address of the page.

The next step is simple: open your robots.txt and sitemap today, run the checklist above, and fix the one problem you find first. For more practical guides, see the blog.

Frequently asked questions

Does every website need an XML sitemap?

No. Google says a small site of about 500 pages or fewer, where every page is reachable through links, usually does not need one. If you are unsure, a sitemap that lists only the pages you want in search does no harm.

Can robots.txt hide a page from Google search?

Not reliably. Robots.txt controls crawling, not indexing. A blocked page can still appear in results with its address and no description. To keep a page out of search, add a noindex tag to the page and make sure the page is not blocked in robots.txt, so Google can read the tag.

Where should I put my sitemap file?

Put it at the root of your domain, such as https://yourdomain.in/sitemap.xml, if you can. Google says a sitemap placed at the root can cover the whole site. If your hosting or builder uses another location, use the address it gives you and submit that address in Search Console.

How long does Google take to read a new sitemap?

Google tries to fetch a submitted sitemap as soon as you submit it. Reading the URLs listed in it can take longer, and not every URL is crawled straight away. Check the Sitemaps report after a day or two. Google does not give a fixed time for indexing, so judge the result over weeks, not hours.

Should I list my images and videos in the sitemap?

Google supports image and video extensions, but most small sites do not need them at first. Focus on clean page URLs, useful titles and working links, and add media details later if you publish a lot of visual content.

What is the most common mistake with these two files?

The most common mistake is a leftover rule from a test or a move. A line that blocks the whole site, or a noindex tag that stayed on after launch, can hide pages for months without anyone noticing. Check robots.txt and a sample of key pages after every redesign, host change or plugin update.

Ready for the next step? Compare services and real prices, then order in a minute.

Browse services

← More articles

Ready to grow?

Pick a package, add your public link, and delivery starts within hours.

Browse all services