Robots.txt example file

Last updated: August 6, 2021

A lot of people look for example robots.txt files. Rightfully so, because robots.txt are easy to mess up.

In this article we'll cover a simple and a slightly more advanced example robots.txt file.

A simple robots.txt file

Here's an example of a simple robots.txt file that:

User-agent: *
Disallow: 

Sitemap: https://www.example.com/sitemap.xml

Is your robots.txt limiting your SEO performance?

Audit your robots.txt and find out right away!

Here's an example of a slightly more advanced robots.txt file example that:

User-agent: googlebot
Disallow: /shoes/

User-agent: BingBot
Disallow: /socks/

Sitemap: https://www.example.com/sitemap.xml

Read the full Academy article to learn everything about Robots.txt

Name:	`viewedOuibounceModal`
Provider:	ContentKing
Purpose:	Store state of having closed the exit intent modal.
Expiry:	1 month
Type:	http_cookie

Name:	`nette-samesite`
Provider:	ContentKing
Purpose:	Communication with account service.
Expiry:	Session
Type:	http_cookie

Name:	`session_timestamp`
Provider:	ContentKing
Purpose:	Utility for tracking of session start time.
Expiry:	Session
Type:	SessionStorage

Name:	`what-intent`
Provider:	ContentKing
Purpose:	Utility for tracking the current input method.
Expiry:	Session
Type:	SessionStorage

Name:	`what-input`
Provider:	ContentKing
Purpose:	Utility for tracking the current input method.
Expiry:	Session
Type:	SessionStorage

Name:	`_dc_gtm_*`
Provider:	Google Analytics
Purpose:	Store number of service requests.
Expiry:	1 minute
Type:	http_cookie

Name:	`wd`
Provider:	Facebook
Purpose:	Information about browser window dimensions.
Expiry:	1 day
Type:	http_cookie

Name:	`intercom-session-*`
Provider:	Intercom
Purpose:	Contact, communication, support.
Expiry:	1 week
Type:	http_cookie

Name:	`intercom-id-*`
Provider:	Intercom
Purpose:	Contact, communication, support.
Expiry:	9 months
Type:	http_cookie