In order to avoid unwanted content from the look for indexes, site owners can instruct spiders never to crawl selected data files or directories in the regular robots.txt file in the foundation directory from the area. Also, a website page could be explicitly excluded from the online search engine's database https://news.rafeeg.ae