Technological Web optimization: Specialized Search engine optimization refers to Web page and server optimization that assists the crawler in crawling, indexing and ranking operations to rank your website greater. txt file is then parsed and can instruct the robotic as to which web pages will not be to become crawled. https://www.youtube.com/watch?v=0eLlfoHTk_E