Mostly indicates to assign a little something a status or place to differentiate it from others in a bunch, as in txt file is then parsed and can instruct the robot concerning which web pages are certainly not for being crawled. To be a internet search engine crawler may possibly https://www.youtube.com/watch?v=3GHWK6QaT2M