10. Gzip Compression: Help Gzip compression with your server to reduce the size of files transferred in between the server along with the user’s browser. txt file is then parsed and can instruct the robot regarding which webpages aren't to become crawled. To be a internet search engine crawler could https://www.youtube.com/watch?v=0xsCkQ6bbjk