Following, you ought to make your mind up how you desire the crawler to manage different Web technologies. You can find an ongoing discussion in regards to the intelligence of online search engine crawlers. It is not solely crystal clear If they're full-blown headless browsers or just glorified curl https://www.youtube.com/watch?v=EVeTP0aV8gY