Improve crawling performance
Scrapy get a dedicated page about broad crawling
I will try to :
- follow some recommendations that fit our case (BFO order, memory usage...)
- Implement a parrallel crawler
An other useful link, on how to crawl dynamic loaded content