Ciugk

Clash Royale CLAN TAG #URR8PPP Scrapy Login Authentication I am fairly new to scraping, and I have a page I would like to login to so that I can scrape. From the Scrapy documentation, it appears we should use FormRequest. Here is the example that is given in the documentation: import scrapy class LoginSpider(scrapy.Spider): name = 'example.com' start_urls = ['http://www.example.com/users/login.php'] def parse(self, response): return scrapy.FormRequest.from_response( response, formdata={'username': 'john', 'password': 'secret'}, callback=self.after_login ) def after_login(self, response): # check login succeed before going on if "authentication failed" in response.body: self.logger.error("Login failed") return # continue scraping with authenticated session... However, this seems oversimplified in most case...

Clash Royale CLAN TAG #URR8PPP Cycle in Scrapy ItemLoader https://doc.scrapy.org/en/latest/topics/loaders.html#using-item-loaders-to-populate-items from scrapy.loader import ItemLoader from myproject.items import Product def parse(self, response): l = ItemLoader(item=Product(), response=response) l.add_xpath('name', '//div[@class="product_name"]') l.add_xpath('name', '//div[@class="product_title"]') l.add_xpath('price', '//p[@id="price"]') l.add_css('stock', 'p#stock]') l.add_value('last_updated', 'today') # you can also use literal values return l.load_item() But if I get from webpage 2 names, prices, etc, how to add it to l.load_item() ? l.load_item() Because I added the cycle but if in the end, I write return cycle will work only once. How to right to do it? return By clickin...