Ciugk

Clash Royale CLAN TAG #URR8PPP Concurrent futures webscraping whoever is reading his! Thank you for taking the time to look at this. I am currently trying to develop a fast webscraping function so I can scrape a large list of files. This is the code I have currently: import time import requests from bs4 import BeautifulSoup from concurrent.futures import ProcessPoolExecutor, as_completed def parse(url): r = requests.get(url) soup = BeautifulSoup(r.content, 'lxml') return soup.find_all('a') with ProcessPoolExecutor(max_workers=4) as executor: start = time.time() futures = [ executor.submit(parse, url) for url in URLs ] results = for result in as_completed(futures): results.append(result) end = time.time() print("Time Taken: {:.6f}s".format(end-start)) this brings backs results for websites i.e www.google.com, however my problem is I have no idea to view the data it brings back I only get future objects. Please can som...

Clash Royale CLAN TAG #URR8PPP What kind of magic does this website use to detect bots? I'm currently working on a scarper. While testing, I found this website: http:// a t c a v o c a t s .com/ This website works fine on a browser, but I get a timeout when scrapping. No matter what I do: I follow redirects, I activated cookies, I use a keep-alive connection, and I'm using all headers that Chrome ordinarily use { 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp, / ;q=0.8', 'Accept-Encoding': 'gzip, deflate, sdch, br', 'Accept-Language': 'en-US,en;q=0.5', 'Referer': 'http://google.com/', 'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661...

Clash Royale CLAN TAG #URR8PPP Scrapy Login Authentication I am fairly new to scraping, and I have a page I would like to login to so that I can scrape. From the Scrapy documentation, it appears we should use FormRequest. Here is the example that is given in the documentation: import scrapy class LoginSpider(scrapy.Spider): name = 'example.com' start_urls = ['http://www.example.com/users/login.php'] def parse(self, response): return scrapy.FormRequest.from_response( response, formdata={'username': 'john', 'password': 'secret'}, callback=self.after_login ) def after_login(self, response): # check login succeed before going on if "authentication failed" in response.body: self.logger.error("Login failed") return # continue scraping with authenticated session... However, this seems oversimplified in most case...

Clash Royale CLAN TAG #URR8PPP looking to get dynamic website value using selenium and phantomJS i am trying to get the value for a timer >http://prntscr.com/kcbwd8 on this website > https://www.whenisthenextsteamsale.com/ and hopefully store it in a variable. import urllib from bs4 import BeautifulSoup as bs import time import requests from selenium import webdriver from urllib.request import urlopen, Request headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.3"} browser = webdriver.PhantomJS() browser.get('https://www.whenisthenextsteamsale.com/') soup = bs(browser.page_source, "html.parser") result = soup.find_all("p",{"id":"subTimer"}) for item in result: print(item.text) browser.quit() i have tried using the code above but it returns this error > C:UsersroberAnaconda3libsite-packagesseleniumwebdriverphantomjswebdriver.py:49: User...

Clash Royale CLAN TAG #URR8PPP How do I integrate a web scraper into a working website? I'm working on a project that requires web scraping to retrieve data from a website. I want to create a website that allows the user to input data, like a few keywords, and have a scraper scrape data relating to that keyword and then display that data on the website. I've gotten the scraper part down, but now I'm not sure how to make that into a website. Does that mean I'd have to integrate Python (web scraping lang) into HTML? I have to install packages to scrape too (like BeautifulSoup), will this impact the feasibility of the project? Overall, is this possible, and if not, what alternatives are there? Thanks in advance. By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to...

Clash Royale CLAN TAG #URR8PPP Can't figure out Beautifulsoup find() command for this HTML I am trying to scrape some info from a page with python and Beautiful soup and i cant seem to write the right path to what i need, the html is: <div class="operator active" data-operator_name="Etisalat" data- operator_id="5"><div class="operator_name_etisalat"></div></div> And i am trying to get that operator name "Etisalat", i got this far: def list_contries(): select = Select(driver.find_element_by_id('international_country')) select.select_by_visible_text('France') request = requests.get("https://mobilerecharge.com/buy/mobile_recharge?country=Afghanistan&operator=Etisalat") content = request.content soup = BeautifulSoup(content, "html.parser") # print(soup.prettify()) prov=soup.find("div", {"class": "operator active"})['data-operator_name']...