Posts

Showing posts with the label python-requests

beautifulsoup for looping and getting text and Href

Image
Clash Royale CLAN TAG #URR8PPP beautifulsoup for looping and getting text and Href Im in a bit of a quinch here: its an ASP site which is really messy that I am trying to get data from: Im trying to use a FOR loop to get an href and the text of all the rows of the 4th table that is on the site, so I first did: table = soup.findAll('table')[3] Then from this table I need to get all text inside the tags and the href's of the inside. i tried something like this: for product in table.findAll('tbody'): product_title = product.find('tr').text product_link = product.find('a')['href'] print (product_title, product_link) But I get nothing in return... :( Can you show a print of product ? – Rakesh 7 mins ago product What does your HTML document look like? –...

Combining Multiprocessing.map() and itertools.cycle in python3

Image
Clash Royale CLAN TAG #URR8PPP Combining Multiprocessing.map() and itertools.cycle in python3 my question is how can I utilize itertools.cycle with multiprocessing in a way that my end result allows me to iterate through the 'accounts' file which is an email:password list as well as provide a new proxy to each request I make(in each process created with multiprocessing.map). This is somewhat hard to explain so I have provided my current code to help illustrate my problem. In each of the allocated processes(processes=10), the email combination changes for every post request but the proxy changes every 10 requests. I want every request to be completely unique. My issue seems to be that I am relying on each process created to switch the proxy with the next() function. Results are below. Thanks in advance! import requests def main(account): proxy_switch = next(proxy_pool) proxy = {"http": proxy_switch} requests.post(url, proxies=proxy) accounts=open('file....