How can I download full webpage by a Python program?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP


How can I download full webpage by a Python program?



Currently I have a program that can only download the HTML of a given page. Now I want a program that can download all the files of the web page including HTML, CSS, JS and image files(Same as we get on ctrl-s of any website).



My current program is:


import urllib
urllib.urlretrieve ("https://en.wikipedia.org/wiki/Python_%28programming_language%29", "t3.html")



I have visited many such questions in Stack Overflow, but they are all only downloading the HTML file.





So you want to go through the links in the HTML and download the content they point to? Note that a Wikipedia page contains links to other pages; do you want to do that recursively?
– jonrsharpe
Jul 3 '15 at 11:21





Yes i want to download all the links in the main link along with their css and js files.
– Rahul Satal
Jul 3 '15 at 11:29





Or just tell me how to download only one given pages css and js files
– Rahul Satal
Jul 3 '15 at 11:41





Decompose the problem. Break it down into small steps, and research each one separately. You know how to get the first page, so now work out how to extract the links you want from the HTML (hint: this is called parsing).
– jonrsharpe
Jul 3 '15 at 12:11







@jonrsharpe i just know to download the HTML of first web page but its css files are not downloading
– Rahul Satal
Jul 3 '15 at 12:25




3 Answers
3



The following implementation enables you to get the sub-HTML websites. It can be more developed in order to get the other files you need. I sat the depth variable for you to set the maximum sub_websites that you want to parse to.


depth


import urllib2
from BeautifulSoup import *
from urlparse import urljoin


def crawl(pages, depth=None):
indexed_url = # a list for the main and sub-HTML websites in the main website
for i in range(depth):
for page in pages:
if page not in indexed_url:
indexed_url.append(page)
try:
c = urllib2.urlopen(page)
except:
print "Could not open %s" % page
continue
soup = BeautifulSoup(c.read())
links = soup('a') #finding all the sub_links
for link in links:
if 'href' in dict(link.attrs):
url = urljoin(page, link['href'])
if url.find("'") != -1:
continue
url = url.split('#')[0]
if url[0:4] == 'http':
indexed_url.append(url)
pages = indexed_url
return indexed_url


pagelist=["https://en.wikipedia.org/wiki/Python_%28programming_language%29"]
urls = crawl(pagelist, depth=2)
print urls



Try the Python library Scrapy. You can program Scrapy to recursively scan a website by downloading its pages, scanning, following links:



An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.





Thanks @barny , but can you please tell can it be implemented using beautifulSoup lib or HTTP Requests bcoz i am having some knowledge of it.
– Rahul Satal
Jul 3 '15 at 11:39





Good heavens, my Answer has been Revised. Read the python, err, Python library Scrapy documentation, for example the FAQ says as its first answer: Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead. doc.scrapy.org/en/1.0/faq.html
– barny
Jul 3 '15 at 12:08







ok let me first read the scrapy doc.
– Rahul Satal
Jul 3 '15 at 12:26



You can easily do that with simple python library 'pywebcopy'



import pywebcopy



pywebcopy.core.init('http://www.vodafone.de/privat/tarife/red-smartphone-tarife.html')



Done!



You will have your complete webpage in the following folder



C:WebCopyProjectsvodafone.de



You can choose to change this location by passing mirrors_dir='your-choice-location to init call.. Ex.



from pywebcopy.core import init



init(
url = 'http://your-site.com/',
mirrors_dir = 'your-choice-location`
)



You will have html, css, js all at your-choice-location. Completely working like original site..



P.S. I have written pywebcopy






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

Arduino Mega cannot recieve any sketches, stk500_recv() programmer is not responding

Image
Clash Royale CLAN TAG #URR8PPP Arduino Mega cannot recieve any sketches, stk500_recv() programmer is not responding Yesterday we were trying to update the firmware of our 3D-printer a Tevo Tarantula i3 The printer was running Marlin 3D Printer Firmware. We installed a new module a TL-Touch Sensor, a chinese copy of the Bl-Touch an auto bedleveling touch sensor for a 3D printer. After re-configurating a new version of the firmware for our setup in the Arduino IDE we went on with uploading the sketch to the Arduino Mega that was controling our 3D printer. Before we started uploading we made sure the right COM-port was selected in the Arduino IDE and that also the right type of Arduino was selected. The Arduino Mega 2260. Then we started with compiling the firmware and then uploading it to the Mega, Somewhere in between that moment the IDE freezed and crashed. As a response we removed the USB cable that was connected from the PC uploading the firmware onto the Arduino Mega, (This was a st...
Read more

Visual Studio Code: How to configure includePath for better IntelliSense results

Image
Clash Royale CLAN TAG #URR8PPP Visual Studio Code: How to configure includePath for better IntelliSense results so I am a complete beginner to using Visual Studio Code and I have no clue what I am doing. I've searched around (maybe not enough), but I cant find just a simple explanation for someone like me on how to configure the c_cpp_properties.json file that I am redirected to whenever I click on the yellow lightbulb next to a line that is underlined with a green squiggle. Lightbulb example c_cpp_properties.json I just want to know what to put in the .json to make IntelliSense work properly This question has answers that may be good or bad; the system has marked it active so that they can be reviewed. 2 Answers 2 From: https://code.visualstudio.com/docs/languages/cpp Below you can see that the MinGW C++ include path has been added to browse.path for Windows: { "name": "Win32...
Read more

Better method to check all objects' parameters with a single call

Image
Clash Royale CLAN TAG #URR8PPP Better method to check all objects' parameters with a single call I want to get all objects which age is less than 18. How can easily do that? Should I had to pass all objects to get_nonage to get it? class People: def __init__(self, uid, age): self.uid = uid self.age = age def get_nonage(self): # print nonage here which age<18 # expected output: {1: 11, 3: 15} pass p1 = People(1, 11) p2 = People(2, 20) p3 = People(3, 15) ... People.get_nonage() People.get_nonage() First, get_nonage should probably not be an instance method. Second, yes, you will have to pass it all of the People object if you want it to search all of the People objects. But that implies that you should be storing them in a list, instead of in a bunch of separate variables. Then you just pass the list. – abarnert 13 mins ago ...
Read more