downloading a web page
I have an odd little background task that is vexingme slightly.
I want to download a web page that displays certain offers that can be filtered by a date range and location and then the resultant offers are grouped and displayed 10 per page. You can view each group of pages using pre/next buttons or selecting page 1 of x from a drop down menu.
The web page does not use html variables ie on filtering the url remains the same.
I want to programatically select a date range and then select and save each page - an example of some of the underlying code is
Does anyone have any pointers to a resource that might explain how I can go about this?
This is a bit of an intellectual exersize as I have saved the pages I want manually (there weren't too many) but it got me wandering how one would automatically save a web page where the data behind is the result of a filter of this sort.
You could take a look at Python and Selenium for this exercise.
If you identify the form fields by XPath, you can change variables and submit forms. You can then either save the resulting HTML or parse it with something like BeautifulSoup.
Just had a very brief look - seems to be just what I want.