downloading a web p...
 
Notifications
Clear all

downloading a web page

3 Posts
2 Users
0 Likes
438 Views
PaulSanderson
(@paulsanderson)
Posts: 651
Honorable Member
Topic starter
 

I have an odd little background task that is vexingme slightly.

I want to download a web page that displays certain offers that can be filtered by a date range and location and then the resultant offers are grouped and displayed 10 per page. You can view each group of pages using pre/next buttons or selecting page 1 of x from a drop down menu.

The web page does not use html variables ie on filtering the url remains the same.

I want to programatically select a date range and then select and save each page - an example of some of the underlying code is

<div class="offer-nav">
Skip to
<select onchange="webpage.offer.goToPage(0,'', $(this).val(), $('.filterLocn').val(), $('.fliterDate').val(), 1234);">
<option value="1" selected="selected">
Page 1
</option>
<option value="2">
Page 2
</option>
<option value="3">
Page 3
</option>
</select>
</div>

Does anyone have any pointers to a resource that might explain how I can go about this?

This is a bit of an intellectual exersize as I have saved the pages I want manually (there weren't too many) but it got me wandering how one would automatically save a web page where the data behind is the result of a filter of this sort.

 
Posted : 23/01/2017 12:51 am
(@jdcoulthard)
Posts: 98
Trusted Member
 

Hi Paul,

You could take a look at Python and Selenium for this exercise.

If you identify the form fields by XPath, you can change variables and submit forms. You can then either save the resulting HTML or parse it with something like BeautifulSoup.

HTH

James

 
Posted : 23/01/2017 1:43 am
PaulSanderson
(@paulsanderson)
Posts: 651
Honorable Member
Topic starter
 

Thanks james

Just had a very brief look - seems to be just what I want.

P

 
Posted : 23/01/2017 2:09 am
Share: