1. Home

Product groups

Support

Can you scrape content with WodHttpDLX in Delphi - WeOnlyDo Discussion board

Can you scrape content with WodHttpDLX in Delphi (wodHttpDLX)

by buddy, Sunday, October 11, 2009, 22:17 (5522 days ago)

Hey,

I have just recently started messing with WodHttp and so far love what I see although im still unsure what capabilities it can do... Right now Im working on a project where I need to post items to a web site then scrape the information that is give back... I was curious if WodHttpDLX could do this or not?

I know it can post the info for as i have that part coded but was not sure if it could then scrape the content that is returned. I know that there is a .Get property however I think that is just for the HTML source but could be wrong...

Here is a small example of what I want it to do....
Go To tinyurl.com
Then post link...
then scrape the new url that tiny url has gathered...

Very simple example but again curious if wodhttp has the skills to get it done lol...

I am coding in Delphi 7 and Delphi 2009 either one is fine as I have both.

locked
  4307 views

Re: Can you scrape content with WodHttpDLX in Delp

by wodDamir, Sunday, October 11, 2009, 22:41 (5522 days ago) @ buddy

Hi Buddy,

When you do a POST, the component automatically retrieves the response returned by server.

However, since tinyurl.com returns the URL in a webpage, the response will contain whole HTML source of the page. You will need to parse response body in order to find the URL.

Hope this helps.

Regards,
Damba

locked
  4150 views

Re: Can you scrape content with WodHttpDLX in Delp

by buddy, Sunday, October 11, 2009, 23:33 (5522 days ago) @ wodDamir

Ah I see....

Im not sure if I quite understand when u say it automatically retrieves the response by the server? Could u explain or give an example of this?

As for the tinyurl i see what you mean... It would be best if i just take the source of the page and then trim it down to what i need exactly.

Lets say if I want to do the following...
Go to Craigslist.org and search for all Mazdas for sale and then scrape all ads for that... Would the Get function be able to do that or any other function of WodHttp?

Thanks all ur help is much appreciative :D

locked
  4130 views

Re: Can you scrape content with WodHttpDLX in Delp

by wodDamir, Sunday, October 11, 2009, 23:50 (5522 days ago) @ buddy

Buddy,

Post method works the same as Get. The only difference is that Post actually sends provided data, while Get only retrieves the requested page.

What I mean is when you call Post method, it's the same as if you pressed the Submit Button ( Make TinyURL! ). The url you provided is submitted to page, and the request is processed. After that, the server provides response, which you receive as HttpResponse object.

If you check Response at that point, you should have the page source in Response.Body property.

The same applies to the other sites. You need to submit your search criteria, so I assume you will need to use Post method. However, extracting the data you require from the response is up to you.

Regards,
Damba

locked
  4121 views

Re: Can you scrape content with WodHttpDLX in Delp

by buddy, Monday, October 12, 2009, 00:00 (5522 days ago) @ wodDamir

Ah now i fully understand.... LOL sorry im still new to coding in delphi as well... So sorry for so many questions and etc. But again I do appreciate how much u have helped me and being so patient with me as well.

Im glad if i have any questions I can come to u guys lol...

Thanks again

locked
  4149 views

Count us as a satisfied WeOnlyDo customer. We appreciate your terrific support to get the secure Telnet working properly.

Don Fitzpatrick
InterNetworx Systems

Congratulations on an excellent product and first rate support!

Sander Pool
Rhapsody Networks

I can only hope I will have the pleasure to work with other products by "We Only Do" in the future.

Tomer Spivak
Verint Systems Ltd

Thanks! We love your software and your great support!

Sandy Fliderman
Primary Development Corporation

Brilliant, even works on the mobile phone...

Conrad Leigh
Leaf Wireless

Many thanks for your excellent support service. I can see why you have a long list of excellent testimonials on your website.

Martyn Walker
Batmore Technolgies

Brilliant, even works on the mobile phone...

Conrad Leigh
Leaf Wireless

May I say how nice it is to work with components so complete and self-explanatory that using them in my application is as instinctive as though I wrote them myself?

Michael Barnett
Full Spectrum Telecommunications

You made it so simple for us to integrate SFTP - a technology we had little experience with...

Zack Menendez
Mapframe Corporation

We have recently completed a large 1200+ location project where reliable Telnet and SSH connectivity was required. wodSSH worked flawlessly and allowed our staff to complete our project ontime and within budget.

James Sievers
Harbor Technologies