Setting search parameters for sites NOT using URL-encoding?

Hello everyone,

I have a Windows Forms application that sends automated search requests to several websites related to cars, and displays the results after parsing the responses. (using HttpWebRequest)

The typical search page allows you to search for cars with given specs. These parameters (model year, price, car name, etc) are selected from listboxes or typed into edit fields. It is easy automating this search request when the parameters are url-encoded.

The issue with one of the sites is that the parameters I set do not appear in the url (the url remains fixed always), so I cannot recreate this search request by sending a url with url-encoded parameters. I tried viewing the page source to get to the definitions of the listboxes and editfields in question, to set their values through code, but until now I've had no luck.

Any pointers or suggestions on how to automate this case in a Windows Forms will be greatly appreciated.

Regards,

Jack

jack johnson said:

Hello everyone,

I have a Windows Forms application that sends automated search requests to several websites related to cars, and displays the results after parsing the responses. (using HttpWebRequest)

The typical search page allows you to search for cars with given specs. These parameters (model year, price, car name, etc) are selected from listboxes or typed into edit fields. It is easy automating this search request when the parameters are url-encoded.

The issue with one of the sites is that the parameters I set do not appear in the url (the url remains fixed always), so I cannot recreate this search request by sending a url with url-encoded parameters. I tried viewing the page source to get to the definitions of the listboxes and editfields in question, to set their values through code, but until now I've had no luck.

Any pointers or suggestions on how to automate this case in a Windows Forms will be greatly appreciated.

Regards,

Jack

Hi Jack,

Thank you for posting your question.

Besides the query-string based search queries in sites, many times the state of search (search filters) in sites can be saved in many other ways, including Session, Cookies or Form Posting with the values entered as filter. Sometimes, it may depend of the technology used for building the website (in some technologies some of these methods is more often used by developers, in other technologies another methods).

If they keep the value of the search filter in Session, you can't do much except making some automated mechanism that will automatically set the values to the website (as used has done that manually) and submit the search... But, this might get a lot painful for developing... Imagine, if they change even small thing in how everything works, your 'automated' system will get broken and won't work at all.

If they make submit to the search page using POST, you may try to add headers and make your HttpWebRequest of type POST. Try to post some values and see what happens and if they recognize in their server-side code as posted values. You can check their form input type element ids/names to get hint of the keys of values you need to post.

If you don't have access to the site or you can't get response from site owner how they do that, it would be hard for you to complete it, because in some cases, malicious users may try to perform malicious activities, injections and hackings. Thus, these are the methods to try eligibly to do something.

Hope this helps.

Hello,

Thank you for your detailed and helpful reply. I went through some of the cases you mentioned. HttpWebRequest with POST, using their field names/ids and using a namevaluecollection didn't make a difference, and the values for these fields remained empty after the POST.

I also retrieved the site's cookies and examined their contents, but didn't come accross any of the search values saved within these too.

It seems one of the few options left for me is to check if they save such variables in Session. However, I am working within a Windows Forms application and Session variables do not exist for windows apps. If you could point me out at a way to get session variables for a request from a windows app. I would give this case a try too.

Once again I thank you for your time and effort your putting to answer my tedious questions :)

Best Regards

Jack