Tuesday, October 9, 2007

Using exsiting IE cookies with HttpRequest

I've come accross an idea of making my own web spider with .Net. My tests were to grab information from the Experts-exchange site. I've used a code like this

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

The request passes correct and getting the response was working well also. However the content of the response was not the correct one - I mean that it was different than the one I can see if I open an explorer.

What was the difference? The difference was the cookies! When I open the Internet Explorer and navigate to EE the EE server accepts me as logged in user using my cookies. However using the programatically created web request no cookies are associated with the request and the server does not show me the information I want.

During some research and some test I've come across the InternetGetCookie function. This function allows you to get the cookies stored by the Internet Explorer and assign them to your web request.

Here is the code I've come to:



public class RequesstWithCookie
{


[DllImport("wininet.dll", CharSet = CharSet.Auto, SetLastError = true)]

static extern bool InternetGetCookie(string lpszUrlName, string lpszCookieName,
[Out] StringBuilder lpszCookieData, [MarshalAs(UnmanagedType.U4)] out int lpdwSize);


CookieContainer cookies;


public RequesstWithCookie(String siteUrl)

{

LoadCookies(siteUrl);

}


public HttpWebRequest CreateRequest(string pageUrl)

{

Uri uri = new Uri(pageUrl);


HttpWebRequest request = null;
request = (HttpWebRequest)WebRequest.Create(uri);
request.CookieContainer = cookies;


return request;

}


private void LoadCookies(String url)
{


cookies = new CookieContainer();
Uri uri = new Uri(url);


StringBuilder cookieBuilder = new StringBuilder(new string(' ', 256), 256);

int cookieSize = cookieBuilder.Length;


if (!InternetGetCookie(url, null, cookieBuilder, out datasize))

{

if (datasize <> 0)

{

return;

}


cookieBuilder = new StringBuilder(datasize);

InternetGetCookie(url, null, cookieBuilder, out datasize);

}


cookies.SetCookies(uri, cookieBuilder.ToString().Replace(";", ","));

}

}




Hope this code will help somebody else to not lose an hour or more to find this out.

3 comments:

Anonymous said...

Dude! what do you mean by
if (cookieSize <>
{
return;
}

Anonymous said...

Martin, what does the following do?

if (cookieSize <>
{
return;
}

Martin Marinov said...

Sorry for this error. Actually the member is datasize and not cookieSize. Also the comparison is with 0. The post is updated.