Back to Questions
4.7k
views1
comment
WebRequest returning 403 forbidden error
f#webhtml

I am trying to connect to the distrowatch web site using a code snippet from the WebCrawl example, but am getting a 403 Forbidden error. This occurs at the req.GetResponse() line. I think this is caused by the web site's security policy rejecting a request from an unidentified requestor, so I have been trying to setup the WebRequest to look like a standard web browser such as Mozilla. I saw a C# example where UserAgent was used with HttpRequest, like this: req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1; )" - but I cannot get this to work in F#. Below is the code snippet. It works fine with the google url. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14

open System.Net
open System
open System.IO

let http(url: string) = 
  let req = WebRequest.Create(url) in
  let resp = req.GetResponse() in
  let stream = resp.GetResponseStream() in
  let reader = new IO.StreamReader(stream) in
  let html = reader.ReadToEnd() in
  resp.Close();
  html
let distro = http("http://distrowatch.com")
let google = http("http://www.google.com")

Any suggestions appreciated.

.

I received the same error when I tried to access the site without setting the user-agent header. I was able to get the distrowatch page with the following code (WebClient is much easier to use than WebRequest):

1
2
3
4
5

#light
open System.Net
let client = new WebClient()
client.Headers.["user-agent"] <-  "Mozilla/4.0 (compatible; MSIE 6.0;Windows NT 5.1"
let distro = client.DownloadString("http://distrowatch.com")

Hope that helps!

By on 8/16/2008 8:01 PM ()

    Topic tags

    IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
    Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy     		Built with WebSharper