Back to Questions
4.7k
views9
comments
Async web requests - why are there two Async operations?
f#webasyncasynchronous

This pattern is seen in numerous examples of asynchronous web requests:
let loadWebStringAsync url =
async {
let req = WebRequest.Create(url: string)
use! response = req.AsyncGetResponse()
use reader = new StreamReader(response.GetResponseStream())
return! reader.AsyncReadToEnd()}The first AsyncGetResponse, when bound with "use!", executes the web request and gets a response asynchronously. Nice. No problems. But why is the response read asynchronously (AsyncReadToEnd)? Isn't it in memory now?

.

It might not be all in memory yet. That's why it's a stream. Below is a program that demos this (Client will print it has read some bytes before server has finished serving up the whole stream).

(I am at work, have not tried on Beta1, but I think it should all be Beta1-compatible.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

 

open System
open System.Net
open System.IO
open System.ServiceModel
open System.ServiceModel.Web
open Microsoft.FSharp.Control.WebExtensions 

// A stream implementation that lets you read 2M of bytes, 
// printing out as it gets halfway through
type MyStream() =
    inherit Stream()
    let NI() = raise <| new NotImplementedException()
    let mutable bytesLeft = 2000000
    let mutable gotHalfWay = false
    override this.CanRead = true
    override this.CanSeek = false
    override this.CanWrite = false
    override this.Length = NI() 
    override this.SetLength(_) = NI()
    override this.Position with get() = NI() and set(_) = NI()
    override this.Flush() = ()
    override this.Seek(_,_) = NI()
    override this.Write(_,_,_) = NI()
    override this.Read(buffer, offset, count) =
        if not gotHalfWay && bytesLeft < 1000000 then
            printfn "S: Served half the stream..."
            gotHalfWay <- true
            System.Threading.Thread.Sleep(200)
        if count <= bytesLeft then
            for i = 0 to count-1 do
                buffer.[offset+i] <- 1uy
            bytesLeft <- bytesLeft - count
            count
        else
            for i = 0 to bytesLeft-1 do
                buffer.[offset+i] <- 1uy
            let r = bytesLeft
            bytesLeft <- 0
            printfn "S: Served rest of the stream..."
            r

// A WCF HTTP GET server that returns the stream
[<ServiceContract>]
type IMyContract =
    [<WebGet>]
    [<OperationContract(Action="*")>]
    abstract member Foo : s:Stream -> Stream

type MyServer() =
    interface IMyContract with
        member this.Foo(s) =
            printfn "S: Starting to send result..."
            new MyStream() :> Stream

// The aforementioned async web code, instrumented to print at a few points            
let LoadWebStringAsync url =    
    async {         
        let req = WebRequest.Create(url: string)        
        use! response = req.AsyncGetResponse()        
        use reader = new StreamReader(response.GetResponseStream())        
        printfn "C: Received the stream"
        let a = Array.create 100 'a'
        reader.Read(a, 0, 100) |> ignore
        printfn "C: Read first 100 bytes"
        let! rest = reader.AsyncReadToEnd() 
        printfn "C: Finished reading entirety"
        return 0
    }

let Main() =    
    let addr = "http://localhost/MyService"
    let binding = new WebHttpBinding()
    binding.TransferMode <- TransferMode.StreamedResponse 
    let host = new ServiceHost(typeof<MyServer>)
    host.AddServiceEndpoint(typeof<IMyContract>, binding, addr) |> ignore
    host.Open()
    LoadWebStringAsync addr |> Async.RunSynchronously |> ignore
    host.Close()
Main()
By on 7/23/2009 12:23 PM ()

Hi, Brianmcn:
I have tried your code in VS2010 Beta 2 with F# 1.9.7.4. But I got a lot of compiler errors.
The important ones are:
The namespace 'ServiceModel' is not defined
The type 'ServiceContract' is not defined
The field, constructor or member 'AsyncReadToEnd' is not defined
The type 'WebHttpBinding' is not defined
The type 'ServiceHost' is not defined
In short, it seems the default settings in VS2010 Beta 2 with F# Application is missing quite a lot of references.
If you have time, is it possible that you try to compile your code in VS2010 Beta 2 and let me know how you can fix the issues?
Thanks,
John

By on 12/16/2009 2:56 PM ()

You need to add references to 

1
2
3

System.ServiceModel.dll
System.ServiceModel.Web.dll
FSharp.PowerPack.dll
By on 12/16/2009 3:21 PM ()

Hi, Brian:
Great! I have added the references, and it got compiled.
Now I have the question:
I want to use your code to get the stock quote in real time.
For example, for a single stock quote, let's say "MSFT", there is one URL for this, the server will send the quote in real time in binary format, each section of binary data is delimited by 2 Hex data: 0xFF 0x0A.
The data is rather small, less than 500 bytes for each section; but the data can come in any time when the stock market is open. For active stocks like MSFT, each day, the streaming quote data could come 10,000+ times, but the URL to get the same quote is always the same; but we don't know when it will come.
Let me know if I want to use your code listed above to retrieve the stock quote in real time, do I have to modify something or just provide the correct URL, the data will come.
But how I can save the incoming binary data in a stream and call another F# function to parse it?
If you have any idea, please show me some code sample. (We can now just say the URL is something like: [link:localhost] I have a function called Parser, which accepts the binary stream and provides the parse results)
Thanks,
John

By on 12/17/2009 1:28 PM ()

response.GetResponseStream() is a stream, just pass that to some function to parse it however.

By on 12/17/2009 1:34 PM ()

Hi, Brian:
I have tried your code with the stock quote ("DELL"), just want to see if your code can show some streaming data continuously. But I found it first printed "C: Received the stream"; then printed "C: Read first 100 bytes"; finally it showed "C: Finished reading entirety". Then no more messages displayed.
But the stock quote comes in real time, and never stops. Since I am doing the test when the stock markets close, it is not easy to verify it. But I have a tool, if provided with the correct URL, the tool can show the incoming stock quote in real time when market opens. Hope you understand my meaning.
For example, you can try this URL with your code, and let me know if you can verify the stock quotes come all the time, not only once, but many times?
Here comes the URL: [link:ameritrade02.streamer.com]

Thank you very much for your help.
John
PS: Wish you have a good Christmas holiday!

By on 12/18/2009 2:27 PM ()

I might be mistaken, but I don't believe Brian intended his code to continuously stream a web request. This is technically impossible, given the fact that HTTP does not allow for persistent connections like some other TCP-based protocols. In fact, because of HTTP's stateless request/response model, the client must repeatedly (re)issue the request. What Brain's code does, quite nicely, demonstrate, is incrementally fetching an individual request. This is useful for a variety of reasons (user experience, resource allocation, etc.); but for continuous retrieval, the client code should contain some sort of looping mechanism.

By on 12/18/2009 7:18 PM ()

Thanks very much for that Brian. It all makes sense.

I couldn't find a dll for Microsoft.FSharp.Control.WebExtensions, but I commented it out, and it compiled fine :)

It ran in the 1.9.6.16 release, and the showed the client getting the response before server had sent the whole stream

I like all that WCF code for a server!

I've done a couple of experiments too, and it looks like a normal web page is returned as part of the initial response. (I couldn't find a way to inspect the response object - I just disconnected the internet after reading it, then did the stream read with the internet off). So, IMHO, the second Async may be the more general approach, but may be unneccessary for normal web pages.

By on 7/24/2009 12:28 AM ()

Right, yes, I don't recall details of http protocol right now, but I think possibly you'll only see the difference in a case that corresponds to the WCF TransferMode.StreamedResponse case of the example (http streaming). Probably your typical 'web page' is not leveraging this aspect of http, which may mean that the entire stream is already in memory and you don't gain anything by 'going async' on the second call.

By on 7/24/2009 2:07 PM ()

    Topic tags

    IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
    Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy     		Built with WebSharper