Back to Questions
639
views2
comments
Need help chunking a sequence
f#

Hello, all,

I have an indefinitely long sequence of records of the form:
'tag * 'value * 'day
sorted by the value of 'day.

I wish to transform this into a sequence of Day records of the form:
'day * Map<'tag, value>
There can be many (100Ks) of records for a given day, and there can be many (1M or so) days. Seq seems a good bet since it supports the streaming paradigm I want here.
DSymes' "lines_of_file" gets the stream started fine. My problem is I can't figure out how to yield a sequence of chunks (Days) -- I can generate the first chunk but don't know how to iterate the outer Seq.

I can insert a special record with tag = 'end-of-record' after each set of records for a given day if that makes the solution easier (in fact, that's what I'm doing below).

I want to stay with lazy eval because of the large and indeterminate size of the input.

Here's my latest try -- I'd rather return a Map than a Dictionary for each chunk, but was trying a mutable structure.
I'd be very grateful if anyone can point me in the right direction on this,

thanks
David Glaubman
Risk Management Solutions

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

// Code sample follows
open System.Collections.Generic
 
type RawLossRecord = string * int64 * int32
type LossRecord = string * int64
 
let makeRawLossRecord( line:string ) =
   let array = line.Split [| ' ' |] 
   array.[0], int64 (array.[1]), int32 (array.[2])
 
let lines_of_file name =
   seq { use stream = File.OpenRead name
         use reader = new StreamReader( stream )
         while not reader.EndOfStream do
            yield reader.ReadLine()
      }
 
let records_of_seq (sequence: RawLossRecord seq) =
   seq { let record = new Dictionary<string, int64>()
         for (tag, loss, day) in sequence do
            if tag <> "EOR" then
               record.Add( tag, loss)
            else yield day, record
      }

// Driver
let filename = "c:/sample.txt"
let input = lines_of_file filename
input |> Seq.map makeRawLossRecord |> records_of_seq |> Seq.iter (printf("%A\n") )

// c:/sample.txt
tag 100 1
tag2 120 1
EOR 0 1
tag4 140 2
EOR 0 2
.

I guess I didn't test the latest try enough before I posted it. I can get the desired output by modifying the outer (second?) sequence to keep track of the chunk record's state and clear it after a yield.
This feels like a kluge, so I'd still be delighted to find out how to do it a la mode f#. The problem seems like a simple and common one -- surely there is an elegant answer.
thanks
David

// Ugly, but works

1
2
3
4
5
6
7
8
9
10
11
12
13

let records_of_seq (sequence: RawLossRecord seq) =
   seq { let resetDictionary = ref true
            let record = new Dictionary<string, int64>()
            for (tag, loss, day) in sequence do
                  if tag <> "EOR" then
                        if !resetDictionary then
                              record.Clear()
                              resetDictionary := false
                              record.Add( tag, loss)
                  else
                        resetDictionary := true 
                        yield day, record
}
By on 7/20/2009 4:18 PM ()

Take a look at Seq.scan

By on 7/20/2009 4:32 PM ()

    Topic tags

    IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
    Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy     		Built with WebSharper