Back to Questions
1.5k
views8
comments
Reading large data files
f#

I want to read large numeric data sets stored in text files (with unknown coding) into a matrix (powerpack). What is the most efficient code for this?

.

Do you have any information on how the data is written in the text file? That will certainly make a difference in how you handle its input.

If the file is essentially a listing of row vectors, you can read the file, lion-by-line into a seq of strings, split the strings using whatever token you need, then create a matrix using Matrix.ofSeq.

By on 6/22/2010 2:02 PM ()

The data is stored in varying numbers of columns. They could be separated by commas, tabs or spaces. And, the encoding is unknown.

By on 6/23/2010 12:25 AM ()

Do all rows in any given file have the same number of values?
i.e., can a file be represented as a matrix?

If so, as pointed out above, you could do something like:

1
2
3
4
5
6
7
8
9
10
11

let readFile file =
    use reader = System.IO.File.OpenText(file)
    seq {
        while not reader.EndOfStream do
            let line = reader.ReadLine()
            if line.Trim().Length > 0 then
                yield line.Split([| ' '; '\t'; ',' |], System.StringSplitOptions.RemoveEmptyEntries)
    } |> Seq.toList


let mat = matrix (readfile "path")

If the lines were all different in length, then you could avoid turning it into a matrix, and just handle it as a list of lists, or turn it into a list of maps, each map mapping a 'column' index to a value.

By on 6/24/2010 3:25 AM ()

This yields the following error:

The type 'float' does not match the type 'string'

By on 6/24/2010 7:47 AM ()

Well, yeah. matrix = Matrix<float>

Try:

1
2
3
4
5
6
7
8
9
10
11

let readFile file =
    use reader = System.IO.File.OpenText(file)
    [
        while not reader.EndOfStream do
            let line = reader.ReadLine()
            if line.Trim().Length > 0 then
                yield line.Split([| ' '; '\t'; ',' |], System.StringSplitOptions.RemoveEmptyEntries) |> Array.toList
    ]


let mat = Matrix.Generic.ofList (readFile "path")
By on 6/24/2010 8:10 AM ()

Good, I now get the values into the matrix. However, they are stored as:

val mat : Matrix<string> = matrix

Is it possible to store them as float directly?

By on 6/24/2010 8:40 AM ()

I they are all floats, you can do:

1
2
3
4
5
6
7
8
9
10
11

let readFile file =
    use reader = System.IO.File.OpenText(file)
    [
        while not reader.EndOfStream do
            let line = reader.ReadLine()
            if line.Trim().Length > 0 then
                yield
                    line.Split([| ' '; '\t'; ',' |], System.StringSplitOptions.RemoveEmptyEntries)
                    |> Array.map (fun s -> System.Double.Parse(s))
                    |> Array.toList
    ]
By on 6/24/2010 8:50 AM ()

Or even more concisely:

.....
|> Array.map float
.....

By on 6/24/2010 9:31 AM ()

    Topic tags

    IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
    Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy     		Built with WebSharper