Back to Questions
7.2k
views17
comments
Writing arrays to a file
f#

Hi. Very simple question for you.

How do you get an array of double precision floats into a file?

I need the file to be one of the following formats (whichever is easiest):

"CSV", "TSV", "XLS", "HarwellBoeing", "MAT", "MTX", "DIF", "FITS", "HDF5", "MPS", "SDTS"

e.g. if you had a function:

export array:array2D filename:string

how would that function be defined?

Ideally it would be possible to export several arrays of different dimensions to the same file (1 or 2 1D float arrays corresponding to axes, one 1D or 2D float array for data, and a few strings for labels).

I'm used to scientific languages, and initially found F# very productive, but have been stuck for a long time at the point of plotting graphs, and really need to get some data into Mathematica to plot asap. I couldn't find a way to plot good 3D graphs from F#, or to control a Mathematica notebook from F#, so the data will have to go into Mathematica via files.

Thanks for any help!

.

Hello

Great question. I hadn't (really) noticed the HDF5 file format before now. I think I will give that some thought.

The file format is described at this site [link:www.hdfgroup.org] . Also the needed DLLs (the API they call it) are found here.

There is an .Net-wrapper (running with framework 2.0 it seems) here ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/hdf5/Csharp-prototype/ .

Usage is something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

let myNativeData = [| 3.; 1.; 4.; 1.; 5.; 9.; 2.; 6.; 5. |]

#r @"HDF5DotNet.dll"
open HDF5DotNet

let myFile = H5F.create(@"MyFile.hdf5", H5F.CreateMode.ACC_TRUNC)
// cannot be done: let myArray = new H5Array<double> (myNativeData)
let myArray = null: H5Array<double>
let myDataTypeId = new H5DataTypeId (H5T.H5Type.IEEE_F64LE)
let myDataSpaceId = H5S.create_simple (1, [| uint64 (myNativeData.Length) |])
let myDataSetID = H5D.create(myFile, "MyDataSetsName", myDataTypeId, myDataSpaceId)
do H5D.write(myDataSetID, myDataTypeId, myArray);
do H5F.close myFile

The problem is that F# throws a compile error my way because the class H5Array has some nice multi-dimensional constructors. Maybe someone knows how to override this behaviour(?).

Best regards
Robert Nielsen

By on 12/1/2010 3:35 PM ()

Hello

Great question. I hadn't (really) noticed the HDF5 file format before now. I think I will give that some thought

Hi Robert. This sounds promising: let me know what you find!

By on 12/1/2010 3:41 PM ()

Hi CSMR

I've googled a bit.

At this site Unidata Program Center I found a project called "netCDF". They provide some DLLs that allow us to write HDF5-files. It appears that writing HDF5-files is only a secondary objective for them, so the functionality is a bit crude compared to "The HDF Group"-library mentioned in my earlier answer. Sometimes simplicity is what we want though, so why not give this 'solution' a test run.

First download the newest set of

software libraries, it is the five DLLs found here netcdf-4.1.1-win32 . The zip file have more files than just the five DLLs but only the DLLs are needed as far as I can tell. The DLLs must be accessible to your application at run time.

Somebody has written a set of DLLImports for us. Check this file dotNet-wrapper , particularly the file called "NetCDF.vb". It bears this text "Copyright 2004 University Corporation for Atmospheric Research/Unidata" and also this "This VB.NET wrapper created by Ed Hartnett, 3/10/4". Credits go to the author of course. With that in mind let us choose not to use the actual wrapper, as it is a bit old (in a computer perspective) now.

I will post (only) the most important DLLImports in F#-syntax shortly. The idea is to get this usage

1
2
3

open HDF5Write
HDFSaveMatrixToFile @"c:\myhdffile.hdf" "A"
 <| (array2D [ [ 1.1; 1.2; 1.3]; [2.1; 2.2; 2.3] ])

There is a tool called "HDFView" found here HDFView . I have run the tool, and see the matrix "A" - or array if you like - showing up, so there is light at the end of the tunnel [:)] .

Let me know if it turns out workable for you.

Best regards
Robert

p.s. the above ftp links are
ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf-4.1.1-win32-bin.zip and
ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/win32/netcdf_vb_net_wrapper.zip

By on 12/5/2010 5:48 AM ()

Hello again

I tried using the technique myself at a computer at work. The computer at work is newer than my old one at home, so I noticed that using the 'platform target' "Any CPU" was no longer good enough. The netCDF DLL could not be loaded (into the executing "x64"-application). The netCDF DLL loads fine at home (into a "x86"-application). The 'platform target' can be changed from the Visual Studio project option build tab (to "x86" in this case, to make it work).

I guess I should not have been surprised, the netCDF DLLs were found in this file, after all:
netcdf-4.1.1-win32-bin.zip [;)].

The first library i mentioned, the HDF5 one, is more up to date and there are prebuilt Windows binaries for win32 and win64 alike.

I think maybe the plan is to hard code a method that can save a float[,]-array using just the HDF5 binaries (hdf5dll.dll). I will let you know when I get a workable example (which is not necessarily this year).

Best regards and merry christmas to all,
Robert Nielsen

By on 12/22/2010 2:10 PM ()

Hi

I have pasted some code to here: HDF5_185

There are a few caveats to the code - and a fair bunch of copy-paste errors, omissions and typos no doubt, but if you dare I think it might be manageable as is.

You need to download the HDF library DLLs and place them where it says in the code, or you may update the quoted paths in the code instead.

I will post some more information at a later time I hope; but don't really have the time this month. Questions are very welcome though (there is always that slight theoretical possibility that maybe my code does not all end up self-explanatory).
[;)]

Usage is e.g. like this:

1
2
3
4

open HDF5_185
printfn "HDF5 library version is %A" (LibraryVersion())
let matrixdata = Array2D.init 3 2 (fun i j -> (float i) + 0.1 * (float j))
SaveMatrixToHDFFile "MyFile.H5" "MyMatrix" matrixdata

Best regards
Robert

By on 1/9/2011 4:19 PM ()

Hello

I have pasted an update to here: HDF5_188

Best regards
Robert

By on 10/19/2012 11:09 AM ()

Nice - thanks!

By on 10/22/2012 4:05 AM ()

Hi Adam and others

I am pleased if you find the script useful.

Unfortunately there was a serious error in my version 1.8.8 file. The calling conventions used in the newer versions of the 32 bit binary HDF5 libraries were changed (it is no longer StdCall). I have fixed this error in my import script. There is (most likely) a bunch of bugs left. Let us hope the remaining bugs are not as severe. The new version is uploaded as an fs and an fsi (signature) file to here:
HDF5-189.fsi
HDF5-189.fs

I added a function (GroupGetHierarchyInformation) to easier read the structure of a simple HDF5 file. Why this sudden interest for the structure of data in a time of F# 3.0 you ask? Well, I will let you figure that one out for yourselves :-) .

Best regards
Robert

By on 11/24/2012 1:17 PM ()

Hi

I have posted a blog entry with some example files that invoke the "GroupGetHierarchyInformation" function in an F# Type Provider context.

The blog entry is a Wordpress post.

Best regards
Robert

By on 4/7/2013 7:24 AM ()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

module HDF5Write
  open System
  open System.Text
  open System.Runtime.InteropServices
  
  type double1d = System.Double[]
  type double2d = System.Double[,]
  type double3d = System.Double[,,]
  // type intbyref = byref<System.Int32> // is not allowed in the compiled code
  #nowarn "51" // no warnings for native pointer use (they are stand ins for byref)
  
  let hdf5const = 0x1000
  let doubletypeid = 6
  
  [< DllImport("netcdf.dll", EntryPoint="nc_create", CharSet=CharSet.Ansi, SetLastError=true, ExactSpelling=true) >]
  extern int nc_create([< MarshalAs(UnmanagedType.LPStr) >] StringBuilder path, int cmode, [<Out>] int* ncidp)

  [< DllImport("netcdf.dll", EntryPoint="nc_def_dim", CharSet=CharSet.Ansi, SetLastError=true, ExactSpelling=true) >]
  extern int nc_def_dim(int ncid, [< MarshalAs(UnmanagedType.LPStr) >] StringBuilder name, int len, [<Out>] int* idp)

  [< DllImport("netcdf.dll", EntryPoint="nc_def_var", CharSet=CharSet.Ansi, SetLastError=true, ExactSpelling=true) >]
  extern int nc_def_var(int ncid, [< MarshalAs(UnmanagedType.LPStr) >] StringBuilder name, int nctype, int ndims, [<In>] int[] dimids, [<Out>] int* varid)

  [< DllImport("netcdf.dll", EntryPoint="nc_enddef", CharSet=CharSet.Ansi, SetLastError=true, ExactSpelling=true) >]
  extern int nc_enddef(int ncid)

  [< DllImport("netcdf.dll", EntryPoint="nc_put_var_double", CharSet=CharSet.Ansi, SetLastError=true, ExactSpelling=true) >]
  extern int nc_put_var_double1d(int ncid, int varid, [<In>] double1d data)

  [< DllImport("netcdf.dll", EntryPoint="nc_put_var_double", CharSet=CharSet.Ansi, SetLastError=true, ExactSpelling=true) >]
  extern int nc_put_var_double2d(int ncid, int varid, [<In>] double2d data)

  [< DllImport("netcdf.dll", EntryPoint="nc_put_var_double", CharSet=CharSet.Ansi, SetLastError=true, ExactSpelling=true) >]
  extern int nc_put_var_double3d(int ncid, int varid, [<In>] double3d data)

  [< DllImport("netcdf.dll", EntryPoint="nc_close", CharSet=CharSet.Ansi, SetLastError=true, ExactSpelling=true) >]
  extern int nc_close(int ncid)
  
  let checkerror (s: string) (result: int) =
      if result = 0
       then ()
       else failwith <| sprintf "Failed '%s' with errorcode %d." s result
  
  let HDFCreate (path: string) =
      let mutable id = 0
      nc_create (new StringBuilder(path), hdf5const, &&id)
       |> checkerror "nc_create"
      id
  
  let HDFDefineDimension fileid (dimensionname: string) (dimensionlength: int) =
      let mutable id = 0
      nc_def_dim(fileid, new StringBuilder(dimensionname), dimensionlength, &&id)
       |> checkerror "nc_def_dim"
      id
 
  let HDFDefineDataset fileid (datasetname: string) (dimensionIds: int list) =
      let mutable id = 0
      let dimensions = List.toArray dimensionIds
      nc_def_var(fileid, new StringBuilder(datasetname), doubletypeid, dimensions.Length, dimensions, &&id)
       |> checkerror "nc_def_var"
      id
  
  let HDFEndDefinitions fileid =
      nc_enddef fileid |> checkerror "nc_enddef"
 
  let HDFWriteVector fileid datasetid data =
      nc_put_var_double1d (fileid, datasetid, data)
       |> checkerror "nc_put_var_double1d"
  
  let HDFWriteMatrix fileid datasetid data =
      nc_put_var_double2d (fileid, datasetid, data)
       |> checkerror "nc_put_var_double2d"
  
  let HDFWriteArray3d fileid datasetid data =
      nc_put_var_double3d (fileid, datasetid, data)
       |> checkerror "nc_put_var_double3d"
 
  let HDFClose fileid =
      nc_close fileid |> checkerror "nc_close"
  
  let HDFSaveMatrixToFile (filename: string) (matrixname: string) (data: double[,]) =
      let fileid = HDFCreate filename
      let xdimid = HDFDefineDimension fileid "x_dimension_non_data" (data.GetLength 0)
      let ydimid = HDFDefineDimension fileid "y_dimension_non_data" (data.GetLength 1)
      let dataid = HDFDefineDataset fileid matrixname [xdimid; ydimid]
      do HDFEndDefinitions fileid
      do HDFWriteMatrix fileid dataid data
      do HDFClose fileid
By on 12/5/2010 5:57 AM ()

Thanks for this Robert! I'll try it out when I get the chance.

By on 12/13/2010 4:50 PM ()

BTW, this isn't necessarily the right way, but this should create your CSV for for a simple array, assuing one entry per line:

1
2
3
4

[| 1; 2; 3 |]
       |> Seq.fold (fun a e -> a + e.ToString() + "\r\n") "" 
       |> (fun txt -> System.IO.File.WriteAllText(@"c:\temp\out.txt", txt))

File:
1
2
3

By on 12/1/2010 12:02 PM ()

BTW, this isn't necessarily the right way, but this should create your CSV for for a simple array, assuing one entry per line

Thanks!

But are there no libraries available? I guess I could read the spec on CSV and code by hand as you did. It does seem curious for computer programs to be communicating with each other via human-readable text files!

By on 12/1/2010 3:38 PM ()

> But are there no libraries available?

I'm sure there are score of them - if you Google "C# csv writer" or something I'm sure there will be matches. As you can use C# and VB.net classes from F#, they should work - however, they may not handle F# primitive types correctly, etc.

> I guess I could read the spec on CSV and code by hand as you did.

There's a spec? :) AFAIK, a "comma seperated file" is just that - there aren't too many variations - header line or not, how quotations are handled, etc.

>It does seem curious for computer programs to be communicating with each other via human-readable text files!

Not really - many advantages for authoring/editing, debugging, cross-platform, etc., hence XML, JSON, etc.

By on 12/1/2010 7:36 PM ()

There's a spec? :)  AFAIK, a "comma seperated file" is just that - there aren't too many variations - header line or not, how quotations are handled, etc.

Well, reading the wikipedia article, there's not just one spec, but many. And sometimes it's not comma separated, but semi-colon separated. And numbers are represented as strings, but in an inconsistent way between countries/languages. And it can only cope with 2d data structures.

I will probably just convert to Mathematica's list notation, as that's direct, simpler, more consistent, more powerful, and the disadvantage of converting floats to strings and back is also present with CSV.

By on 12/2/2010 12:36 PM ()

The books and libraries here should have what you are looking for. I don't work for the FFC, but can recommend F# for Technical Comuting & (and the older F# for Scientists on Amazon), and the F# for Visualization should do what you want w/o export to m'm'ica. (F# for numerics may also do other things you need.) I beleive import/export is covered somewhere in there.

[link:www.ffconsultancy.com]

By on 12/1/2010 11:46 AM ()

F# for Technical Comuting & (and the older F# for Scientists on Amazon)

I have the older book; it was a very good book but doesn't contain anything useful on graphing or export. Maybe the newer one has relevant material?

F# for Visualization should do what you want w/o export to m'm'ica.

It must be quite hard to write a proper graphing library or there would be more around. If I saw enough to make me think you could do good 3D surface plots of various types, exportable as vector graphics with publishable quality, then I'd definitely look at buying it. It's not clear from their site whether it can do all this.

(F# for numerics may also do other things you need.)

Here standards are important, and unless this becomes the de-facto standard for technical computing in F#/.NET I will be wary. At the moment maybe Math.Net seems most likely to win? At any rate I hope a dominant mathematical library emerges soon.

By on 12/1/2010 3:31 PM ()

    Topic tags

    IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
    Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy     		Built with WebSharper