Back to Questions
1.7k
views6
comments
Sizes of .FS files and general capacity of the F# environment?
f#compiler

Hello,

I'm developing a compiler system that needs access to a lot of metadata to do its work correctly. My first idea was to implement the metadata as F# data structures, to compile them into the compiler, so to speak. However, I find that the F# compiler cannot compile files of any significant size! Both the VS compiler and the standalone compiler break when trying to compile a file of 1Mb. This may seem like a large file, and I would certainly hate having to input it at the keyboard, but these are generated files, untouched by humans. I have never seen the C# compiler break in the face of large files or projects containing many files.

There are alternatives to doing what I want to do: I can load the metadata from files when needed, or build a database and put the data there etc.

I am starting to wonder if F# is useful for production work, handling lots of data?

BR PVillads

.

Hi,

I've had a couple of problems in this area. First was the F# compiler crashing out when compiling a really big, but otherwise quite simple, function. This got fixed about a week after I sent the bug report.

The next issue wasn't F# related. I was auto-generating a C# class (by directly writing C# code, not via Code DOM) with one particularly huge method. It compiled fine but when the byte code for that method reached about 650Kbytes, as shown by ildasm, the .NET runtime would just give up and throw an InvalidProgramException. In fact ildasm was the only thing that could read the function. No one MS over at the MSDN forums seemed interested getting a bug report so the problem still exists as far as I know.

On the other hand I have auto-generated a 4 Mb, 75K line F# file which contains 1000's of functions and, even though it takes a while to compile, it does get there and works fine.

The upshot of all this is I would recommend breaking your generated code up a bit into subfunctions (or perhaps methods) and tie it all together at the end. My impression is that .NET can get itself a bit stressed out at times.

Cheers,

Andy

By on 1/22/2007 3:33 PM ()

I think it's not too surprising that if you throw 1MB of generated code at a compiler that it might break. The algorithms used during compilation and optimization are non-trivial, and different kinds of stress tests will expose different kinds of problems. That said, just send us a bug report with your generated file and we'll almost certainly fix the problem. It will make a good stress test!

FWIW we have many users using F# to manipulate terabytes of data. We also have users (including ourselves) regularly using the F# Visual Studio mode in multi-project settings with ~100KLOC of user-written code. User files tend not to be > 10K lines. We've had one previous report of the compiler falling over on a 300Kbyte generated file, but we fixed the problem.

Are you compiling on Linux or Windows?

Thanks,

don

By on 1/21/2007 10:55 AM ()

I'm compiling under Windows. I've created a small repro scenario containing only what is needed to demonstrate the 2 problems:

1) The compiler cannot handle the compilation of the kernelclasses.fs file.

2) The VS integration fails (VS crashes) when trying to open the kernelclasses.fs file.

You may use either the makefile to do the compilation:

nmake

or use the supplied Visual Studio environment.

Note, that the contents of the big file is commented out. There is (nearly) no content for the compiler to work on, and still it complains.

I'm not sure how you actually go about posting an attachment, so I've placed the zip file on

[link:villadsen.dk]

By on 1/23/2007 2:15 PM ()

Thanks - we're looking into this.

don

By on 1/23/2007 8:10 PM ()

OK, there's a couple of things here. Firstly, we've fixed the bug that was causing fsc.exe to fail to lex/parse the 2MB comment - the fix will be in the next external release.

Now, if your comment markers are removed then the resulting program compiles to .NET code in ~16s (no doubt that could be made faster - we'll use this as a stress/perf test) However, the generated .NET code does indeed run into the problem mentioned above, i.e. that .NET won't run programs with individual methods bigger than ~650K bytecode instructions. We can fix this for F# by splitting method bodies (and we sometimes do that). We will also give a warning in the code generator. In the meantime here's what I suggest:

  • I think it's easy for you to change the code you generate so you don't have a single static initialization of an enormous data term. You should use let-bindings to name subterms in your graph. These can be functions taking unit arguments. For example, your top list is very long, and you can easily divide its construction into a series of functions, each with say ~100 elements, and stitch the list together by calling these functions.
  • For enormous large object terms you should always consider generating a different format and running an interpreter. Like all .NET languages F# creates real code to initialize static structured data. You can save a lot of work if you just embed your static data as a resource in your assembly and run a small interpreter to recreate that data at runtime.

I understand the appeal of just generating an enormous static term, but these are the techniques I would use.

Cheers

don

By on 1/23/2007 8:46 PM ()

Hi again,

Just to let you know that we've adjusted the compiler to ensure the code can compile and run. The cumulative fixes will be in the next release.

However, because the static constant terms is still being compiled as code I strongly recommend you just use a simple technique such as "output_value"/"input_value" (i.e. .NET serialization) or one of the other techniques mentioned above to manage terms of this size. Representing enoromous static constant data terms as F# programs is not recommended because other quite reasonable techniques are availble.

Cheers

don

By on 1/25/2007 7:51 AM ()

    Topic tags

    IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
    Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy     		Built with WebSharper