[:D]
I've revised the code in-line with how fslex works. Fine.
[:$]However the program only works if we take input from a file specifed on the program command-line. If we use stdin,we get an infinite loop as the code calls through to the default case for unrecognised input.
Do we have some form of bytes/text confusion going on in the input stream [8-)]?
New code :-
{
open Printf
let create_hashtable size init =
let tbl = Hashtbl.create size in
List.iter (fun (key, data) -> Hashtbl.add tbl key data) init;
tbl
type token =
| IF
| THEN
| ELSE
| BEGIN
| END
| FUNCTION
| ID of string
| OP of char
| INT of int
| FLOAT of float
| CHAR of char
let keyword_table =
create_hashtable 8 [
("if", IF);
("then", THEN);
("else", ELSE);
("begin", BEGIN);
("end", END);
("function", FUNCTION)
]
}
let digit = ['0'-'9']
let inum = digit+
let fnum = digit+ '.' digit*
let id = ['a'-'z' 'A'-'Z']['a'-'z' '0'-'9']*
let word = id
let op = '+' | '-' | '*' | '/'
rule toy_lang = parse
| inum
{ let num = int_of_string ( Lexing.lexeme lexbuf ) in
printf "integer: %s (%d)\n" ( Lexing.lexeme lexbuf ) num;
INT num
}
| fnum
{ let num = float_of_string ( Lexing.lexeme lexbuf ) in
printf "float: %s (%f)\n" ( Lexing.lexeme lexbuf ) num;
FLOAT num
}
| id
{ try
let token = Hashtbl.find keyword_table ( Lexing.lexeme lexbuf ) in
printf "keyword: %s\n" ( Lexing.lexeme lexbuf );
token
with Not_found ->
printf "identifier: %s\n" ( Lexing.lexeme lexbuf );
ID ( Lexing.lexeme lexbuf )
}
| op
{ printf "operator: %c\n" ( Lexing.lexeme_char lexbuf 1 );
OP ( Lexing.lexeme_char lexbuf 1 )
}
| '{' [^ '\n']* '}' (* eat up one-line comments *)
| [' ' '\t' '\n'] (* eat up whitespace *)
{ toy_lang lexbuf }
| _
{ printf "Unrecognized character: %c\n" ( Lexing.lexeme_char lexbuf 0 ) ;
CHAR ( Lexing.lexeme_char lexbuf 0 )
}
| eof
{ raise End_of_file }
{
let rec parse lexbuf =
let token = toy_lang lexbuf in
(* do nothing in this example *)
parse lexbuf
let main () =
let cin =
if Array.length Sys.argv > 1
then open_in Sys.argv.(1)
else stdin
in
let lexbuf = Lexing.from_channel cin in
try parse lexbuf
with End_of_file -> ()
let _ = Printexc.print main ()
}
Hi Mark,
You may well want Lexing.from_text_reader for stdin, though it's hard to be sure - it depends what encoding the text is coming in as and what encoding you want to use to present the bytes to the lexer. If it's straight-up ASCII input it shouldn't make much difference.
What characters are being printed?
don
Don,
The lexer works fine until it runs out of input. It then just seems to get stuck in a loop, and spits out the null character for ever. I know that this is a trivial problem, given the right level of knowledge, but right now, my knowledge of .net and f# is around zero percent.
Regards, Mark
eg :-
12 + 12
12
12
3/2/2/2/2/1
yields :-
integer: 12 (12)
operator: +
integer: 12 (12)
integer: 12 (12)
integer: 12 (12)
integer: 3 (3)
operator: /
integer: 2 (2)
operator: /
integer: 2 (2)
operator: /
integer: 2 (2)
operator: /
integer: 2 (2)
operator: /
integer: 1 (1)
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
On a quick look, in the F# lexer we use
| _ { unexpected_char lexbuf }
| eof { EOF }
i.e. return an explicit token in the eof condition. That may be worth trying here rather than raising an exception.
Kind regards
don
OK, I've sorted this one, just needed to replace the aliases with separate let definitions.
A bit of a bind though..........
Topic tags
- f# × 3705
- websharper × 1897
- compiler × 286
- functional × 201
- ui next × 139
- c# × 121
- classes × 97
- web × 97
- .net × 84
- book × 84
- async × 76
- ui.next × 67
- bug × 54
- core × 49
- website × 49
- server × 45
- parallel × 43
- ui × 43
- enhancement × 41
- parsing × 41
- testing × 41
- trywebsharper × 41
- typescript × 37
- html × 35
- javascript × 35
- owin × 35
- asynchronous × 30
- monad × 28
- ocaml × 28
- tutorial × 27
- warp × 27
- haskell × 26
- sitelet × 25
- linq × 22
- workflows × 22
- wpf × 20
- fpish × 19
- introduction × 19
- silverlight × 19
- sitelets × 19
- monodevelop × 17
- rpc × 17
- suave × 17
- piglets × 16
- collections × 15
- feature request × 15
- jquery × 15
- templates × 15
- getting started × 14
- pipeline × 14
- kendoui × 13
- reactive × 12
- 4.1.0.171 × 11
- monads × 11
- opinion × 10
- 4.0.190.100-rc × 9
- deployment × 9
- fixed × 9
- formlets × 9
- in × 9
- json × 9
- plugin × 9
- proposal × 9
- scheme × 9
- solid × 9
- basics × 8
- concurrent × 8
- highcharts × 8
- how-to × 8
- python × 8
- 4.1.1.175 × 7
- complexity × 7
- documentation × 7
- visual studio × 7
- 4.1.2.178 × 6
- lisp × 6
- real-world × 6
- released in 4.0.192.103-rc × 6
- remoting × 6
- resources × 6
- scala × 6
- websharper ui.next × 6
- workshop × 6
- xaml × 6
- 4.0.193.110 × 5
- 4.2.3.236 × 5
- aspnetmvc × 5
- authentication × 5
- azure × 5
- bootstrap × 5
- conference × 5
- dsl × 5
- formlet × 5
- java × 5
- list × 5
- metaprogramming × 5
- ml × 5
- released in Zafir.4.0.188.91-beta10 × 5
- sql × 5
- visualstudio × 5
- websharper.forms × 5
- zafir × 5
- 4.0.192.106 × 4
- 4.0.195.127 × 4
- 4.1.0.38 × 4
- 4.2.1.86 × 4
- 4.2.6.118 × 4
- css × 4
- example × 4
- extensions × 4
- fsi × 4
- fsx × 4
- html5 × 4
- jqueryui × 4
- lift × 4
- reflection × 4
- remote × 4
- rest × 4
- spa × 4
- teaching × 4
- template × 4
- websocket × 4
- wontfix × 4
- 4.0.196.147 × 3
- 4.1.0.34 × 3
- 4.1.6.207 × 3
- 4.2.1.223-beta × 3
- 4.2.11.258 × 3
- 4.2.4.114 × 3
- 4.2.4.247 × 3
- 4.2.5.115 × 3
- 4.2.6.253 × 3
- 4.2.9.256 × 3
- ajax × 3
- alt.net × 3
- aml × 3
- asp.net mvc × 3
- canvas × 3
- cloudsharper × 3
- compilation × 3
- database × 3
- erlang × 3
- events × 3
- extension × 3
- file upload × 3
- forums × 3
- inline × 3
- issue × 3
- kendo × 3
- macro × 3
- mono × 3
- msbuild × 3
- mvc × 3
- pattern × 3
- piglet × 3
- released in Zafir.4.0.187.90-beta10 × 3
- svg × 3
- type provider × 3
- view × 3
- 4.1.1.64 × 2
- 4.1.5.203 × 2
- 4.1.7.232 × 2
- 4.2.10.257 × 2
- 4.2.3.111 × 2
- 4.2.5.249 × 2
- android × 2
- asp.net × 2
- beginner × 2
- blog × 2
- chart × 2
- client × 2
- client server app × 2
- clojure × 2
- computation expressions × 2
- constructor × 2
- corporate × 2
- courses × 2
- cufp × 2
- d3 × 2
- debugging × 2
- direct × 2
- discriminated union × 2
- docs × 2
- elm × 2
- endpoint × 2
- endpoints × 2
- enterprise × 2
- entity framework × 2
- event × 2
- f# interactive × 2
- fable × 2
- flowlet × 2
- formdata × 2
- forms × 2
- fsc × 2
- google maps × 2
- hosting × 2
- http × 2
- https × 2
- iis 8.0 × 2
- install × 2
- interactive × 2
- interface × 2
- iphone × 2
- iteratee × 2
- jobs × 2
- jquery mobile × 2
- keynote × 2
- lens × 2
- lenses × 2
- linux × 2
- listmodel × 2
- mac × 2
- numeric × 2
- oauth × 2
- obfuscation × 2
- offline × 2
- oop × 2
- osx × 2
- packaging × 2
- pattern matching × 2
- performance × 2
- pipelines × 2
- q&a × 2
- quotation × 2
- reference × 2
- released in Zafir.4.0.185.88-beta10 × 2
- rx × 2
- script × 2
- security × 2
- self host × 2
- seq × 2
- sockets × 2
- stm × 2
- tcp × 2
- trie × 2
- tutorials × 2
- type × 2
- url × 2
- var × 2
- websharper.charting × 2
- websharper4 × 2
- websockets × 2
- wig × 2
- xna × 2
- zh × 2
- .net interop × 1
- 2012 × 1
- 4.0.194.126 × 1
- 4.1.3.184 × 1
- 4.1.4.189 × 1
- 4.2.0.214-beta × 1
- 4.2.12.259 × 1
- 4.2.2.231-beta × 1
- 4.2.8.255 × 1
- Canvas Sample Example × 1
- DynamicStyle Animated Style × 1
- Fixed in 4.0.190.100-rc × 1
- Released in Zafir.UI.Next.4.0.169.79-beta10 × 1
- SvgDynamicAttribute × 1
- WebComponent × 1
- abstract class × 1
- accumulator × 1
- active pattern × 1
- actor × 1
- addin × 1
- agents × 1
- aggregation × 1
- agile × 1
- alter session × 1
- animation × 1
- anonymous object × 1
- apache × 1
- api × 1
- appcelerator × 1
- architecture × 1
- array × 1
- arrays × 1
- asp.net 4.5 × 1
- asp.net core × 1
- asp.net integration × 1
- asp.net mvc 4 × 1
- asp.net web api × 1
- aspnet × 1
- ast × 1
- attributes × 1
- authorization × 1
- b-tree × 1
- back button × 1
- badimageformatexception × 1
- bash script × 1
- batching × 1
- binding-vars × 1
- bistro × 1
- body × 1
- bundle × 1
- camtasia studio × 1
- cas protocol × 1
- charts × 1
- clarity × 1
- class × 1
- cli × 1
- clipboard × 1
- clojurescript × 1
- closures × 1
- cloud × 1
- cms × 1
- coding diacritics × 1
- color highlighting × 1
- color zones × 1
- combinator × 1
- combinators × 1
- compile × 1
- compile code on server × 1
- config × 1
- confirm × 1
- content × 1
- context × 1
- context.usersession × 1
- continuation-passing style × 1
- coords × 1
- cordova × 1
- cors × 1
- coursera × 1
- cross-domain × 1
- csla × 1
- current_schema × 1
- custom content × 1
- data × 1
- data grid × 1
- datetime × 1
- debug × 1
- declarative × 1
- delete × 1
- devexpress × 1
- dhtmlx × 1
- dictionary × 1
- directattribute × 1
- disqus × 1
- distance × 1
- do binding × 1
- doc elt ui.next upgrade × 1
- docker × 1
- dojo × 1
- dol × 1
- dom × 1
- domain × 1
- du × 1
- duf-101 × 1
- dynamic × 1
- eastern language × 1
- eclipse × 1
- edsl × 1
- em algorithm × 1
- emacs × 1
- emotion × 1
- enums × 1
- error × 1
- etw × 1
- euclidean × 1
- eventhandlerlist × 1
- examples × 1
- ext js × 1
- extension methods × 1
- extra × 1
- facet pattern × 1
- failed to translate × 1
- fake × 1
- fantomas × 1
- fear × 1
- float × 1
- form × 1
- form-data × 1
- forum × 1
- fp × 1
- frank × 1
- fsdoc × 1
- fsharp × 1
- fsharp.core × 1
- fsharp.powerpack × 1
- fsharpx × 1
- fsunit × 1
- function × 1
- functional style × 1
- game × 1
- games × 1
- gc × 1
- generic × 1
- geometry × 1
- getlastwin32error × 1
- getting-started × 1
- google × 1
- google.maps × 1
- grid × 1
- group × 1
- guide × 1
- hash × 1
- headers × 1
- hello world example × 1
- heroku × 1
- highchart × 1
- history × 1
- how to × 1
- html-templating × 1
- http405 × 1
- httpcontext × 1
- hubfs × 1
- i18n × 1
- ie 8 × 1
- if-doc × 1
- iis × 1
- image × 1
- images × 1
- inheritance × 1
- initialize × 1
- input × 1
- install "visual studio" × 1
- installer × 1
- int64 × 1
- interfaces × 1
- internet explorer × 1
- interop × 1
- interpreter × 1
- io × 1
- iobservable × 1
- ios × 1
- iot × 1
- ipad × 1
- isomorphic × 1
- javascript optimization × 1
- javascript semanticui resources × 1
- jquery-plugin × 1
- jquery-ui × 1
- jquery-ui-datepicker × 1
- js × 1
- kendo datasource × 1
- kendochart × 1
- kendoui compiler × 1
- knockout × 1
- l10n × 1
- learning × 1
- library × 1
- libs × 1
- license × 1
- licensing × 1
- lineserieszonescfg × 1
- local setting × 1
- localization × 1
- logging × 1
- loop × 1
- macros × 1
- mailboxprocessor × 1
- mapping × 1
- maps × 1
- markerclusterer × 1
- markup × 1
- marshal × 1
- math × 1
- mathjax × 1
- message × 1
- message passing × 1
- message-passing × 1
- meta × 1
- metro style × 1
- micro orm × 1
- minimum-requirements × 1
- mix × 1
- mobile installation × 1
- mod_mono × 1
- modal × 1
- module × 1
- mouseevent × 1
- mouseposition × 1
- multidimensional × 1
- multiline × 1
- multithreading × 1
- mysql × 1
- mysqlclient × 1
- nancy × 1
- native × 1
- nested × 1
- nested loops × 1
- node × 1
- nunit × 1
- object relation mapper × 1
- object-oriented × 1
- om × 1
- onboarding × 1
- onclick × 1
- optimization × 1
- option × 1
- orm × 1
- os x × 1
- output-path × 1
- override × 1
- paper × 1
- parameter × 1
- persistence × 1
- persistent data structure × 1
- phonegap × 1
- pola × 1
- post × 1
- powerpack × 1
- prefix tree × 1
- principle of least authority × 1
- privacy × 1
- private × 1
- profile × 1
- programming × 1
- project × 1
- project euler × 1
- projekt_feladat × 1
- protected × 1
- provider × 1
- proxy × 1
- ptvs × 1
- public × 1
- pure f# × 1
- purescript × 1
- qna × 1
- quant × 1
- query sitelet × 1
- question × 1
- quotations × 1
- range × 1
- raphael × 1
- razor × 1
- rc × 1
- reactjs × 1
- real-time × 1
- ref × 1
- region × 1
- released in 4.0.190.100-rc × 1
- reporting × 1
- responsive design × 1
- rest api × 1
- rest sitelet × 1
- restful × 1
- round table × 1
- router × 1
- routing × 1
- rpc reverseproxy × 1
- runtime × 1
- sales × 1
- sample × 1
- sampleapp × 1
- scriptcs × 1
- scripting × 1
- search × 1
- self hosted × 1
- semanticui × 1
- sequence × 1
- serialisation × 1
- service × 1
- session-state × 1
- sharepoint × 1
- signals × 1
- sitelet website × 1
- sitelet.protect × 1
- sitlets × 1
- slickgrid × 1
- source code × 1
- sqlentityconnection × 1
- ssl × 1
- standards × 1
- static content × 1
- stickynotes × 1
- streamreader × 1
- stress × 1
- strong name × 1
- structures × 1
- submitbutton × 1
- subscribe × 1
- svg example html5 websharper.ui.next × 1
- sweetalert × 1
- system.datetime × 1
- system.reflection.targetinvocationexception × 1
- table storage × 1
- targets × 1
- tdd × 1
- templates ui.next × 1
- templating × 1
- text parsing × 1
- three.js × 1
- time travel × 1
- tls × 1
- tooltip × 1
- tracing × 1
- tsunamiide × 1
- turkish × 1
- twitter-bootstrap × 1
- type erasure × 1
- type inference × 1
- type providers × 1
- type-providers × 1
- typeprovider × 1
- ui next forms × 1
- ui-next × 1
- ui.next jqueryui × 1
- ui.next charting × 1
- ui.next formlets × 1
- ui.next forms × 1
- ui.next suave visualstudio × 1
- ui.next templating × 1
- unicode × 1
- unittest client × 1
- upload × 1
- usersession × 1
- validation × 1
- vb × 1
- vb.net × 1
- vector × 1
- view.map × 1
- visal studio × 1
- visual f# × 1
- visual studio 11 × 1
- visual studio 2012 × 1
- visual studio shell × 1
- vs2017 compiler zafir × 1
- vsix × 1
- web api × 1
- web-scraping × 1
- webapi × 1
- webcomponents × 1
- webforms × 1
- webgl × 1
- webrtc × 1
- webshaper × 1
- websharper async × 1
- websharper codemirror × 1
- websharper f# google × 1
- websharper forms × 1
- websharper reactive × 1
- websharper rpc × 1
- websharper sitelets routing × 1
- websharper warp × 1
- websharper-interface-generator × 1
- websharper.chartsjs × 1
- websharper.com × 1
- websharper.exe × 1
- websharper.owin × 1
- websharper.ui.next × 1
- websharper.ui.next jquery × 1
- websockets iis × 1
- why-websharper × 1
- windows 7 × 1
- windows 8 × 1
- windows-phone × 1
- winrt × 1
- www.grabbitmedia.com × 1
- xamarin × 1
- xml × 1
- yeoman × 1
- yield × 1
- zafir beta × 1
- zafir websharper4 × 1
- zarovizsga × 1
![]() |
Copyright (c) 2011-2012 IntelliFactory. All rights reserved. Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy |
Built with WebSharper |
Hi, I've tried building the 'toy' language lexer by SooHYoung Ho, but it fails with an error which suggests fslex isn't just a plug-in replacement of ocamllex. Any help would be appreciated as I can find very little detail on fslex itself on the web.
I get the following error :-
..\..\..\bin\fslex toy.fsl
compiling to dfas (can take a while...)
Error: The macro inum is not defined
The code for toy.fsl is below :-
-------------
{
open Printf
let create_hashtable size init =
let tbl = Hashtbl.create size in
List.iter (fun (key, data) -> Hashtbl.add tbl key data) init;
tbl
type token =
| IF
| THEN
| ELSE
| BEGIN
| END
| FUNCTION
| ID of string
| OP of char
| INT of int
| FLOAT of float
| CHAR of char
let keyword_table =
create_hashtable 8 [
("if", IF);
("then", THEN);
("else", ELSE);
("begin", BEGIN);
("end", END);
("function", FUNCTION)
]
}
let digit = ['0'-'9']
let id = ['a'-'z' 'A'-'Z']['a'-'z' '0'-'9']*
rule toy_lang = parse
| digit+ as inum
{ let num = int_of_string inum in
printf "integer: %s (%d)\n" inum num;
INT num
}
| digit+ '.' digit* as fnum
{ let num = float_of_string fnum in
printf "float: %s (%f)\n" fnum num;
FLOAT num
}
| id as word
{ try
let token = Hashtbl.find keyword_table word in
printf "keyword: %s\n" word;
token
with Not_found ->
printf "identifier: %s\n" word;
ID word
}
| '+'
| '-'
| '*'
| '/' as op
{ printf "operator: %c\n" op;
OP op
}
| '{' [^ '\n']* '}' (* eat up one-line comments *)
| [' ' '\t' '\n'] (* eat up whitespace *)
{ toy_lang lexbuf }
| _ as c
{ printf "Unrecognized character: %c\n" c;
CHAR c
}
| eof
{ raise End_of_file }
{
let rec parse lexbuf =
let token = toy_lang lexbuf in
(* do nothing in this example *)
parse lexbuf
let main () =
let cin =
if Array.length Sys.argv > 1
then open_in Sys.argv.(1)
else stdin
in
let lexbuf = Lexing.from_channel cin in
try parse lexbuf
with End_of_file -> ()
let _ = Printexc.print main ()
}