rs274ngc bison/flex parser {was: question on gcode parsing}

Discussion:

Michael Haberler

2012-01-31 23:10:04 UTC

[I guess this better belongs here]

ok, while this wonderful discussion was raging on, I built a working parser for the current linuxcnc dialect, as an experiment in feasability (this is NOT an end-user tool!)

http://git.mah.priv.at/gitweb/emc2-dev.git/shortlog/refs/heads/parser-v2-dev

- Michael

ps: I'd appreciate feedback from language geeks wrt to removing shift/reduce conflicts.

-------------- src/emc/parser-v2/README: --------

RS274NGC parser based on Bison/Flex
===================================

usage:

parser [options] ngcfile ...

example:
$ parser emc2-dev/nc_files/*.ngc

no output means good news (parse ok)
other that that, no actions - this is an experiment to find out wether this is feasible

options are:
-v: create listing with interspersed error messages
-s: trace scanning
-p: trace parsing

I'd be grateful for hints how to remove the remaining shift/reduce
conflicts in the grammar. I guess there's something wrong/missing with
operator precedence and/or associativity.

Error messages
--------------
There are no custom error messages yet. All error messages are
autogenerated from the parser, and derived from the grammar, including
the set of permitted tokens, which is already quite reasonable:

$ parser -v emc2-dev/src/emc/parser-v2/ngc/foo.txt

...
#<bar> = [EXISTS[23
-----------------^^
ngc/foo.txt:8:unexpected integer, expecting #
...
m73.1
---^^
ngc/foo.txt:18:unexpected real, expecting one of: <<EOF>> EOL ^ @ % if while break do repeat sub endsub return call # G-code M-code A B C D E F H I J K L P Q R S T U V W X Y Z
...
N3242 G10 x[1

^
ngc/foo.txt:24:unexpected EOL, expecting one of: ] EQ NE GT GE LT LE and or xor + - * / mod pow
...
N3242 G10 x[1]]
--------------^
ngc/foo.txt:25:unexpected ], expecting one of: <<EOF>> EOL ^ @ % if while break do repeat sub endsub return call # G-code M-code A B C D E F H I J K L P Q R S T U V W X Y Z
..
4 error(s)

Error recovery:
---------------

currently, on syntax error the parser skips to the next of line, and
continues, which seems to work ok.

Build notes
-----------
All files needed to compile are in this branch
If you change the grammar: bison 2.5 is needed (Ubuntu 10.04 has 2.4.1 which is too old)
I compiled from source - found no backport.
Flex: the stock flex coming with 10.04 is fine (flex 2.5.35)

Speed is quite good - without spending a thought on optimization and
all debug options on, this parser does about 300.000 lines/second on my MBP.

added Submakefile

base taken from https://idlebox.net/2007/flex-bison-cpp-example/
much of that example code is still to be deleted but then this is
a proof of concept

-mah
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Emc-users mailing list
Emc-users-5NWGOfrQmneRv+***@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/emc-users

Kent A. Reed

2012-02-01 04:47:35 UTC

Permalink

Post by Michael Haberler
[I guess this better belongs here]

I happen to agree with you, Michael, but Erik has expressed his dislike
of this "second" list (indeed, I don't see any messages from him in my
archive of this list) and he's our most likely contributor of wisdom on
this subject. For that reason, I replied instead to the duplicate of
this message that you posted to emc-users. My apology in advance if this
upsets your attempt to re-situate the thread.

Regards,
Kent

EBo

2012-02-01 09:58:50 UTC

Permalink

If it is of any interest I could send you an experimental parser I
wrote a decade ago that uses Sprit++ (a C++ template tool, part of
Boost.org now, which replaces bison and flex, and reads like EBNF). At
that time I was experimenting with runtime polymorphic parsers (which
would allow you to overload the language description and support
different interpretations of the rs274*...

EBo --

Post by Michael Haberler
[I guess this better belongs here]
ok, while this wonderful discussion was raging on, I built a working
parser for the current linuxcnc dialect, as an experiment in
feasability (this is NOT an end-user tool!)
http://git.mah.priv.at/gitweb/emc2-dev.git/shortlog/refs/heads/parser-v2-dev
- Michael
ps: I'd appreciate feedback from language geeks wrt to removing shift/reduce conflicts.
-------------- src/emc/parser-v2/README: --------
RS274NGC parser based on Bison/Flex
===================================
parser [options] ngcfile ...
$ parser emc2-dev/nc_files/*.ngc
no output means good news (parse ok)
other that that, no actions - this is an experiment to find out wether this is feasible
-v: create listing with interspersed error messages
-s: trace scanning
-p: trace parsing
I'd be grateful for hints how to remove the remaining shift/reduce
conflicts in the grammar. I guess there's something wrong/missing with
operator precedence and/or associativity.
Error messages
--------------
There are no custom error messages yet. All error messages are
autogenerated from the parser, and derived from the grammar,
including
$ parser -v emc2-dev/src/emc/parser-v2/ngc/foo.txt
...
#<bar> = [EXISTS[23
-----------------^^
ngc/foo.txt:8:unexpected integer, expecting #
...
m73.1
---^^
if while break do repeat sub endsub return call # G-code M-code A B C
D E F H I J K L P Q R S T U V W X Y Z
...
N3242 G10 x[1
^
ngc/foo.txt:24:unexpected EOL, expecting one of: ] EQ NE GT GE LT LE
and or xor + - * / mod pow
...
N3242 G10 x[1]]
--------------^
while break do repeat sub endsub return call # G-code M-code A B C D
E
F H I J K L P Q R S T U V W X Y Z
..
4 error(s)
---------------
currently, on syntax error the parser skips to the next of line, and
continues, which seems to work ok.
Build notes
-----------
All files needed to compile are in this branch
If you change the grammar: bison 2.5 is needed (Ubuntu 10.04 has 2.4.1 which is too old)
I compiled from source - found no backport.
Flex: the stock flex coming with 10.04 is fine (flex 2.5.35)
Speed is quite good - without spending a thought on optimization and
all debug options on, this parser does about 300.000 lines/second on my MBP.
added Submakefile
base taken from https://idlebox.net/2007/flex-bison-cpp-example/
much of that example code is still to be deleted but then this is
a proof of concept
-mah
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft
developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft
developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Emc-developers mailing list
https://lists.sourceforge.net/lists/listinfo/emc-developers

Michael Haberler

2012-02-01 11:16:59 UTC

Permalink

yes, please - I looked at Sprit++ and would be happy to shorten the learning curve.

-m
ps: I just love these concise C++ template compile time errors - how could I have ever lived without them ;-?

Post by EBo
If it is of any interest I could send you an experimental parser I
wrote a decade ago that uses Sprit++ (a C++ template tool, part of
Boost.org now, which replaces bison and flex, and reads like EBNF). At
that time I was experimenting with runtime polymorphic parsers (which
would allow you to overload the language description and support
different interpretations of the rs274*...
EBo --

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Emc-developers mailing list
https://lists.sourceforge.net/lists/listinfo/emc-developers

ebo@swcp.com

2012-02-05 09:10:13 UTC

Permalink

I'm in the middle of a move and will only have sporatic email access for the next week. Also, i do not have ready access to those files.

Could you give me a poke in a week or two?

EBo

Sent from my Android phone on T-Mobile. Americaâs first nationwide 4G network.

----- Reply message -----
From: "Michael Haberler" <***@mah.priv.at>
To: "EBo" <***@sandien.com>
Subject: [Emc-developers] rs274ngc bison/flex parser {was: question on gcode parsing}
Date: Sat, Feb 4, 2012 12:00 pm

Hi Ebo,

the more I look into it the more Spirit looks interesting to me. Would you mail me your example?

regards

-Michael

Post by Michael Haberler
[I guess this better belongs here]
ok, while this wonderful discussion was raging on, I built a working
parser for the current linuxcnc dialect, as an experiment in
feasability (this is NOT an end-user tool!)
http://git.mah.priv.at/gitweb/emc2-dev.git/shortlog/refs/heads/parser-v2-dev
- Michael
ps: I'd appreciate feedback from language geeks wrt to removing
shift/reduce conflicts.
-------------- src/emc/parser-v2/README: --------
RS274NGC parser based on Bison/Flex
===================================
parser [options] ngcfile ...
$ parser emc2-dev/nc_files/*.ngc
no output means good news (parse ok)
other that that, no actions - this is an experiment to find out
wether this is feasible
-v: create listing with interspersed error messages
-s: trace scanning
-p: trace parsing
I'd be grateful for hints how to remove the remaining shift/reduce
conflicts in the grammar. I guess there's something wrong/missing
with
operator precedence and/or associativity.
Error messages
--------------
There are no custom error messages yet. All error messages are
autogenerated from the parser, and derived from the grammar,
including
$ parser -v emc2-dev/src/emc/parser-v2/ngc/foo.txt
...
#<bar> = [EXISTS[23
-----------------^^
ngc/foo.txt:8:unexpected integer, expecting #
...
m73.1
---^^
if while break do repeat sub endsub return call # G-code M-code A B C
D E F H I J K L P Q R S T U V W X Y Z
...
N3242 G10 x[1
^
ngc/foo.txt:24:unexpected EOL, expecting one of: ] EQ NE GT GE LT LE
and or xor + - * / mod pow
...
N3242 G10 x[1]]
--------------^
while break do repeat sub endsub return call # G-code M-code A B C D
E
F H I J K L P Q R S T U V W X Y Z
..
4 error(s)
---------------
currently, on syntax error the parser skips to the next of line, and
continues, which seems to work ok.
Build notes
-----------
All files needed to compile are in this branch
If you change the grammar: bison 2.5 is needed (Ubuntu 10.04 has
2.4.1 which is too old)
I compiled from source - found no backport.
Flex: the stock flex coming with 10.04 is fine (flex 2.5.35)
Speed is quite good - without spending a thought on optimization and
all debug options on, this parser does about 300.000 lines/second on
my MBP.
added Submakefile
base taken from https://idlebox.net/2007/flex-bison-cpp-example/
much of that example code is still to be deleted but then this is
a proof of concept
-mah
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft
developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft
developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Emc-developers mailing list
https://lists.sourceforge.net/lists/listinfo/emc-developers

ebo@swcp.com

2012-02-05 13:54:39 UTC

Permalink

Oops,that was intended off list... All the same, catch me in a couple of weeks if I have not sent the code.

EBo

Sent from my Android phone on T-Mobile. Americaâs first nationwide 4G network.

----- Reply message -----
From: "***@swcp.com" <***@swcp.com>
To: "EMC developers" <emc-***@lists.sourceforge.net>
Subject: [Emc-developers] rs274ngc bison/flex parser {was: question on gcode parsing}
Date: Sun, Feb 5, 2012 3:10 am

I'm in the middle of a move and will only have sporatic email access for the next week. Also, i do not have ready access to those files.

Could you give me a poke in a week or two?

EBo

Sent from my Android phone on T-Mobile. Americaâs first nationwide 4G network.

----- Reply message -----
From: "Michael Haberler" <***@mah.priv.at>
To: "EBo" <***@sandien.com>
Subject: [Emc-developers] rs274ngc bison/flex parser {was: question on gcode parsing}
Date: Sat, Feb 4, 2012 12:00 pm

Hi Ebo,

the more I look into it the more Spirit looks interesting to me. Would you mail me your example?

regards

-Michael

Post by Michael Haberler
[I guess this better belongs here]
ok, while this wonderful discussion was raging on, I built a working
parser for the current linuxcnc dialect, as an experiment in
feasability (this is NOT an end-user tool!)
http://git.mah.priv.at/gitweb/emc2-dev.git/shortlog/refs/heads/parser-v2-dev
- Michael
ps: I'd appreciate feedback from language geeks wrt to removing
shift/reduce conflicts.
-------------- src/emc/parser-v2/README: --------
RS274NGC parser based on Bison/Flex
===================================
parser [options] ngcfile ...
$ parser emc2-dev/nc_files/*.ngc
no output means good news (parse ok)
other that that, no actions - this is an experiment to find out
wether this is feasible
-v: create listing with interspersed error messages
-s: trace scanning
-p: trace parsing
I'd be grateful for hints how to remove the remaining shift/reduce
conflicts in the grammar. I guess there's something wrong/missing
with
operator precedence and/or associativity.
Error messages
--------------
There are no custom error messages yet. All error messages are
autogenerated from the parser, and derived from the grammar,
including
$ parser -v emc2-dev/src/emc/parser-v2/ngc/foo.txt
...
#<bar> = [EXISTS[23
-----------------^^
ngc/foo.txt:8:unexpected integer, expecting #
...
m73.1
---^^
if while break do repeat sub endsub return call # G-code M-code A B C
D E F H I J K L P Q R S T U V W X Y Z
...
N3242 G10 x[1
^
ngc/foo.txt:24:unexpected EOL, expecting one of: ] EQ NE GT GE LT LE
and or xor + - * / mod pow
...
N3242 G10 x[1]]
--------------^
while break do repeat sub endsub return call # G-code M-code A B C D
E
F H I J K L P Q R S T U V W X Y Z
..
4 error(s)
---------------
currently, on syntax error the parser skips to the next of line, and
continues, which seems to work ok.
Build notes
-----------
All files needed to compile are in this branch
If you change the grammar: bison 2.5 is needed (Ubuntu 10.04 has
2.4.1 which is too old)
I compiled from source - found no backport.
Flex: the stock flex coming with 10.04 is fine (flex 2.5.35)
Speed is quite good - without spending a thought on optimization and
all debug options on, this parser does about 300.000 lines/second on
my MBP.
added Submakefile
base taken from https://idlebox.net/2007/flex-bison-cpp-example/
much of that example code is still to be deleted but then this is
a proof of concept
-mah
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft
developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users
------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft
developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3,
MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
_______________________________________________
Emc-developers mailing list
https://lists.sourceforge.net/lists/listinfo/emc-developers