Author Topic: AST: Delimiters not detectable in string literals?  (Read 1633 times)

RIL

  • EA User
  • **
  • Posts: 142
  • Karma: +3/-0
  • -- There's always a solution --
    • View Profile
AST: Delimiters not detectable in string literals?
« on: August 23, 2013, 01:55:10 pm »
This really puzzles me.  For some reason it seems impossible to get more detail out of a string literal than only whole strings between the quote chars (using skip()) as shown below.

Only the first expression below works, all others fail, no matter how I try. Since "<IDENTIFIER>" is registered as a "token" I've tried to define my own <MYIDENTIFIER> (assuming that token would somehow cause a problem) but not even the simplest string (like "fmt") will be possible to detect and strip out with anything but the skip() command between the quotes. Why is this? (see far below what end I want to achieve)

Strings to Match:

Code: [Select]
import (
    "fmt"
    f "fmt"
    sys1 "path.to.system"
    sys2 "path/to/system"
    // or a mix of the above
)

AST Expressions:

Code: [Select]
<importPackage>      ::= [<IDENTIFIER>)] <stringliteral>;            // * OK

<importPackage> ::= [<IDENTIFIER>)] "\"" <qualifiedName> "\"";      // FAIL
<importPackage> ::= [<IDENTIFIER>)] "\"" <qualifiedPath> "\"";      // FAIL
<importPackage> ::= [<IDENTIFIER>)] "\"" <IDENTIFIER> "\"";      // FAIL

<qualifiedName>      ::= <IDENTIFIER> ("." <IDENTIFIER>)*;            // Nah
<qualifiedPath>      ::= <IDENTIFIER> ("/" <IDENTIFIER>)*;            // Nah
<stringLiteral>      ::= "\"" skip("\"") "\"";                  // * Works!

What I want to achieve is picking the path (qualifier) and name parts out of the import packages (strings).  For example I'd like to separate the alias, the path and the final name parts from this sampole code:
Code: [Select]
import (
    sys "github.com/riluser/mysyspkg"
)
like so:
    [*]ALIAS: sys
    [*]PATH: github.com/riluser/
    [*]NAME: mysyspkg
    [/list]

    But for now I can only separate the alias and the string literal as a whole. Delimiters simply don't work, nor does it work with any kind of expression but the skip() function between quotes.

    How can that be? Puzzled. I must be doing something wrong.

    // Rolf
    « Last Edit: August 23, 2013, 02:02:01 pm by Rolf_Lampa »
    -- There's always a solution --

    RIL

    • EA User
    • **
    • Posts: 142
    • Karma: +3/-0
    • -- There's always a solution --
      • View Profile
    Re: AST: Delimiters not detectable in string liter
    « Reply #1 on: August 23, 2013, 09:50:05 pm »
    OK, I removed the string rule from the token list of lexer rules, and now I get what I want from the following source:

    Code: [Select]
    import  (
          "fmt"
          _ "hello/suppress"
          mux "github.com/gorilla/mux"
          "../LingualDomain"
          "net/http"
    )
    The above is parsed into the following (desired) AST tree:

    Code: [Select]
    FILE([0-3811][0:0])
          PACKAGE([0-14][1:1]): {NAME = Server}
          IMPORT([599-7051 [19:1])
                PKG([611-616][20 2]) {NAMEPART = fmt, NAME = "fmt"}
                PKG([619-637][21 2]) {ALIAS = _, QUALIFIER = hello, NAMEPART = suppress, NAME = "hello/suppress")
                PKG([640-668][22 2]) {ALIAS = mux, QUALIFIER = github.com, QUALIFIER = gorilla, NAMEPART = mux, NAME = "github.com/gorilla/mux"}
                PKG([671-689][23 2]) {QUALIFIER = .., NAMEPART = LingualDomain, NAME = "../LingualDomain"}
                PKG([692-702][24 2]) {QUALIFIER = net, NAMEPART = http, NAME = "net/http"}
          VARIABLE([709-752][27:1])... aso
    I used thie following expression to extract the above info:

    Code: [Select]
    <importDeclaration>      ::= node("IMPORT", "import" "(" <importPackage>* ")" ) ;                                                      
    <importPackage>       ::= node("PKG", [attribute("ALIAS", <IDENTIFIER>)] attribute("NAME", (<importQualifiedName>| <stringliteral>) ) );
    <importQualifiedName>      ::= "\"" (attribute("QUALIFIER", (<qualifiedName> | [".."])) "/")* attribute("NAMEPART", <qualifiedName> ) "\"";

    Comments still suffer from the same problem which stopped this solution earlier (that is, comment being defined as  a token makes it not very useful in expressions for picking them up for further processing).

    // Rolf
    « Last Edit: August 24, 2013, 02:05:07 am by Rolf_Lampa »
    -- There's always a solution --

    Eve

    • EA Administrator
    • EA Guru
    • *****
    • Posts: 7732
    • Karma: +103/-18
      • View Profile
    Re: AST: Delimiters not detectable in string liter
    « Reply #2 on: August 26, 2013, 08:47:59 am »
    Excellent, that's the suggestion that I was going to make.

    PS. See your other post about the comments.
    « Last Edit: August 26, 2013, 08:48:16 am by simonm »
    Eve

    support@sparxsystems.com