AST: Delimiters not detectable in string literals?

[RIL]

This really puzzles me.  For some reason it seems impossible to get more detail out of a string literal than only whole strings between the quote chars (using skip()) as shown below.

Only the first expression below works, all others fail, no matter how I try. Since "<IDENTIFIER>" is registered as a "token" I've tried to define my own <MYIDENTIFIER> (assuming that token would somehow cause a problem) but not even the simplest string (like "fmt") will be possible to detect and strip out with anything but the skip() command between the quotes. Why is this? (see far below what end I want to achieve)

Strings to Match:

Code: [Select]

import (
    "fmt"
    f "fmt"
    sys1 "path.to.system"
    sys2 "path/to/system"
    // or a mix of the above
)
AST Expressions:

Code: [Select]

<importPackage>      ::= [<IDENTIFIER>)] <stringliteral>;            // * OK

<importPackage> ::= [<IDENTIFIER>)] "\"" <qualifiedName> "\"";      // FAIL
<importPackage> ::= [<IDENTIFIER>)] "\"" <qualifiedPath> "\"";      // FAIL
<importPackage> ::= [<IDENTIFIER>)] "\"" <IDENTIFIER> "\"";      // FAIL

<qualifiedName>      ::= <IDENTIFIER> ("." <IDENTIFIER>)*;            // Nah
<qualifiedPath>      ::= <IDENTIFIER> ("/" <IDENTIFIER>)*;            // Nah
<stringLiteral>      ::= "\"" skip("\"") "\"";                  // * Works!
What I want to achieve is picking the path (qualifier) and name parts out of the import packages (strings).  For example I'd like to separate the alias, the path and the final name parts from this sampole code:

Code: [Select]

import (
    sys "github.com/riluser/mysyspkg"
)like so:

[*]ALIAS: sys
[*]PATH: github.com/riluser/
[*]NAME: mysyspkg
[/list]

But for now I can only separate the alias and the string literal as a whole. Delimiters simply don't work, nor does it work with any kind of expression but the skip() function between quotes.

How can that be? Puzzled. I must be doing something wrong.

// Rolf

[RIL]

OK, I removed the string rule from the token list of lexer rules, and now I get what I want from the following source:

Code: [Select]

import  (
      "fmt"
      _ "hello/suppress"
      mux "github.com/gorilla/mux"
      "../LingualDomain"
      "net/http"
)The above is parsed into the following (desired) AST tree:

Code: [Select]

FILE([0-3811][0:0])
      PACKAGE([0-14][1:1]): {NAME = Server}
      IMPORT([599-7051 [19:1])
            PKG([611-616][20 2]) {NAMEPART = fmt, NAME = "fmt"}
            PKG([619-637][21 2]) {ALIAS = _, QUALIFIER = hello, NAMEPART = suppress, NAME = "hello/suppress")
            PKG([640-668][22 2]) {ALIAS = mux, QUALIFIER = github.com, QUALIFIER = gorilla, NAMEPART = mux, NAME = "github.com/gorilla/mux"}
            PKG([671-689][23 2]) {QUALIFIER = .., NAMEPART = LingualDomain, NAME = "../LingualDomain"}
            PKG([692-702][24 2]) {QUALIFIER = net, NAMEPART = http, NAME = "net/http"}
      VARIABLE([709-752][27:1])... asoI used thie following expression to extract the above info:

Code: [Select]

<importDeclaration>      ::= node("IMPORT", "import" "(" <importPackage>* ")" ) ;                                                      
<importPackage>       ::= node("PKG", [attribute("ALIAS", <IDENTIFIER>)] attribute("NAME", (<importQualifiedName>| <stringliteral>) ) );
<importQualifiedName>      ::= "\"" (attribute("QUALIFIER", (<qualifiedName> | [".."])) "/")* attribute("NAMEPART", <qualifiedName> ) "\"";
Comments still suffer from the same problem which stopped this solution earlier (that is, comment being defined as  a token makes it not very useful in expressions for picking them up for further processing).

// Rolf

[Eve]

Excellent, that's the suggestion that I was going to make.

PS. See your other post about the comments.