Class CommonTokenFactory

    • Field Detail

      • copyText

        protected final boolean copyText
        Indicates whether CommonToken.setText(java.lang.String) should be called after constructing tokens to explicitly set the text. This is useful for cases where the input stream might not be able to provide arbitrary substrings of text from the input after the lexer creates a token (e.g. the implementation of CharStream.getText(org.antlr.v4.runtime.misc.Interval) in UnbufferedCharStream throws an UnsupportedOperationException). Explicitly setting the token text allows Token.getText() to be called at any time regardless of the input stream implementation.

        The default value is false to avoid the performance and memory overhead of copying text for every token unless explicitly requested.

    • Constructor Detail

      • CommonTokenFactory

        public CommonTokenFactory​(boolean copyText)
        Constructs a CommonTokenFactory with the specified value for copyText.

        When copyText is false, the DEFAULT instance should be used instead of constructing a new instance.

        Parameters:
        copyText - The value for copyText.
      • CommonTokenFactory

        public CommonTokenFactory()
        Constructs a CommonTokenFactory with copyText set to false.

        The DEFAULT instance should be used instead of calling this directly.

    • Method Detail

      • create

        public CommonToken create​(Pair<TokenSource,​CharStream> source,
                                  int type,
                                  java.lang.String text,
                                  int channel,
                                  int start,
                                  int stop,
                                  int line,
                                  int charPositionInLine)
        Description copied from interface: TokenFactory
        This is the method used to create tokens in the lexer and in the error handling strategy. If text!=null, than the start and stop positions are wiped to -1 in the text override is set in the CommonToken.
        Specified by:
        create in interface TokenFactory<CommonToken>