Initially, this question may not align well with the typical queries on StackOverflow. The site generally caters to inquiries that are precise and revolve around actual code.
Ideally, I am interested in parsing the language using a method other than regex.
Before delving into code, it's crucial to grasp how a language parser operates. Avoid relying solely on regular expressions, as they're tailored for regular languages while JavaScript isn't one.
The process of language parsing comprises two main phases: lexical analysis and token parsing. Breaking down text into tokens simplifies the parsing task compared to directly parsing the text. Despite JS introducing minor complexities due to lexical ambiguities stemming from the use of / for division, comments, and regular expressions, these challenges are surmountable.
Create a lexer followed by a parser to tackle your parsing needs. However, a lexer alone might suffice for your requirements.
It's essential to handle correct parsing of potentially flawed JS code, considering that users typically input incomplete or erroneous programs during typing. Implementing effective error recovery mechanisms in your lexer and parser is pivotal for ensuring a seamless user experience.
Is it necessary to reparse the entire text and apply formatting after every keystroke?
Performance considerations hinge on the efficiency of your parser and the size of the file being processed.
In developing the Roslyn syntax highlighter, our team encountered scenarios involving files with extensive line counts still undergoing edits. Due to performance constraints, we refrain from re-parsing the complete file post each keypress. Instead, we leverage an immutable parse tree for swift traversal to identify the impacted token. Subsequently, we determine the parse nodes requiring re-evaluation, initiating re-lexing and re-parsing exclusively on those nodes. This approach enables us to construct a new immutable parse tree incorporating unaltered segments from the prior tree.
Furthermore, syntax coloring operations are confined to portions of the file visible to the user.
It's worth noting that alongside syntactic evaluations between keystrokes, Roslyn extends its capabilities to encompass semantic analyses, marking a distinct facet of its functionality.