Version Changelog

@lezer/generator 1.7.0 (2024-03-12)

Bug fixes

Include type declarations for the Rollup plugin.

Named or @export-ed specialized tokens are now available in the terms file.

New features

The generator now emits a warning when rules generate a lot of different variants (usually due to a combinatory explosion of ? and | operators).

@lezer/lr 1.4.0 (2024-01-22)

New features

The new InputStream.acceptTokenTo method allows a tokenizer to create a token that ends at some absolute position.

@lezer/common 1.2.1 (2024-01-16)

Bug fixes

Fix a bug where getChild and getChildren would, if the first and second arguments matched the same node, return incorrect results.

@lezer/generator 1.6.0 (2024-01-08)

Bug fixes

Fix an issue where the generator could output invalid JavaScript when a specialization used a string that started with a number.

Adjust TypeScript output to compile with recent tsc versions. Add a test for zero-length node mounts

New features

Support an exportName option to the Rollup plugin.

@lezer/common 1.2.0 (2023-12-28)

New features

The new NodeProp.isolate can be used by parser to signal that a given node should be treated as isolated when it comes to bidirectional text rendering.

@lezer/common 1.1.2 (2023-12-07)

Bug fixes

Fix a crash that could happen in mixed-language parsing when mounting a tree to a zero-length node.

@lezer/highlight 1.2.0 (2023-11-12)

New features

The new highlightCode function provides a higher-level interface for emitting highlighted code.

@lezer/common 1.1.1 (2023-11-10)

Bug fixes

Fix a bug where resolveStack could sometimes yield the same node multiple times.

Allow mixed-parsing trees to be mounted for zero-length nodes.

Fix a bug in mixed-language parsing that could some parts of an inner language to be missed when incrementally reparsing from a stopped parse.

In Tree.build, when given an extremely deeply nested tree structure, flatten it instead of overflowing the stack.

@lezer/lr 1.3.14 (2023-11-02)

Bug fixes

Further reduce maximum parse depth—the old limit could still cause the tree building function to overflow the stack on V8 when all rules on the stack produce actual visible tree nodes.

@lezer/lr 1.3.13 (2023-10-03)

Bug fixes

The parser will now emit an error node when it reaches the end of the top rule but not the end of the input, and restarts parsing the top rule.

Fix an issue that could, when error-recovery happened after a skipped token, cause the parser to emit nodes whose start position was after their end position.

@lezer/lr 1.3.12 (2023-09-22)

Bug fixes

Reduce tree depth limit to avoid V8 overflowing its stack.

@lezer/common 1.1.0 (2023-09-19)

New features

The new Tree.resolveStack method returns an iterator through the nodes covering a position, including those from overlays not active at that point.

@lezer/lr 1.3.11 (2023-09-19)

Bug fixes

When parsing input ranges with gaps, don't treat the token after a gap as covering the entire gap.

@lezer/generator 1.5.1 (2023-09-15)

Bug fixes

Fix a quadratic complexity in state merging.

@lezer/generator 1.5.0 (2023-08-20)

Bug fixes

Fix a build issue that made the ES version of the Rollup plugin fail to load.

New features

The new typeScript option to buildParserFile (and --typeScript option to lezer-generator) makes the tool emit TypeScript code.

@lezer/common 1.0.4 (2023-08-17)

Bug fixes

Make the package usable in TypeScript with node16/nodenext resolution.

@lezer/lr 1.3.10 (2023-08-17)

Bug fixes

Make this package usable in TypeScript setups using node16/nodenext resolution.

@lezer/generator 1.4.2 (2023-08-17)

Bug fixes

Fix a regression in the build process that caused the Rollup plugin to not be part of the npm package.

@lezer/generator 1.4.1 (2023-08-17)

Bug fixes

Make this package usable in TypeScript setups with node16/nodenext resolution.

@lezer/generator 1.4.0 (2023-08-11)

New features

BuildOptions.contextTracker now takes a function, so that the code that produces it has access to the term IDs.

@lezer/lr 1.3.9 (2023-07-03)

Bug fixes

Fix another issue in the way local token groups and input gaps interact.

@lezer/lr 1.3.8 (2023-07-03)

Bug fixes

Fix an issue that caused incorrect trees to be emitted when a forced reduction happened inside a skipped term.

@lezer/lr 1.3.7 (2023-06-22)

Bug fixes

Fix a bug where the parser could skip characters while tokenizing in a local token group.

@lezer/generator 1.3.0 (2023-06-15)

New features

The test utilities can now be imported as "@lezer/generator/test".

@lezer/generator 1.2.4 (2023-06-12)

Bug fixes

Fix a bug where precedences specified for local tokens were not properly applied.

@lezer/common 1.0.3 (2023-06-02)

Bug fixes

Tree.iterate now properly includes anonymous nodes when IterMode.IncludeAnonymous is enabled.

@lezer/lr 1.3.6 (2023-06-02)

Bug fixes

Fix an issue where context-tracking parsers, in specific circumstances, could create incorrectly nested trees.

@lezer/highlight 1.1.6 (2023-05-26)

Bug fixes

Fix a bug in that could cause it to add random highlighting to the text between parts of an overlaid tree.

@lezer/lr 1.3.5 (2023-05-25)

Bug fixes

Fix an issue where some grammars could make the parser unable to force reductions on invalid input, leading to trees that were missing some wrapping nodes around the broken input.

@lezer/highlight 1.1.5 (2023-05-24)

Bug fixes

Inheritable (/...) styles no longer apply across mounted subtrees.

@lezer/generator 1.2.3 (2023-04-28)

Bug fixes

Make sure the Rollup plugin imports the rest of the library using a full file path.

Make --help show the correct executable name.

@lezer/lr 1.3.4 (2023-04-24)

Bug fixes

Fix tokenizing of @else tokens that span separate input ranges.

@lezer/highlight 1.1.4 (2023-03-24)

Bug fixes

Make sure TypeScript declaration file has the same name as the ES module to avoid some TypeScript resolution issues.

@lezer/lr 1.3.3 (2023-02-02)

Bug fixes

Fix a crash when parsing a non-atomic skippable at the top of a nested parse.

Fix a bug that caused the tree-depth limit on left-associative rules to be applied to rules generated for repeat operators.

@lezer/lr 1.3.2 (2023-01-31)

Bug fixes

Place a limit on the amount of (non-recovering) parse stacks active at the same time to avoid exponential slowdown on inputs that nest ambiguous constructs.

Reduce the maximum left-associative nesting depth, since the old value could still cause Chrome (but not Firefox) to overflow its call stack.

@lezer/generator 1.2.2 (2023-01-18)

Bug fixes

Make sure require isn't used as an identifier in generator output.

@lezer/lr 1.3.1 (2023-01-13)

Bug fixes

Fix a bug that could corrupt the parse stack when parsing a non-single-token skippable expression from the initial parse state.

@lezer/generator 1.2.1 (2023-01-13)

Bug fixes

Fix an invalid optimization that sometimes led to incomplete token precedence tables, leading to incorrect tokenization.

@lezer/lr 1.3.0 (2023-01-09)

Bug fixes

Avoid the problem of huge numbers of repeated left-associative operator applications creating a stack too deep to recurse on by detecting such parses and force-reducing out of them.

New features

Add support for grammars that use @local tokens declarations.

@lezer/generator 1.2.0 (2023-01-09)

Bug fixes

Fix a bug where the error to use () to denote empty options in a choice expression was signalled even when that is what the input did.

New features

Grammars can now declare @local tokens blocks defining all the tokens that may appear in a set of state, and allowing @else tokens that match everything else.

@lezer/highlight 1.1.3 (2022-11-25)

Bug fixes

Fix a bug where the highlighting over overlaid ranges falling within a highlighted node in the parent tree was sometimes broken.

@lezer/common 1.0.2 (2022-11-23)

Bug fixes

Fix a bug in mixed parsing that would sometimes produce invalid trees.

@lezer/lr 1.2.5 (2022-11-16)

Bug fixes

Fix a bug that made Stack.canShift incorrectly return true in some circumstances.

@lezer/generator 1.1.3 (2022-11-07)

Bug fixes

Fix broken Rollup plugin due to a bad import.

@lezer/generator 1.1.2 (2022-11-07)

Bug fixes

Fix a bug where the _ notation in tokens would match only half of a surrogate pair.

Fix a bug that caused invalid tokenizer data to be generated for character ranges ending in \uffff.

@lezer/lr 1.2.4 (2022-11-05)

Bug fixes

Fix a tokenizer bug that broke tokenizing of the character \uffff.

@lezer/highlight 1.1.2 (2022-10-18)

Bug fixes

Fix an issue where unmodified tags were treated as having higher specificity than modified ones when computing precedence for tags with multiple modifiers.

@lezer/highlight 1.1.1 (2022-09-23)

Bug fixes

Make sure all highlighting is applied even to nodes with no associated highlight tags.

@lezer/highlight 1.1.0 (2022-09-20)

New features

The new getStyleTags function can be used to query the style tags that match a given syntax nodes.

@lezer/common 1.0.1 (2022-08-30)

Bug fixes

Fix a bug that could cause incremental parsing to incorrectly reuse nodes inside nested mixed parses.

@lezer/lr 1.2.3 (2022-08-16)

Bug fixes

Fix a regression in the way tokenizers read input when parsing multiple separate ranges.

@lezer/lr 1.2.2 (2022-08-12)

Bug fixes

Fix an issue that could cause the tokenizer to crash when tokens ended directly on a gap in the input ranges.

@lezer/lr 1.2.1 (2022-08-03)

Bug fixes

Include LRParser.deserialize in the TypeScript declarations.

Fix a bug that broke ParserConfig.specializers (requires @lezer/generator 1.1.1).

@lezer/generator 1.1.1 (2022-08-03)

Bug fixes

Emit enough information about external specializers to make ParserConfig.specializers fixable.

@lezer/lr 1.2.0 (2022-07-04)

New features

External specializers can now be replaced when reconfiguring an LRParser.

@lezer/lr 1.1.0 (2022-06-27)

New features

Support the output produced by the new @eof marker in lezer-generator.

@lezer/generator 1.1.0 (2022-06-27)

New features

Things that used to be written like std.digit are now written @digit. The old notation will remain available until a breaking release.

The new @eof marker can be used in tokens to match the end of the input.

@lezer/common 1.0.0 (2022-06-06)

New features

First stable version.

@lezer/lr 1.0.0 (2022-06-06)

New features

First stable version.

@lezer/highlight 1.0.0 (2022-06-06)

New features

First stable version.

@lezer/generator 1.0.0 (2022-06-06)

New features

First stable version.

@lezer/common 0.16.1 (2022-06-01)

Bug fixes

Declare matchContext as a property of SytaxNodeRef.

@lezer/lr 0.16.3 (2022-05-11)

Bug fixes

Don't crash when a non-Node environment defines a process global that doesn't have an env property.

Fix a bug that could cause partially-parsed nodes to be reused when they ended in skipped nodes, when parsing incrementally.

@lezer/lr 0.16.2 (2022-04-21)

Bug fixes

Fix an issue that could make the parser merge adjacent error nodes even when they had different parent nodes.

@lezer/lr 0.16.1 (2022-04-21)

Bug fixes

Fix an issue that caused error nodes created for inserted tokens to sometimes end up in the wrong parent node.

@lezer/common 0.16.0 (2022-04-20)

Breaking changes

The mixed parsing interface now passes SyntaxNodeRef values instead of TreeCursor instances around.

Creating a cursor at a given position is now done with Tree.cursorAt, not cursor.

Getting a cursor from a syntax node is now done with a method (cursor) rather than a getter.

Tree.fullCursor was removed and replaced by the IncludeAnonymous iteration mode.

The optional arguments to enter tree traversal methods were replaced with a single mode argument.

Tree.iterate now passes node objects, rather than type/from/to as separate arguments, to its callback functions.

New features

The new SyntaxNodeRef type provides an interface shared by SyntaxNode and TreeCursor.

TreeCursor instances now how have an iterate method that calls a function for each descendant of the current node.

The new matchContext method on SyntaxNode and TreeCursor provides a convenient way to match the names of direct parent nodes.

Cursors can now be passed flags to control their behavior.

@lezer/lr 0.16.0 (2022-04-20)

Bug fixes

Avoid the need for generated parser files to directly import @lezer/common.

@lezer/highlight 0.16.0 (2022-04-20)

Breaking changes

First numbered release.

@lezer/generator 0.16.0 (2022-04-20)

Bug fixes

Fix an issue in the tokenizer for grammars that could cause it to run very slowly on some inputs.

New features

A grammar can now include an @external propSource a from "b" declaration to import a programmatically defined node prop source.

@lezer/common 0.15.12 (2022-03-18)

Bug fixes

Work around a TypeScript issue that caused it to infer return type any for resolve and resolveInner.

Fix a bug in incremental mixed-language parsing where it could incorrectly add a parse range twice, causing a crash in the inner parser.

@lezer/lr 0.15.8 (2022-02-04)

Bug fixes

Fix a bug that caused reductions that didn't consume anything to sometimes end up outside their parent node in the tree.

@lezer/generator 0.15.4 (2022-01-28)

Bug fixes

Fix a bug where explicitly specified token precedences were sometimes not properly enforced.

@lezer/lr 0.15.7 (2022-01-21)

Bug fixes

Fix a bug that could cause some kinds of composite skipped expressions to not be parsed correctly.

@lezer/generator 0.15.3 (2022-01-21)

Bug fixes

Fix a bug that caused some kind of skip rules (those ending in something optional) to not work correctly.

@lezer/lr 0.15.6 (2022-01-11)

Bug fixes

Make sure tree depth does not grow unbounded, so that recursive tree traversal is safe from overflowing the stack.

Be less agressive about pruning long-running GLR parse splits.

@lezer/common 0.15.11 (2021-12-16)

Bug fixes

Fix a bug where nested parsing would sometimes corrupt the length of parent nodes around the nesting point.

@lezer/lr 0.15.5 (2021-12-01)

Bug fixes

Fix a bug that caused node lookahead to be one less than it should be.

Fix an issue that could cause the parser to, when recovering, parse beyond the position given to stopAt.

@lezer/common 0.15.10 (2021-11-30)

New features

SyntaxNode now has a resolveInner method (analogous to Tree.resolveInner).

@lezer/common 0.15.9 (2021-11-24)

Bug fixes

Full tree cursors no longer automatically enter mounted subtrees.

Fix a bug where a nested parser would not re-parse inner sections when given fragments produced by a parse that finished the outer tree but was stopped before the inner trees were done.

@lezer/common 0.15.8 (2021-11-10)

Bug fixes

Fix a bug that could cause incorrectly structured nodes to be created for repeat rules, breaking incremental parsing using the resulting tree.

@lezer/common 0.15.7 (2021-10-05)

Bug fixes

Fix an issue in parseMixed where parses nested two or more levels deep would use the wrong document offsets.

@lezer/common 0.15.6 (2021-09-30)

Bug fixes

Fix a null-dereference crash in mixed-language parsing.

@lezer/lr 0.15.4 (2021-09-27)

Bug fixes

Fix a mistake in the way forced reductions are checked.

@lezer/lr 0.15.3 (2021-09-24)

Bug fixes

Fix crashes or infinite recursion caused by applying forced reductions in situations where they were not valid.

@lezer/generator 0.15.2 (2021-09-24)

Bug fixes

Fix an infinite recursion caused by some kinds of (obscure) token state machines.

@lezer/common 0.15.5 (2021-09-09)

New features

Syntax node objects now have a method enterUnfinishedNodesBefore to scan down the tree for nodes that were broken off directly in front of a given position (which can provide a more accurate context that just resolving the position).

@lezer/generator 0.15.1 (2021-09-03)

Bug fixes

Fix a bug that could lead to spurious 'inconsistent skip sets' errors.

Fix a bug that caused an unescaped '-' at the start or end of a character set to silently be converted to a nonsensical character.

Fix a confusing behavior where literal tokens declared in the @tokens block didn't get names when they started with a lower-case character.

New features

Top rules may now be defined inside @skip scopes.

The parser no longer treats an empty position in a choice operator as the empty expression, but requires an explicit () marker (to avoid a common mistake).

@lezer/common 0.15.4 (2021-08-31)

Bug fixes

parseMixed will now scan children not covered by the ranges of an eagerly computed overlay for further nesting.

@lezer/lr 0.15.2 (2021-08-31)

Bug fixes

Fix a bug where the parse position could get corrupted during a parse, leading to crashes or nonsensical output.

@lezer/lr 0.15.1 (2021-08-16)

Bug fixes

Fix an inconsistency in the .d.ts emitted by the build.

New features

LRParser.configure now takes a contextTracker option to replace the context tracker used by the parser.

@lezer/common 0.15.3 (2021-08-12)

Bug fixes

Fix an issue where parseMixed could create overlay mounts with zero ranges, which were useless and confused CodeMirror's highlighter.

@lezer/common 0.15.2 (2021-08-12)

Bug fixes

Fix a bug that would cause enter to return incorrect results when called entering children in a buffer with .

@lezer/common 0.15.1 (2021-08-12)

Bug fixes

Fix a bug where parseMixed could crash by dereferencing null.

@lezer/common 0.15.0 (2021-08-11)

Breaking changes

The module name has changed from lezer-tree to @lezer/common.

TreeBuffers no longer accept a node type.

Tree.balance no longer takes a buffer size as argument.

NodeProp.string, NodeProp.number, and NodeProp.flag have been removed (the thing they provided is trivial to write by hand).

A node's context hash is now stored in the NodeProp.contextHash prop.

Reused nodes passed to Tree.build must now be Tree instances (not tree buffers).

Parser is now an abstract class that all parser implementations must extend, implementing the createParse method.

The PartialParse interface changed to make multi-pass parsers possible.

Parser.startParse now takes different arguments (input, fragments, ranges) instead of (input, startPos, context).

The Input interface has changed (to become chunk-based and more low-level). A single Input object is now shared between outer and inner parses.

stringInput is no longer exported (Parser methods will automatically wrap strings when appropriate).

Bug fixes

Fix a bug in TreeFragment.applyChanges that prevented some valid reuse of syntax nodes.

Fix a bug where reused nodes could incorrectly be dropped by Tree.build.

New features

Node props can now be per-node, in which case they are stored on Tree instances rather than NodeTypes.

Tree nodes can now be replaced with other trees through NodeProp.mountedTree.

Tree.resolveInner can now be used to resolve into overlay trees.

SyntaxNode objects now have a toTree method to convert them to a stand-alone tree.

Tree.balance now accepts a helper function to create the inner nodes.

Tree cursors' next/prev methods now take an enter argument to control whether they enter the current node.

SyntaxNode and TreeCursor now have an enter method to directly enter the child at the given position (if any).

Tree.iterate callbacks now get an extra argument that allows them to create a SyntaxNode for the current node.

The parsing interface now supports parsing specific, non-contiguous ranges of the input in a single parse.

The module now exports a parseMixed helper function for creating mixed-language parsers.

@lezer/lr 0.15.0 (2021-08-11)

Breaking changes

The module's name changed from lezer to @lezer/lr.

The Parser class was renamed to LRParser (Parser is now the abstract class that all parsers extend).

Nested parsing is no longer handled inside the LR parser (instead, it is done by using parseMixed from the @lezer/common package as a wrapper).

External tokenizers are passed different arguments and get a different input stream abstraction (to make accidental uncontrolled lookahead and lookbehind, which were easy ways to break incremental parsing, harder).

Drops support for Stack.startOf (which is no longer useful without lookbehind).

Context trackers also get passed different arguments, following the changed input stream format.

This package no longer re-exports bindings from @lezer/common.

Bug fixes

Slightly prefer deleting token to inserting them to avoid far-fetched parses.

Fix a problem where node reuse didn't take the amount of look-ahead done by the tokenizer into account, and could reuse nodes whose content would tokenize differently due to changes after them.

Track tokenizer lookahead to fix a problem where some incremental parses would produce incorrect results.

New features

LR parsers now support a wrapper configuration option that allows you to inject additional logic by wrapping the PartialParse object it returns.

@lezer/generator 0.15.0 (2021-08-11)

Breaking changes

The module's name changed from lezer-generator to @lezer/generator.

Nested parsers can no longer be specified in the grammar (they must now be set up programmatically).

Bug fixes

Fix an issue where newlines in string tokens could silently corrupt the token.

Handle alternative output file extensions more gracefully.

New features

@export props may now have a value to set a specific export name.

lezer 0.13.5 (2021-05-14)

Bug fixes

Fix a bug with overeager reuse of nodes on change boundaries.

lezer-generator 0.13.4 (2021-05-14)

Bug fixes

Don't add inline rules to the terms file (since they may not be uniquely identified by name).

Generate more minimal state machines for the tokenizer.

lezer 0.13.4 (2021-03-03)

New features

Parser instances now have a topNode property that holds the type of the parser's top node.

lezer-tree 0.13.2 (2021-02-17)

New features

Add support for context tracking.

lezer 0.13.3 (2021-02-17)

New features

Context trackers can now disable strictness in node reuse.

lezer 0.13.2 (2021-02-17)

New features

Add support for context trackers.

lezer-generator 0.13.3 (2021-02-17)

New features

Support @context syntax to register a context tracker for a grammar.

lezer-tree 0.13.1 (2021-02-11)

Bug fixes

Fix a bug where building a tree from a buffer would go wrong for repeat nodes whose children were all repeat nodes of the same type.

lezer-generator 0.13.2 (2021-01-20)

Bug fixes

Fix an issue where imported identifiers could clash with the export name in generated code.

lezer-tree 0.13.0 (2020-12-04)

Breaking changes

NodeType.isRepeated is now called isAnonymous, which more accurately describes what it means.

NodeGroup has been renamed to NodeSet to avoid confusion with NodeProp.group.

The applyChanges method on trees is no longer supported (TreeFragment is now used to track reusable content).

Trees no longer have cut and append methods.

New features

It is now possible to pass a node ID to SyntaxNode.getChild/getChildren and NodeType.is. Allow specifying a tree length in Tree.build

Tree.build now allows you to specify the length of the resulting tree.

Tree.fullCursor() can now be used to get a cursor that includes anonymous nodes, rather than skipping them.

Introduces NodeType.define to define node types.

The new TreeFragment type is used to manage reusable subtrees for incremental parsing.

Tree.build now accepts a start option indicating the start offset of the tree.

The Input type, which used to be InputStream in the lezer package, is now exported from this package.

This package now exports a PartialParse interface, which describes the interface used, for example, as return type from Parser.startParse.

lezer 0.13.1 (2020-12-04)

Bug fixes

Fix versions of lezer packages depended on.

lezer 0.13.0 (2020-12-04)

Breaking changes

Parser.group is now called Parser.nodeSet.

Nested parsers now work differently. They don't have to be Lezer parsers, but have to produce object conforoming to the PartialParse interface. The interface with which non-trivial nested parsers are specified also changed—see the NestedParser type.

Parser objects no longer have a topType property (scan their node set for types with isTop set instead).

Parser objects no longer have withProps, withNested, and withTokenizer methods (use configure instead).

Both Parser.parse and Parser.startParse now take an optional start position as second parameter and an optional parse context as third. startParse returns an instance of the PartialParse interface instead of the old ParseContext class (whose name is now used for something else). Parse options are no longer passed to these methods, but configured in advance through Parser.configure.

During incremental parsing, instead of passing a tree as the cache option, reusable subtrees (see TreeFragment from lezer-tree) are now retrieved from the fragments property of the parse context object, if provided.

Parser.parse and Parser.startParse no longer take an options parameter. Instead, bufferLength and strict can be configured with Parser.configure now, and the start position and context are passed as optional arguments.

The InputStream type has been replaced by Input from the lezer-tree package (which has the same interface but a more appropriate name).

New features

The Parser class now has a configure method that is used to create a parser instance with a more specific configuration.

lezer-generator 0.13.1 (2020-12-04)

Bug fixes

Fix versions of lezer packages depended on.

lezer-generator 0.13.0 (2020-12-04)

Breaking changes

Adjust to the new way nested parsers work in Lezer.

Bug fixes

Top rule node types will now show up in the terms file.

It is no longer allowed for a top rule to share a name with another rule.

lezer 0.12.1 (2020-11-19)

Bug fixes

Fix an infinite loop in incremental parsing when repeatedly reusing a zero-length cached node.

lezer-tree 0.12.3 (2020-11-02)

New features

Make NodePropSource a function type.

lezer-tree 0.12.2 (2020-10-28)

Bug fixes

Fix a bug that made SyntaxNode.prevSibling fail in most cases when the node is part of a buffer.

lezer-tree 0.12.1 (2020-10-26)

Bug fixes

Fix issue where using Tree.append with an empty tree as argument would return a tree with a nonsensical length property.

lezer-tree 0.12.0 (2020-10-23)

Breaking changes

Tree.iterate no longer allows returning from inside the iteration (use cursors directly for that kind of use cases).

Subtree has been renamed to SyntaxNode and narrowed in scope a little.

The top, skipped, and error node props no longer exist.

New features

The package now offers a TreeCursor abstraction, which can be used for both regular iteration and for custom traversal of a tree.

SyntaxNode instances have nextSibling/prevSibling getters that allow more direct navigation through the tree.

Node types now expose isTop, isSkipped, isError, and isRepeated properties that indicate special status.

Adds NodeProp.group to assign group names to node types.

Syntax nodes now have helper functions getChild and getChildren to retrieve direct child nodes by type or group.

NodeType.match (and thus NodeProp.add) now allows types to be targeted by group name.

Node types have a new is method for checking whether their name or one of their groups matches a given string.

lezer 0.12.0 (2020-10-23)

Breaking changes

Follow the change from Subtree to TreeCursor in lezer-tree.

The serialized parser format changed.

Stack.startOf now returns null, rather than -1, when it can't find the given element.

New features

Stack.startOf now takes an optional second argument that allows you to select a match beyond the innermost one.

lezer-generator 0.12.0 (2020-10-23)

Breaking changes

The serialized parser format changed.

Pseudo-props like name, dialect, inline and dynamicPrec now require an @ in front of them when specified in a rule's prop list.

@export is now specified as a pseudo-prop instead of in front of the rule.

Top rule names are now required.

New features

Rules can now specify an @isGroup pseudo-prop to automatically attach a group name to all the (single) named nodes they produce.

lezer-generator 0.11.2 (2020-09-29)

Bug fixes

Fix a crash that could happen when reporting a conflict error.

New features

A @conflict block inside @tokens can now be used to explicitly indicate a conflict between two tokens.

Allow rules to be explicitly inlineable with an [inline] pseudo-prop.

lezer-tree 0.11.1 (2020-09-26)

Bug fixes

Fix lezer depencency versions

lezer-tree 0.11.0 (2020-09-26)

Breaking changes

Adjust to new output format of repeat rules.

lezer 0.11.2 (2020-09-26)

Bug fixes

Fix lezer depencency versions

lezer 0.11.1 (2020-09-26)

Bug fixes

Fix an infinite loop that was sometimes hit during error recovery.

lezer 0.11.0 (2020-09-26)

Breaking changes

Follow a breaking change in the way repeat nodes are represented.

Support the new action table format that allows sharing between states.

lezer-generator 0.11.1 (2020-09-26)

Bug fixes

Fix lezer depencency versions

lezer-generator 0.11.0 (2020-09-26)

Breaking changes

Simplify the representation of repeat expressions in the grammar in a way that avoids some spurious conflicts.

The output format has been modified to allow states to share part of their action table for better compression.

Bug fixes

Fix a bug where the state collapsing could introduce GLR parsing in grammars that otherwise didn't require it.

lezer 0.10.4 (2020-09-15)

New features

Parser objects now have a withTokenizer method that can be used to replace external tokenizers.

lezer-generator 0.10.5 (2020-09-15)

Bug fixes

Fix a bug where moduleStyle defaulted to "cjs" when using the node API (rather than to "es" as documented).

New features

You can now import "lezer-generator/rollup" to get a rollup plugin that will transform grammars during the build.

lezer-generator 0.10.4 (2020-09-14)

Bug fixes

Fix a bug that broke @external prop declarations in grammars.

lezer-generator 0.10.3 (2020-09-11)

Bug fixes

Make sure unrelated precedence declarations for non-cyclic overlapping tokens don't end up also defining a relation between those tokens.

lezer 0.10.3 (2020-09-10)

Bug fixes

Fix a bug that caused the value returned by ParseContext.badness to be much higher than intended.

lezer 0.10.2 (2020-09-02)

Bug fixes

Stack.ruleStart will now ignore repeat rules and node-less rules when determining the inner rule.

Work around a failure mode where error-recovery got stuck in an end-of-grammar state and thus could not continue meaningfully parsing anything by restarting such states back to their initial state.

New features

External tokenizers can now provide an extend flag to allow their tokens to be used alongside tokens produced by other tokenizers.

Add support for dynamic precedences.

lezer-generator 0.10.2 (2020-09-02)

Bug fixes

Actually reuse parser states when skip rules refer to rules also used in other contexts.

Fix a bug where the automaton generated for skip rules wasn't being compressed.

Properly raise an error when different specializations for the same token are given different names.

Fix a bug that prevented NodeProp.skipped from being properly attached to node types.

Fix a corner-case infinite loop in the state-collapsing algorithm (and speed it up).

Compile + and * operators in a way that is less likely to lead to conflicts.

Emit all shift/reduce and reduce/reduce conflicts in a single run, rather than stopping on the first one.

Emit all overlapping token errors, rather than only the first one.

New features

Inline rules can now be anonymous (with syntax [props..] { body }).

Dynamic precedences can now be associated with productions, which can help pick the preferred GLR parse when a grammar is ambiguous.

Token @precedence declarations can now refer to a parameterized rule by name (without arguments) to indicate that all instances of that rule have a given precedence.

lezer 0.10.1 (2020-08-20)

Bug fixes

Fixes an issue where repeated error recovery could lead to a tree so deep that recursive functions on it would overflow the stack.

lezer-tree 0.10.0 (2020-08-07)

Breaking changes

No longer list internal properties in the type definitions.

lezer 0.10.0 (2020-08-07)

New features

Add support for grammar dialects.

Add support for external specializers.

Stacks now have a parser accessor that gets you the active parser instance.

Breaking changes

No longer list internal properties in the type definitions.

Follow changes in the serialized parser format.

The way different tokenizers are combined is now slightly different. The first one to return a token wins, even if that token has no actions in the current state. The old behavior, where further tokenizers are tried until actions are found, can be enabled for a given tokenizer by setting its fallback flag.

lezer-generator 0.10.1 (2020-08-07)

Bug fixes

Fix an issue where the output file would in some cases have a superfluous comma.

lezer-generator 0.10.0 (2020-08-07)

Bug fixes

Fix a bug in the reuse of compiled rules (which sometimes got confused by inline rules).

The error message for overlapping tokens is a bit more concrete now, including an example of a string that matches both tokens.

New features

Add support for grammar dialects.

Add support for external specializers.

Commas in precedence tag, dialect, or external token lists are now optional.

Breaking changes

Changes the serialized parser format.

lezer-generator 0.9.1 (2020-07-08)

New features

The test runner helper now allows tests to pass additional configuration options.

lezer 0.9.1 (2020-06-29)

Bug fixes

Fix accidental use of non-ES5 library methods.

lezer-tree 0.9.0 (2020-06-08)

Breaking changes

Drop NodeProp.delim in favor of NodeProp.openedBy/closedBy.

lezer 0.9.0 (2020-06-08)

Breaking changes

Upgrade to 0.9 parser serialization

lezer-generator 0.9.0 (2020-06-08)

Breaking changes

The @detectDelim directive now assigns NodeProp.openedBy/closedBy props to the bracket nodes, instead of NodeProp.delim to the parent node.

lezer 0.8.5 (2020-05-01)

Bug fixes

Publish less useless cruft to npm, reducing package size.

lezer-generator 0.8.5 (2020-05-01)

Bug fixes

Publish less useless cruft to npm, reducing package size.

lezer 0.8.4 (2020-04-14)

Bug fixes

Fix a bug in Stack.startOf that made it fail to find rules that were actually on the stack in many situations.

lezer-generator 0.8.4 (2020-04-14)

Bug fixes

Fix an issue where token groups were inappropriately merged when conflicting tokens didn't appear in overlapping state sets.

Fix an issue where external tokenizers needed for skipped tokens were not properly enabled for some states.

Fix a bug where the tool would sometimes overeagerly merge states, resulting in incorrect output.

lezer-generator 0.8.3 (2020-04-09)

Bug fixes

Make dist/test loadable from CommonJS modules again.

Fix a bug that prevented NodeProp.top from being assigned to top rules in most cases.

lezer-tree 0.8.4 (2020-04-01)

Bug fixes

Make the package load as an ES module on node

lezer 0.8.3 (2020-04-01)

Bug fixes

Make the package load as an ES module on node

lezer-generator 0.8.2 (2020-04-01)

Bug fixes

Fix an issue that broke the bin command.

lezer-generator 0.8.1 (2020-04-01)

Bug fixes

Make the package load as an ES module on node

lezer-tree 0.8.3 (2020-02-28)

New features

The package now provides an ES6 module.

lezer 0.8.2 (2020-02-28)

New features

The package now provides an ES6 module.

lezer-tree 0.8.2 (2020-02-26)

Bug fixes

Fix a bug that caused applyChanges to include parts of the old tree that weren't safe to reuse.

lezer 0.8.1 (2020-02-26)

New features

You can now find the top node type parsed by the parser through its topType property.

lezer-tree 0.8.1 (2020-02-14)

Bug fixes

Fix bug that would cause tree balancing of deep trees to produce corrupt output.

lezer-tree 0.8.0 (2020-02-03)

New features

Bump version along with the rest of the lezer packages.

lezer 0.8.0 (2020-02-03)

Breaking changes

The serialized parser format changed.

New features

Add support for multiple @top rules through the top parse option.

lezer-generator 0.8.0 (2020-02-03)

Breaking changes

Changes the serialized parser format.

New features

Add support for multiple @top rules.

lezer-tree 0.7.1 (2020-01-23)

Bug fixes

In applyChanges, make sure the tree is collapsed all the way to the nearest non-error node next to the change.

lezer 0.7.1 (2020-01-23)

Bug fixes

Tweak recovery cost for forced reductions to prefer those to other recovery strategies.

More agressively reuse cached nodes.

lezer-generator 0.7.1 (2020-01-23)

New features

Support Foo(...) syntax in test specs to indicate "ignore the children of this node".

lezer-tree 0.7.0 (2020-01-20)

Bug fixes

Fix a bug that prevented balancing of repeat nodes when there were skipped nodes present between the repeated elements (which ruined the efficiency of incremental parses).

New features

TreeBuffer objects now have an iterate function.

Buffers can optionally be tagged with an (unnamed) node type to allow reusing them in an incremental parse without wrapping them in a tree.

Breaking changes

Tree.build now takes its arguments wrapped in an object. It also expects the buffer content to conform to from lezer 0.7.0's representation of repeated productions.

The repeated node prop was removed (the parser generator now encodes repetition in the type ids).

lezer 0.7.0 (2020-01-20)

Breaking changes

This now consumes the adjusted parser output of lezer-generator 0.7.0.

lezer-generator 0.7.0 (2020-01-20)

New features

You can now write a node name directly after @top to give your top node type a name.

Breaking changes

Changes the way repeated nodes are represented in the generated parser.

lezer 0.6.0 (2020-01-15)

Bug fixes

Rewrite the way the parser advances and recovers from errors, to more rigorously address a number of infinite loops and poor recovery behaviors.

New features

Parse context objects now have a badness property that you can use to estimate just how poorly the input matches the grammar.

lezer-generator 0.5.2 (2020-01-15)

Bug fixes

Fix crash when the top rule had no node name.

Adjust the way states' forced reductions are computed to avoid cycles (where force-reducing multiple times gets you back in your original state).

lezer 0.5.2 (2020-01-09)

Bug fixes

Fix an issue where the parser will sometimes continue, and even pick as result, a parse with error recovery even though there are error-free parses available.

Fix a mistake in our binary heap implementation that would cause stacks to be ordered incorrectly.

Fix an issue where the Stack.startOf method would ignore the top frame of the stack.

lezer 0.5.1 (2020-01-01)

Bug fixes

Fix an issue where the parser would loop infinitely when leaving a nested parse in some circumstances.

Fix an infinite loop on incomplete input at end of file that occurred for some types of mutually recursive rules.

lezer-tree 0.5.1 (2019-10-22)

New features

NodeProp.add now also allows a selector object to be passed.

lezer-tree 0.5.0 (2019-10-22)

New features

Adds NodeProp.top, which flags a grammar's outer node type.

Breaking changes

Drops the NodeProp.lang prop (superseded by top).

lezer 0.5.0 (2019-10-22)

New features

Parser instances now have a hasNested property that tells you whether they nest grammars.

lezer-generator 0.5.1 (2019-10-22)

Bug fixes

Fix issue where serialized parsers didn't have the top node prop set.

lezer-generator 0.5.0 (2019-10-22)

New features

The generator now automatically assigns the top node prop to the grammar's top node.

lezer 0.4.1 (2019-10-14)

Bug fixes

Fix an infinite loop where error recovery keeps finding zero-length tokens and imagining it's making progress.

lezer-tree 0.4.0 (2019-09-10)

Bug fixes

Export BufferCursor again, which was accidentally removed from the exports in 0.3.0.

Breaking changes

The iterate method now takes an object instead of separate parameters.

lezer 0.4.0 (2019-09-10)

Bug fixes

Don't rely on additional data stored in the parse table during recovery (shrinking the parse tables).

Fix a crash that could occur when starting a nested parse when there were multiple active stacks.

Fix an issue where error nodes would sometimes not be merged.

Don't reuse cached tokens for states that have a different token group.

Breaking changes

The on-disk parse table format changed again.

lezer-generator 0.4.0 (2019-09-10)

Bug fixes

Fix bug that made matching single-token skipped expressions unnecessarily expensive.

Breaking changes

Do not emit recovery actions in the parse table anymore.

lezer-tree 0.3.0 (2019-08-22)

New features

Introduces node props.

Node types are now objects holding a name, id, and set of props.

Breaking changes

Tags are gone again, nodes have plain string names.

lezer 0.3.0 (2019-08-22)

Bug fixes

Don't treat reused nodes as if they are error terms when merging errors.

Add badness penalty for forced reductions at end of input.

Fix several infinite loops around forced reductions.

Don't move error nodes out of reduces.

New features

Add a Parser.withProps method for extending a parser with new node props.

Breaking changes

Emits lezer-tree 0.3.0 style trees with NodeType type objects.

Parser.deserialize's interface changed (now taking an object rather than a long list of parameters).

lezer-generator 0.3.0 (2019-08-22)

Bug fixes

Fix several issues in the way forced reductions were picked, to avoid infinite reduction loops.

New features

Add support for props and custom node names in the grammar notation.

Allow importing of props via @external prop.

Rule capitalization is now relevant (again), only capitalized rules appear in the tree by default.

Breaking changes

Remove support for node tags, tag expressions, the @tags block, and everything else related to tags.

Replace tagged expression syntax with inline rule syntax.

Literal declarations must now go into the @tokens block.

The @detectDelim declaration must now appear at the top level.

The dash in @external-... syntax was dropped—@external is now a separate token.

External grammars that default to null must now have the word empty instead of from "..." (to resolve an ambiguity that syntax introduced).

lezer-tree 0.2.0 (2019-08-02)

Bug fixes

Fix incorrect node length calculation in Tree.build.

New features

Tree nodes are now identified with tags.

New Tag data structure to represent node tags.

Breaking changes

Drop support for grammar ids and node types.

lezer 0.2.0 (2019-08-02)

Bug fixes

Don't include lezer-tree inline in dist/index.js.

New features

The output tree now uses tags, rather than term names, to identify its nodes.

Export Tag data structure from lezer-tree.

Support per-grammar global tag suffixes in Parser.deserialize.

Breaking changes

Grammars no longer have ids.

Removes export of allocateGrammarID and TagMap.

lezer-generator 0.2.0 (2019-08-02)

Bug fixes

Fix bug where the grammar parser unintentionally required semicolon between rules in skip blocks.

Actually throw an error when detecing a skip inconsistency.

Track skip context more accurately through parse states.

Fix specializing of external tokens.

New features

Add support for tags.

Add @tags blocks, allow tags for literals.

Add @punctuation to succinctly declare punctuation tags.

Add @infer-delim to enable automatic delimiter detection.

Add @all as a way to append tags to all tagged rules in the grammar.

Allow a choice of literals to be passed to @specialize/@extend.

Add dist/test.js with test helper functions.

Breaking changes

Require @ in front of grammar keywords.

Remove support for =-style tag declarations.

Replace tag.foo syntax with colon suffix syntax.

lezer-tree 0.1.1 (2019-07-09)

Bug Fixes

Actually include the .d.ts file in the published package.

lezer-tree 0.1.0 (2019-07-09)

New Features

First documented release.

lezer 0.1.1 (2019-07-09)

Bug Fixes

Actually include the .d.ts file in the published package.

lezer 0.1.0 (2019-07-09)

New Features

First documented release.

lezer-generator 0.1.1 (2019-07-09)

Bug Fixes

Actually include the .d.ts file in the published package.

lezer-generator 0.1.0 (2019-07-09)

New Features

First documented release.