Version Changelog

lezer 0.13.3 (2021-02-17)

New features

Context trackers can now disable strictness in node reuse.

lezer 0.13.2 (2021-02-17)

New features

Add support for context trackers.

lezer-tree 0.13.2 (2021-02-17)

New features

Add support for context tracking.

lezer-generator 0.13.3 (2021-02-17)

New features

Support @context syntax to register a context tracker for a grammar.

lezer-tree 0.13.1 (2021-02-11)

Bug fixes

Fix a bug where building a tree from a buffer would go wrong for repeat nodes whose children were all repeat nodes of the same type.

lezer-generator 0.13.2 (2021-01-20)

Bug fixes

Fix an issue where imported identifiers could clash with the export name in generated code.

lezer 0.13.1 (2020-12-04)

Bug fixes

Fix versions of lezer packages depended on.

lezer 0.13.0 (2020-12-04)

Breaking changes

Parser.group is now called Parser.nodeSet.

Nested parsers now work differently. They don't have to be Lezer parsers, but have to produce object conforoming to the PartialParse interface. The interface with which non-trivial nested parsers are specified also changed—see the NestedParser type.

Parser objects no longer have a topType property (scan their node set for types with isTop set instead).

Parser objects no longer have withProps, withNested, and withTokenizer methods (use configure instead).

Both Parser.parse and Parser.startParse now take an optional start position as second parameter and an optional parse context as third. startParse returns an instance of the PartialParse interface instead of the old ParseContext class (whose name is now used for something else). Parse options are no longer passed to these methods, but configured in advance through Parser.configure.

During incremental parsing, instead of passing a tree as the cache option, reusable subtrees (see TreeFragment from lezer-tree) are now retrieved from the fragments property of the parse context object, if provided.

Parser.parse and Parser.startParse no longer take an options parameter. Instead, bufferLength and strict can be configured with Parser.configure now, and the start position and context are passed as optional arguments.

The InputStream type has been replaced by Input from the lezer-tree package (which has the same interface but a more appropriate name).

New features

The Parser class now has a configure method that is used to create a parser instance with a more specific configuration.

lezer-tree 0.13.0 (2020-12-04)

Breaking changes

NodeType.isRepeated is now called isAnonymous, which more accurately describes what it means.

NodeGroup has been renamed to NodeSet to avoid confusion with NodeProp.group.

The applyChanges method on trees is no longer supported (TreeFragment is now used to track reusable content).

Trees no longer have cut and append methods.

New features

It is now possible to pass a node ID to SyntaxNode.getChild/getChildren and NodeType.is. Allow specifying a tree length in Tree.build

Tree.build now allows you to specify the length of the resulting tree.

Tree.fullCursor() can now be used to get a cursor that includes anonymous nodes, rather than skipping them.

Introduces NodeType.define to define node types.

The new TreeFragment type is used to manage reusable subtrees for incremental parsing.

Tree.build now accepts a start option indicating the start offset of the tree.

The Input type, which used to be InputStream in the lezer package, is now exported from this package.

This package now exports a PartialParse interface, which describes the interface used, for example, as return type from Parser.startParse.

lezer-generator 0.13.1 (2020-12-04)

Bug fixes

Fix versions of lezer packages depended on.

lezer-generator 0.13.0 (2020-12-04)

Breaking changes

Adjust to the new way nested parsers work in Lezer.

Bug fixes

Top rule node types will now show up in the terms file.

It is no longer allowed for a top rule to share a name with another rule.

lezer 0.12.1 (2020-11-19)

Bug fixes

Fix an infinite loop in incremental parsing when repeatedly reusing a zero-length cached node.

lezer-tree 0.12.3 (2020-11-02)

New features

Make NodePropSource a function type.

lezer-tree 0.12.2 (2020-10-28)

Bug fixes

Fix a bug that made SyntaxNode.prevSibling fail in most cases when the node is part of a buffer.

lezer-tree 0.12.1 (2020-10-26)

Bug fixes

Fix issue where using Tree.append with an empty tree as argument would return a tree with a nonsensical length property.

lezer 0.12.0 (2020-10-23)

Breaking changes

Follow the change from Subtree to TreeCursor in lezer-tree.

The serialized parser format changed.

Stack.startOf now returns null, rather than -1, when it can't find the given element.

New features

Stack.startOf now takes an optional second argument that allows you to select a match beyond the innermost one.

lezer-tree 0.12.0 (2020-10-23)

Breaking changes

Tree.iterate no longer allows returning from inside the iteration (use cursors directly for that kind of use cases).

Subtree has been renamed to SyntaxNode and narrowed in scope a little.

The top, skipped, and error node props no longer exist.

New features

The package now offers a TreeCursor abstraction, which can be used for both regular iteration and for custom traversal of a tree.

SyntaxNode instances have nextSibling/prevSibling getters that allow more direct navigation through the tree.

Node types now expose isTop, isSkipped, isError, and isRepeated properties that indicate special status.

Adds NodeProp.group to assign group names to node types.

Syntax nodes now have helper functions getChild and getChildren to retrieve direct child nodes by type or group.

NodeType.match (and thus NodeProp.add) now allows types to be targeted by group name.

Node types have a new is method for checking whether their name or one of their groups matches a given string.

lezer-generator 0.12.0 (2020-10-23)

Breaking changes

The serialized parser format changed.

Pseudo-props like name, dialect, inline and dynamicPrec now require an @ in front of them when specified in a rule's prop list.

@export is now specified as a pseudo-prop instead of in front of the rule.

Top rule names are now required.

New features

Rules can now specify an @isGroup pseudo-prop to automatically attach a group name to all the (single) named nodes they produce.

lezer-generator 0.11.2 (2020-09-29)

Bug fixes

Fix a crash that could happen when reporting a conflict error.

New features

A @conflict block inside @tokens can now be used to explicitly indicate a conflict between two tokens.

Allow rules to be explicitly inlineable with an [inline] pseudo-prop.

lezer 0.11.2 (2020-09-26)

Bug fixes

Fix lezer depencency versions

lezer 0.11.1 (2020-09-26)

Bug fixes

Fix an infinite loop that was sometimes hit during error recovery.

lezer 0.11.0 (2020-09-26)

Breaking changes

Follow a breaking change in the way repeat nodes are represented.

Support the new action table format that allows sharing between states.

lezer-tree 0.11.1 (2020-09-26)

Bug fixes

Fix lezer depencency versions

lezer-tree 0.11.0 (2020-09-26)

Breaking changes

Adjust to new output format of repeat rules.

lezer-generator 0.11.1 (2020-09-26)

Bug fixes

Fix lezer depencency versions

lezer-generator 0.11.0 (2020-09-26)

Breaking changes

Simplify the representation of repeat expressions in the grammar in a way that avoids some spurious conflicts.

The output format has been modified to allow states to share part of their action table for better compression.

Bug fixes

Fix a bug where the state collapsing could introduce GLR parsing in grammars that otherwise didn't require it.

lezer 0.10.4 (2020-09-15)

New features

Parser objects now have a withTokenizer method that can be used to replace external tokenizers.

lezer-generator 0.10.5 (2020-09-15)

Bug fixes

Fix a bug where moduleStyle defaulted to "cjs" when using the node API (rather than to "es" as documented).

New features

You can now import "lezer-generator/rollup" to get a rollup plugin that will transform grammars during the build.

lezer-generator 0.10.4 (2020-09-14)

Bug fixes

Fix a bug that broke @external prop declarations in grammars.

lezer-generator 0.10.3 (2020-09-11)

Bug fixes

Make sure unrelated precedence declarations for non-cyclic overlapping tokens don't end up also defining a relation between those tokens.

lezer 0.10.3 (2020-09-10)

Bug fixes

Fix a bug that caused the value returned by ParseContext.badness to be much higher than intended.

lezer 0.10.2 (2020-09-02)

Bug fixes

Stack.ruleStart will now ignore repeat rules and node-less rules when determining the inner rule.

Work around a failure mode where error-recovery got stuck in an end-of-grammar state and thus could not continue meaningfully parsing anything by restarting such states back to their initial state.

New features

External tokenizers can now provide an extend flag to allow their tokens to be used alongside tokens produced by other tokenizers.

Add support for dynamic precedences.

lezer-generator 0.10.2 (2020-09-02)

Bug fixes

Actually reuse parser states when skip rules refer to rules also used in other contexts.

Fix a bug where the automaton generated for skip rules wasn't being compressed.

Properly raise an error when different specializations for the same token are given different names.

Fix a bug that prevented NodeProp.skipped from being properly attached to node types.

Fix a corner-case infinite loop in the state-collapsing algorithm (and speed it up).

Compile + and * operators in a way that is less likely to lead to conflicts.

Emit all shift/reduce and reduce/reduce conflicts in a single run, rather than stopping on the first one.

Emit all overlapping token errors, rather than only the first one.

New features

Inline rules can now be anonymous (with syntax [props..] { body }).

Dynamic precedences can now be associated with productions, which can help pick the preferred GLR parse when a grammar is ambiguous.

Token @precedence declarations can now refer to a parameterized rule by name (without arguments) to indicate that all instances of that rule have a given precedence.

lezer 0.10.1 (2020-08-20)

Bug fixes

Fixes an issue where repeated error recovery could lead to a tree so deep that recursive functions on it would overflow the stack.

lezer 0.10.0 (2020-08-07)

New features

Add support for grammar dialects.

Add support for external specializers.

Stacks now have a parser accessor that gets you the active parser instance.

Breaking changes

No longer list internal properties in the type definitions.

Follow changes in the serialized parser format.

The way different tokenizers are combined is now slightly different. The first one to return a token wins, even if that token has no actions in the current state. The old behavior, where further tokenizers are tried until actions are found, can be enabled for a given tokenizer by setting its fallback flag.

lezer-tree 0.10.0 (2020-08-07)

Breaking changes

No longer list internal properties in the type definitions.

lezer-generator 0.10.1 (2020-08-07)

Bug fixes

Fix an issue where the output file would in some cases have a superfluous comma.

lezer-generator 0.10.0 (2020-08-07)

Bug fixes

Fix a bug in the reuse of compiled rules (which sometimes got confused by inline rules).

The error message for overlapping tokens is a bit more concrete now, including an example of a string that matches both tokens.

New features

Add support for grammar dialects.

Add support for external specializers.

Commas in precedence tag, dialect, or external token lists are now optional.

Breaking changes

Changes the serialized parser format.

lezer-generator 0.9.1 (2020-07-08)

New features

The test runner helper now allows tests to pass additional configuration options.

lezer 0.9.1 (2020-06-29)

Bug fixes

Fix accidental use of non-ES5 library methods.

lezer 0.9.0 (2020-06-08)

Breaking changes

Upgrade to 0.9 parser serialization

lezer-tree 0.9.0 (2020-06-08)

Breaking changes

Drop NodeProp.delim in favor of NodeProp.openedBy/closedBy.

lezer-generator 0.9.0 (2020-06-08)

Breaking changes

The @detectDelim directive now assigns NodeProp.openedBy/closedBy props to the bracket nodes, instead of NodeProp.delim to the parent node.

lezer 0.8.5 (2020-05-01)

Bug fixes

Publish less useless cruft to npm, reducing package size.

lezer-generator 0.8.5 (2020-05-01)

Bug fixes

Publish less useless cruft to npm, reducing package size.

lezer 0.8.4 (2020-04-14)

Bug fixes

Fix a bug in Stack.startOf that made it fail to find rules that were actually on the stack in many situations.

lezer-generator 0.8.4 (2020-04-14)

Bug fixes

Fix an issue where token groups were inappropriately merged when conflicting tokens didn't appear in overlapping state sets.

Fix an issue where external tokenizers needed for skipped tokens were not properly enabled for some states.

Fix a bug where the tool would sometimes overeagerly merge states, resulting in incorrect output.

lezer-generator 0.8.3 (2020-04-09)

Bug fixes

Make dist/test loadable from CommonJS modules again.

Fix a bug that prevented NodeProp.top from being assigned to top rules in most cases.

lezer 0.8.3 (2020-04-01)

Bug fixes

Make the package load as an ES module on node

lezer-tree 0.8.4 (2020-04-01)

Bug fixes

Make the package load as an ES module on node

lezer-generator 0.8.2 (2020-04-01)

Bug fixes

Fix an issue that broke the bin command.

lezer-generator 0.8.1 (2020-04-01)

Bug fixes

Make the package load as an ES module on node

lezer 0.8.2 (2020-02-28)

New features

The package now provides an ES6 module.

lezer-tree 0.8.3 (2020-02-28)

New features

The package now provides an ES6 module.

lezer 0.8.1 (2020-02-26)

New features

You can now find the top node type parsed by the parser through its topType property.

lezer-tree 0.8.2 (2020-02-26)

Bug fixes

Fix a bug that caused applyChanges to include parts of the old tree that weren't safe to reuse.

lezer-tree 0.8.1 (2020-02-14)

Bug fixes

Fix bug that would cause tree balancing of deep trees to produce corrupt output.

lezer 0.8.0 (2020-02-03)

Breaking changes

The serialized parser format changed.

New features

Add support for multiple @top rules through the top parse option.

lezer-tree 0.8.0 (2020-02-03)

New features

Bump version along with the rest of the lezer packages.

lezer-generator 0.8.0 (2020-02-03)

Breaking changes

Changes the serialized parser format.

New features

Add support for multiple @top rules.

lezer 0.7.1 (2020-01-23)

Bug fixes

Tweak recovery cost for forced reductions to prefer those to other recovery strategies.

More agressively reuse cached nodes.

lezer-tree 0.7.1 (2020-01-23)

Bug fixes

In applyChanges, make sure the tree is collapsed all the way to the nearest non-error node next to the change.

lezer-generator 0.7.1 (2020-01-23)

New features

Support Foo(...) syntax in test specs to indicate "ignore the children of this node".

lezer 0.7.0 (2020-01-20)

Breaking changes

This now consumes the adjusted parser output of lezer-generator 0.7.0.

lezer-tree 0.7.0 (2020-01-20)

Bug fixes

Fix a bug that prevented balancing of repeat nodes when there were skipped nodes present between the repeated elements (which ruined the efficiency of incremental parses).

New features

TreeBuffer objects now have an iterate function.

Buffers can optionally be tagged with an (unnamed) node type to allow reusing them in an incremental parse without wrapping them in a tree.

Breaking changes

Tree.build now takes its arguments wrapped in an object. It also expects the buffer content to conform to from lezer 0.7.0's representation of repeated productions.

The repeated node prop was removed (the parser generator now encodes repetition in the type ids).

lezer-generator 0.7.0 (2020-01-20)

New features

You can now write a node name directly after @top to give your top node type a name.

Breaking changes

Changes the way repeated nodes are represented in the generated parser.

lezer 0.6.0 (2020-01-15)

Bug fixes

Rewrite the way the parser advances and recovers from errors, to more rigorously address a number of infinite loops and poor recovery behaviors.

New features

Parse context objects now have a badness property that you can use to estimate just how poorly the input matches the grammar.

lezer-generator 0.5.2 (2020-01-15)

Bug fixes

Fix crash when the top rule had no node name.

Adjust the way states' forced reductions are computed to avoid cycles (where force-reducing multiple times gets you back in your original state).

lezer 0.5.2 (2020-01-09)

Bug fixes

Fix an issue where the parser will sometimes continue, and even pick as result, a parse with error recovery even though there are error-free parses available.

Fix a mistake in our binary heap implementation that would cause stacks to be ordered incorrectly.

Fix an issue where the Stack.startOf method would ignore the top frame of the stack.

lezer 0.5.1 (2020-01-01)

Bug fixes

Fix an issue where the parser would loop infinitely when leaving a nested parse in some circumstances.

Fix an infinite loop on incomplete input at end of file that occurred for some types of mutually recursive rules.

lezer 0.5.0 (2019-10-22)

New features

Parser instances now have a hasNested property that tells you whether they nest grammars.

lezer-tree 0.5.1 (2019-10-22)

New features

NodeProp.add now also allows a selector object to be passed.

lezer-tree 0.5.0 (2019-10-22)

New features

Adds NodeProp.top, which flags a grammar's outer node type.

Breaking changes

Drops the NodeProp.lang prop (superseded by top).

lezer-generator 0.5.1 (2019-10-22)

Bug fixes

Fix issue where serialized parsers didn't have the top node prop set.

lezer-generator 0.5.0 (2019-10-22)

New features

The generator now automatically assigns the top node prop to the grammar's top node.

lezer 0.4.1 (2019-10-14)

Bug fixes

Fix an infinite loop where error recovery keeps finding zero-length tokens and imagining it's making progress.

lezer 0.4.0 (2019-09-10)

Bug fixes

Don't rely on additional data stored in the parse table during recovery (shrinking the parse tables).

Fix a crash that could occur when starting a nested parse when there were multiple active stacks.

Fix an issue where error nodes would sometimes not be merged.

Don't reuse cached tokens for states that have a different token group.

Breaking changes

The on-disk parse table format changed again.

lezer-tree 0.4.0 (2019-09-10)

Bug fixes

Export BufferCursor again, which was accidentally removed from the exports in 0.3.0.

Breaking changes

The iterate method now takes an object instead of separate parameters.

lezer-generator 0.4.0 (2019-09-10)

Bug fixes

Fix bug that made matching single-token skipped expressions unnecessarily expensive.

Breaking changes

Do not emit recovery actions in the parse table anymore.

lezer 0.3.0 (2019-08-22)

Bug fixes

Don't treat reused nodes as if they are error terms when merging errors.

Add badness penalty for forced reductions at end of input.

Fix several infinite loops around forced reductions.

Don't move error nodes out of reduces.

New features

Add a Parser.withProps method for extending a parser with new node props.

Breaking changes

Emits lezer-tree 0.3.0 style trees with NodeType type objects.

Parser.deserialize's interface changed (now taking an object rather than a long list of parameters).

lezer-tree 0.3.0 (2019-08-22)

New features

Introduces node props.

Node types are now objects holding a name, id, and set of props.

Breaking changes

Tags are gone again, nodes have plain string names.

lezer-generator 0.3.0 (2019-08-22)

Bug fixes

Fix several issues in the way forced reductions were picked, to avoid infinite reduction loops.

New features

Add support for props and custom node names in the grammar notation.

Allow importing of props via @external prop.

Rule capitalization is now relevant (again), only capitalized rules appear in the tree by default.

Breaking changes

Remove support for node tags, tag expressions, the @tags block, and everything else related to tags.

Replace tagged expression syntax with inline rule syntax.

Literal declarations must now go into the @tokens block.

The @detectDelim declaration must now appear at the top level.

The dash in @external-... syntax was dropped—@external is now a separate token.

External grammars that default to null must now have the word empty instead of from "..." (to resolve an ambiguity that syntax introduced).

lezer 0.2.0 (2019-08-02)

Bug fixes

Don't include lezer-tree inline in dist/index.js.

New features

The output tree now uses tags, rather than term names, to identify its nodes.

Export Tag data structure from lezer-tree.

Support per-grammar global tag suffixes in Parser.deserialize.

Breaking changes

Grammars no longer have ids.

Removes export of allocateGrammarID and TagMap.

lezer-tree 0.2.0 (2019-08-02)

Bug fixes

Fix incorrect node length calculation in Tree.build.

New features

Tree nodes are now identified with tags.

New Tag data structure to represent node tags.

Breaking changes

Drop support for grammar ids and node types.

lezer-generator 0.2.0 (2019-08-02)

Bug fixes

Fix bug where the grammar parser unintentionally required semicolon between rules in skip blocks.

Actually throw an error when detecing a skip inconsistency.

Track skip context more accurately through parse states.

Fix specializing of external tokens.

New features

Add support for tags.

Add @tags blocks, allow tags for literals.

Add @punctuation to succinctly declare punctuation tags.

Add @infer-delim to enable automatic delimiter detection.

Add @all as a way to append tags to all tagged rules in the grammar.

Allow a choice of literals to be passed to @specialize/@extend.

Add dist/test.js with test helper functions.

Breaking changes

Require @ in front of grammar keywords.

Remove support for =-style tag declarations.

Replace tag.foo syntax with colon suffix syntax.

lezer 0.1.1 (2019-07-09)

Bug Fixes

Actually include the .d.ts file in the published package.

lezer-tree 0.1.1 (2019-07-09)

Bug Fixes

Actually include the .d.ts file in the published package.

lezer-generator 0.1.1 (2019-07-09)

Bug Fixes

Actually include the .d.ts file in the published package.