Port @eslint-community/regexpp

[togami2864]

There's some ECMA regex related rules like no-misleading-character-class, no-useless-escape.

ESLint leverages @eslint-community/regexpp, ecma script regex parer to implement these rules, while Biome lacks a dedicated regex parser, leading to the implementation of simple regex parsers within individual rules to manage diverse regex inputs.

e.g.

• biome/crates/biome_js_analyze/src/semantic_analyzers/nursery/no_misleading_character_class.rs

  Lines 291 to 618 in bc3bd4e

  	fn diagnostic_regex_pattern(
  	regex_pattern: &str,
  	has_u_flag: bool,
  	range: TextRange,
  	) -> Option<RuleState> {
  	let mut is_in_character_class = false;
  	let mut escape_next = false;
  	let char_iter = regex_pattern.chars().peekable();
  	for (i, ch) in char_iter.enumerate() {
  	if escape_next {
  	escape_next = false;
  	continue;
  	}
  	match ch {
  	'\\' => escape_next = true,
  	'[' => is_in_character_class = true,
  	']' => is_in_character_class = false,
  	_ if is_in_character_class && i < regex_pattern.len() => {
  	if !has_u_flag && has_surrogate_pair(&regex_pattern[i..]) {
  	return Some(RuleState {
  	range,
  	message: Message::SurrogatePairWithoutUFlag,
  	});
  	}
  	if has_combining_class_or_vs16(&regex_pattern[i..]) {
  	return Some(RuleState {
  	range,
  	message: Message::CombiningClassOrVs16,
  	});
  	}
  	if has_regional_indicator_symbol(&regex_pattern[i..]) {
  	return Some(RuleState {
  	range,
  	message: Message::RegionalIndicatorSymbol,
  	});
  	}
  	if has_emoji_modifier(&regex_pattern[i..]) {
  	return Some(RuleState {
  	range,
  	message: Message::EmojiModifier,
  	});
  	}
  	if zwj(&regex_pattern[i..]) {
  	return Some(RuleState {
  	range,
  	message: Message::JoinedCharSequence,
  	});
  	}
  	}
  	_ => {}
  	}
  	}
  	None
  	}
  	fn make_suggestion(
  	literal: AnyJsCallArgument,
  	flag: Option<AnyJsCallArgument>,
  	) -> Option<JsCallArguments> {
  	let suggestion = match flag {
  	None => Some(AnyJsCallArgument::AnyJsExpression(
  	AnyJsExpression::AnyJsLiteralExpression(
  	AnyJsLiteralExpression::JsStringLiteralExpression(
  	make::js_string_literal_expression(make::js_string_literal("u")),
  	),
  	),
  	)),
  	Some(f) => match f {
  	AnyJsCallArgument::AnyJsExpression(expr) => match expr {
  	AnyJsExpression::AnyJsLiteralExpression(e) => {
  	let text = e.text();
  	if text.starts_with('\'') {
  	Some(AnyJsCallArgument::AnyJsExpression(
  	AnyJsExpression::AnyJsLiteralExpression(
  	AnyJsLiteralExpression::JsStringLiteralExpression(
  	make::js_string_literal_expression(make::js_string_literal(
  	&format!("'{}u'", text),
  	)),
  	),
  	),
  	))
  	} else {
  	Some(AnyJsCallArgument::AnyJsExpression(
  	AnyJsExpression::AnyJsLiteralExpression(
  	AnyJsLiteralExpression::JsStringLiteralExpression(
  	make::js_string_literal_expression(make::js_string_literal(
  	&format!("{}u", text.replace('"', "")),
  	)),
  	),
  	),
  	))
  	}
  	}
  	AnyJsExpression::JsTemplateExpression(expr) => {
  	let mut elements = expr
  	.elements()
  	.iter()
  	.collect::<Vec<AnyJsTemplateElement>>();
  	let uflag = AnyJsTemplateElement::from(make::js_template_chunk_element(
  	make::js_template_chunk("u"),
  	));
  	elements.push(uflag);
  	Some(AnyJsCallArgument::AnyJsExpression(
  	AnyJsExpression::JsTemplateExpression(
  	make::js_template_expression(
  	make::token(T!['`']),
  	make::js_template_element_list(elements),
  	make::token(T!['`']),
  	)
  	.build(),
  	),
  	))
  	}
  	AnyJsExpression::JsIdentifierExpression(_) => None,
  	_ => None,
  	},
  	AnyJsCallArgument::JsSpread(_) => None,
  	},
  	};
  	suggestion.map(|s| {
  	make::js_call_arguments(
  	make::token(T!['(']),
  	make::js_call_argument_list(
  	[literal, s],
  	Some(make::token(T![,]).with_trailing_trivia([(TriviaPieceKind::Whitespace, " ")])),
  	),
  	make::token(T![')']),
  	)
  	})
  	}
  	fn is_emoji_modifier(code: u32) -> bool {
  	(0x1F3FB..=0x1F3FF).contains(&code)
  	}
  	fn has_emoji_modifier(chars: &str) -> bool {
  	let char_vec: Vec<char> = chars.chars().collect();
  	char_vec.iter().enumerate().any(|(i, &c)| {
  	i != 0 && is_emoji_modifier(c as u32) && !is_emoji_modifier(char_vec[i - 1] as u32)
  	})
  	}
  	fn is_regional_indicator_symbol(code: u32) -> bool {
  	(0x1F1E6..=0x1F1FF).contains(&code)
  	}
  	fn has_regional_indicator_symbol(chars: &str) -> bool {
  	let char_vec: Vec<char> = chars.chars().collect();
  	char_vec.iter().enumerate().any(|(i, &c)| {
  	i != 0
  	&& is_regional_indicator_symbol(c as u32)
  	&& is_regional_indicator_symbol(char_vec[i - 1] as u32)
  	})
  	}
  	fn is_combining_character(ch: char) -> bool {
  	match ch {
  	'\u{0300}'..='\u{036F}' | // Combining Diacritical Marks
  	'\u{1AB0}'..='\u{1AFF}' | // Combining Diacritical Marks Extended
  	'\u{1DC0}'..='\u{1DFF}' | // Combining Diacritical Marks Supplement
  	'\u{20D0}'..='\u{20FF}' | // Combining Diacritical Marks for Symbols
  	'\u{FE20}'..='\u{FE2F}' // Combining Half Marks
  	=> true,
  	_ => false
  	}
  	}
  	fn is_variation_selector_16(ch: char) -> bool {
  	('\u{FE00}'..='\u{FE0F}').contains(&ch)
  	}
  	fn has_combining_class_or_vs16(chars: &str) -> bool {
  	chars.chars().enumerate().any(|(i, c)| {
  	i != 0
  	&& (is_combining_character(c) || is_variation_selector_16(c))
  	&& !(is_combining_character(chars.chars().nth(i - 1).unwrap())
  	|| is_variation_selector_16(chars.chars().nth(i - 1).unwrap()))
  	})
  	}
  	fn zwj(chars: &str) -> bool {
  	let char_vec: Vec<char> = chars.chars().collect();
  	let last_index = char_vec.len() - 1;
  	char_vec.iter().enumerate().any(|(i, &c)| {
  	i != 0
  	&& i != last_index
  	&& c as u32 == 0x200D
  	&& char_vec[i - 1] as u32 != 0x200D
  	&& char_vec[i + 1] as u32 != 0x200D
  	})
  	}
  	fn has_surrogate_pair(s: &str) -> bool {
  	s.chars().any(|c| c as u32 > 0xFFFF)
  	}
  	/// Convert unicode escape sequence string to unicode character
  	/// - unicode escape sequences: \u{XXXX}
  	/// - unicode escape sequences without parenthesis: \uXXXX
  	/// - surrogate pair: \uXXXX\uXXXX
  	/// If the unicode escape sequence is not valid, it will be treated as a simple string.
  	///
  	/// ```example
  	/// \uD83D\uDC4D -> 👍
  	/// \u0041\u0301 -> Á
  	/// \uD83D\uDC68\u200D\uD83D\uDC69\u200D\uD83D\uDC66 -> 👨‍👩‍👦
  	/// \u{1F468}\u{200D}\u{1F469}\u{200D}\u{1F466} -> 👨‍👩‍👦
  	/// \u899\uD83D\uDC4D -> \u899👍
  	/// ````
  	fn replace_escaped_unicode(input: &str) -> String {
  	let mut result = String::new();
  	let mut chars_iter = input.chars().peekable();
  	while let Some(ch) = chars_iter.next() {
  	if ch == '\\' {
  	match handle_escape_sequence(&mut chars_iter) {
  	Some(unicode_char) => result.push_str(&unicode_char),
  	None => result.push(ch),
  	}
  	} else {
  	result.push(ch);
  	}
  	}
  	result
  	}
  	fn handle_escape_sequence(chars_iter: &mut std::iter::Peekable<std::str::Chars>) -> Option<String> {
  	if chars_iter.peek() != Some(&'u') {
  	return None;
  	}
  	chars_iter.next();
  	if chars_iter.peek() == Some(&'{') {
  	handle_braced_escape_sequence(chars_iter)
  	} else {
  	handle_simple_or_surrogate_escape_sequence(chars_iter)
  	}
  	}
  	fn handle_braced_escape_sequence(
  	chars_iter: &mut std::iter::Peekable<std::str::Chars>,
  	) -> Option<String> {
  	chars_iter.next();
  	let mut codepoint_str = String::new();
  	while let Some(&next_char) = chars_iter.peek() {
  	if next_char == '}' {
  	chars_iter.next();
  	break;
  	} else {
  	codepoint_str.push(next_char);
  	chars_iter.next();
  	}
  	}
  	u32::from_str_radix(&codepoint_str, 16)
  	.ok()
  	.and_then(char::from_u32)
  	.map(|c| c.to_string())
  	}
  	fn handle_simple_or_surrogate_escape_sequence(
  	chars_iter: &mut std::iter::Peekable<std::str::Chars>,
  	) -> Option<String> {
  	let mut invalid_pair = String::new();
  	let mut high_surrogate_str = String::new();
  	for _ in 0..4 {
  	if let Some(&next_char) = chars_iter.peek() {
  	if next_char.is_ascii_hexdigit() {
  	high_surrogate_str.push(next_char);
  	chars_iter.next();
  	} else {
  	// If the character is not a valid Unicode char, return as simple string.
  	return Some(format!("\\u{}", high_surrogate_str));
  	}
  	} else {
  	// If not enough characters, return as if it were a simple string.
  	return Some(format!("\\u{}", high_surrogate_str));
  	}
  	}
  	if let Ok(high_surrogate) = u32::from_str_radix(&high_surrogate_str, 16) {
  	// Check if it is in the high surrogate range(0xD800-0xDBFF) in UTF-16.
  	if (0xD800..=0xDBFF).contains(&high_surrogate) {
  	// If we have a high surrogate, expect a low surrogate next
  	if chars_iter.next() == Some('\\') && chars_iter.next() == Some('u') {
  	let mut low_surrogate_str = String::new();
  	for _ in 0..4 {
  	if let Some(next_char) = chars_iter.peek() {
  	if !next_char.is_ascii_hexdigit() {
  	// Return as a simple string
  	// - high surrogate on its own doesn't make sense
  	// - low surrogate is not a valid unicode codepoint
  	// e.g \uD83D\u333
  	invalid_pair.push_str(&format!("\\u{}", high_surrogate_str));
  	invalid_pair.push_str(&format!("\\u{}", low_surrogate_str));
  	return Some(invalid_pair);
  	}
  	low_surrogate_str.push(*next_char);
  	chars_iter.next();
  	}
  	}
  	if let Ok(low_surrogate) = u32::from_str_radix(&low_surrogate_str, 16) {
  	// Check if it is in the low surrogate range(0xDC00-0xDFFF) in UTF-16.
  	if (0xDC00..=0xDFFF).contains(&low_surrogate) {
  	// Calculate the codepoint from the surrogate pair
  	let codepoint =
  	((high_surrogate - 0xD800) << 10) + (low_surrogate - 0xDC00) + 0x10000;
  	return char::from_u32(codepoint).map(|c| c.to_string());
  	};
  	}
  	}
  	} else {
  	match char::from_u32(high_surrogate) {
  	Some(c) => return Some(c.to_string()),
  	None => invalid_pair.push_str(&format!("\\u{}", high_surrogate_str)),
  	}
  	}
  	}
  	Some(invalid_pair)
  	}

• biome/crates/biome_js_analyze/src/analyzers/suspicious/no_control_characters_in_regex.rs

  Lines 73 to 198 in 593da22

  	fn decode_hex_character_to_code_point(iter: &mut Peekable<Chars>) -> Option<(String, i64)> {
  	let first = iter.next()?;
  	let second = iter.next()?;
  	let digits = format!("{first}{second}");
  	let code_point = i64::from_str_radix(&digits, 16).ok()?;
  	Some((digits, code_point))
  	}
  	fn decode_unicode_escape_to_code_point(iter: &mut Peekable<Chars>) -> Option<(String, i64)> {
  	let mut digits = String::new();
  	// Loop 4 times as unicode escape sequence has exactly 4 hexadecimal digits
  	for _ in 0..4 {
  	if let Some(&c) = iter.peek() {
  	match c {
  	'0'..='9' | 'a'..='f' | 'A'..='F' => digits.push(iter.next()?),
  	_ => continue,
  	}
  	}
  	}
  	let code_point = i64::from_str_radix(digits.as_str(), 16).ok()?;
  	Some((digits, code_point))
  	}
  	fn decode_escaped_code_point_to_code_point(iter: &mut Peekable<Chars>) -> Option<(String, i64)> {
  	let mut digits = String::new();
  	if iter.peek() == Some(&'{') {
  	iter.next();
  	while let Some(&c) = iter.peek() {
  	if c == '}' {
  	iter.next();
  	let code_point = i64::from_str_radix(&digits, 16).ok()?;
  	return Some((format!("{{{}}}", digits), code_point));
  	} else {
  	digits.push(iter.next()?);
  	}
  	}
  	}
  	None
  	}
  	fn add_control_character_to_vec(
  	prefix: &str,
  	iter: &mut Peekable<Chars>,
  	control_characters: &mut Vec<String>,
  	decode: fn(&mut Peekable<Chars>) -> Option<(String, i64)>,
  	) {
  	if let Some((s, code_point)) = decode(iter) {
  	// ASCII control characters are represented by code points from 0 to 31
  	if (0..=31).contains(&code_point) {
  	control_characters.push(format!("{prefix}{s}"));
  	}
  	}
  	}
  	/// Collecting control characters for regex. The following characters in regular expression patterns are considered as control characters:
  	/// - Hexadecimal character escapes from `\x00` to `\x1F`.
  	/// - Unicode character escapes from `\u0000` to `\u001F`.
  	/// - Unicode code point escapes range from `\u{0}` to `\u{1F}`.
  	/// - The Unicode flag must be set as true in order for these Unicode code point escapes to work: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/unicode.
  	/// - Unescaped raw characters from U+0000 to U+001F.
  	fn collect_control_characters(pattern: &str, flags: &str) -> Option<Vec<String>> {
  	let mut control_characters: Vec<String> = Vec::new();
  	let is_unicode_flag_set = flags.contains('u');
  	let mut iter = pattern.chars().peekable();
  	while let Some(c) = iter.next() {
  	match c {
  	'\\' => match iter.next() {
  	Some('x') => add_control_character_to_vec(
  	"\\x",
  	&mut iter,
  	&mut control_characters,
  	decode_hex_character_to_code_point,
  	),
  	Some('u') if is_unicode_flag_set => add_control_character_to_vec(
  	"\\u",
  	&mut iter,
  	&mut control_characters,
  	decode_escaped_code_point_to_code_point,
  	),
  	Some('u') => add_control_character_to_vec(
  	"\\u",
  	&mut iter,
  	&mut control_characters,
  	decode_unicode_escape_to_code_point,
  	),
  	Some('\\') => continue,
  	_ => break,
  	},
  	_ => continue,
  	}
  	}
  	if !control_characters.is_empty() {
  	Some(control_characters)
  	} else {
  	None
  	}
  	}
  	fn collect_control_characters_from_expression(
  	callee: &AnyJsExpression,
  	js_call_arguments: &JsCallArguments,
  	) -> Option<Vec<String>> {
  	if callee.as_js_reference_identifier()?.has_name("RegExp") {
  	let mut args = js_call_arguments.args().iter();
  	let raw_pattern = args
  	.next()
  	.and_then(|arg| arg.ok())
  	.and_then(|arg| JsStringLiteralExpression::cast_ref(arg.syntax()))
  	.and_then(|js_string_literal| js_string_literal.inner_string_text().ok())?
  	.to_string();
  	let pattern = escape_string(&raw_pattern).unwrap_or(raw_pattern);
  	let regexp_flags = args
  	.next()
  	.and_then(|arg| arg.ok())
  	.and_then(|arg| JsStringLiteralExpression::cast_ref(arg.syntax()))
  	.map(|js_string_literal| js_string_literal.text())
  	.unwrap_or_default();
  	return collect_control_characters(&pattern, &regexp_flags);
  	}
  	None
  	}

And some bugs have been reported.

• 💅 Linter fail on code files contains literal RegExp contains Cyrillic symbols #1522 (comment)
• 💅 Linter fail on code files contains literal RegExp contains Cyrillic symbols #1522 (comment)

Consequently, I propose porting @eslint-community/regexpp into biome. An additional note is made of the absence of an ECMA regex parser in Rust on crates.io.

pros:

• It will be able to provide more accurate lint
• It may prevent duplication of implementation

cons:

• The costs may be high, while the use case remains limited.

This is just an idea. What do you think?

[Conaclos]

I think it is a good idea! Do you want to implement this parser?

[togami2864]

Sure. I'll try implement small PoC.

[Conaclos]

OXC is also adding a regex parser. Could be worth to take a look at :)

[ematipico]

A regex parser is a massive endeavour. Do we have the resources? Can't we use the regex crate at this point?

[Conaclos]

Is this compatible with js regexes?

[togami2864]

It seems that some syntax isn't supported.
https://docs.rs/regex/1.10.2/regex/#syntax
e.g.

• lookahead
• unicode property escape
• lookbehind assertion

[Conaclos]

We could also use the oxc crate if it got merged.

[togami2864]

That sounds good too. Let's wait and see for a little while.