Files
craft-libs/crates/html/src/lex/tests.rs
T
Dylan M. Taylorandshuff57 db616480e2 craft-html: HTML documents on craft-flow (feature flow), from WordCraft (#1050)
* Add craft-wordmodel: WordCraft's document model without its editing code

WordCraft's crates/doc (wordcraft-doc, at 07f97737) moved as it is, so the DOCX,
RTF, ODT and other text-format crates can read into and write from one model.
Module paths, type names and serde attributes are unchanged: an app that
re-exports the crate keeps its native file format.

Left in the app: range editing (edit.rs), IME text and equation editing. Not
model, and already in craft-libs: code pages (craft-rtf), the ZIP entry-count
check (craft-opc), the compound-file guard (craft-cfb). The paragraph walks
moved to walk.rs and the unit helpers of wordcraft-geom to geom.rs.

* craft-wordmodel: math_tree, the arguments of an equation node and a walk over its runs

The DOCX reader and writer walk an equation's runs. The walk and the child
accessors it uses were in WordCraft's equation editor (math_edit.rs), which
stays in the app; they are structure, not editing.

* craft-mime: detect, for bytes that name no character set

is_text, decode (BOM, unmarked UTF-16, UTF-8, else the legacy encoding
the bytes look like), legacy and legacy_label, behind a detect feature
that brings in chardetng. Moved from WordCraft's crates/formats/src/txt.rs
at 06c2ae6f with its text-sniffing tests. Also charset::is_known, so a
caller can tell a label decode would read as UTF-8 from one it knows.

* craft-wordmodel: keep the document operations (edit.rs) with the model

The DOCX and RTF readers use them: alternative-format chunks are merged into
a document with adopt_blocks, and a reader formats ranges with format_range
and inserts objects. They operate on the document, not on a view, so they
belong beside it. The tests that drove them come back with them; the
paragraph walks stay in walk.rs and edit re-exports them.

base64 0.23, as WordCraft and craft-offcrypto use.

* craft-codepage: one table per legacy code page, for craft-rtf, craft-dxf and craft-mime

craft-rtf and craft-dxf each carried tables for DOS 437, 737, 850, 852 and
857 (and craft-rtf one for 866). They are now one set in craft-codepage,
which takes craft-rtf's codepage module (Windows and Mac tables, OEM glyphs,
font charsets) and craft-dxf's DOS tables. craft_rtf::codepage re-exports it
unchanged; craft-dxf keeps its $DWGCODEPAGE names and Decoder and looks the
DOS tables up there.

Every byte of every code page either crate decodes is unchanged: the
public decoders of both crates were dumped over all 256 bytes of each page
(and the East Asian pairs) before and after, and the dumps are identical.

craft-mime now reads the DOS pages by their IANA names (IBM437, cp850,
ibm-852, 857...), which it read as UTF-8 before.

* Add craft-math: the equation tree with OMML, MathML, LaTeX and UnicodeMath

The math tree, LaTeX, linear-format, symbol and tree-walk modules come
from craft-wordmodel (storytold/craft-libs#963, WordCraft's crates/doc at
07f97737) unchanged, with the run colour (Rgb) and the symbol-font tables
they use. MathML is WordCraft's crates/formats/src/mathml.rs and OMML is
WordCraft's crates/docx/src/read/math.rs and write/math.rs (06c2ae6f /
07f97737), moved onto craft-xml. The HTML-specific MathML tests stay in
WordCraft; the rest moved with the code.

mathml::parse and omml::parse/read_str/write_str are new: string entry
points for callers that have no tree of their own.

* craft-math: read MathLive's and other spellings of LaTeX math commands

\abs{...} (an absolute value), \lvert \rvert \lVert \rVert, \nparallel,
\backprime, \infinity, \cross, \differentialD, \exponentialE and
\imaginaryI (MathLive's d, upright e and i), \emph, \textsf, \hbox,
\mathnormal and \underleftarrow / \underrightarrow /
\underleftrightarrow were unknown and read as literal text. They are read
as equaSHion's typed-input parser reads them (storytold/craft-libs#29).

* craft-math: equaSHion's equation corpus as test data

corpus/equations.tsv, corpus/latex.tsv and the OMML samples of
storytold/craft-libs#29 (7a6ef6b9), unmodified.

* craft-math: corpus round trips, GridCraft's OMML cases, entry points and limits tested; README

The corpus equations go through OMML, MathML and LaTeX and back; the OMML
samples are written back as kept and from the tree. GridCraft's OMML to
LaTeX test cases (crates/xlsx/src/omml.rs) give LaTeX that reads back the
same through the tree. omml::parse and read_str are tested against other
prefixes, the strict namespace, kept OMML, junk, the element cap and deep
nesting.

* Add craft-flow: the flow model the text formats map to, on craft-wordmodel

WordCraft's crates/formats/src/model (07f97737) moved as it is: Flow,
FBlock, Para, Inline, Fmt, tables and cells, pictures, to_doc and
from_doc, picture sniffing, colours, base64 and the Dropped report. Only
the paths changed (wordcraft_doc is craft_wordmodel); the report helpers
are public so the format crates can call them; the image re-encoding
helper stays with WordCraft's RTF writer.

* Add craft-markdown: Markdown read and written, on craft-flow

WordCraft's crates/formats/src/markdown (07f97737) with its 35 tests,
moved onto craft-flow and craft-wordmodel. Character references come from
craft-html. The table cell budget lives in craft-flow, which names it in
its warning. Two test lines that wrote plain text through WordCraft's txt
module stay in WordCraft.

* WIP: craft-latex sources from WordCraft (unmodified)

* Add craft-latex: LaTeX documents read and written, on craft-flow

WordCraft's crates/formats/src/latex (07f97737) with its 21 tests, moved
onto craft-flow and craft-wordmodel; bytes are decoded by craft-mime's
detect. The LaTeX parts of WordCraft's cross-format tests (nesting,
column specs, labels, no TeX commands from the document, the fuzz and
round-trip properties) come along as tests/hostile.rs. The ligature test
builds its three-font paragraph as a flow instead of reading HTML.

* craft-math: latex_text, LaTeX math and the linear format converted as text

WordCraft's crates/formats/src/latex/math.rs (07f97737): the converter its
LaTeX document reader and writer use, which keeps the source's spacing.
It moves here so craft-math is the one place that converts LaTeX math;
craft-latex calls it, and its output does not change. Its tests (from
WordCraft's latex tests and cross-format tests) come along.

* craft-math: latex_text's group finders are public, for craft-latex's length parser

* craft-latex: math through craft_math::latex_text; label timing test by ratio

The document math converter moved to craft-math (#1040) unchanged; this
crate calls it there (and its length parser uses its group finders), so
export output is the same. The label test compared one read with a
ten-second wall clock and failed on a loaded machine; it now compares the
fastest of three reads of 25,000 and 100,000 labels, which a linear read
keeps near 4x and a quadratic one near 16x.

* Cargo.lock after the merge

* WIP: WordCraft's HTML document reader and writer, unmodified (crates/formats/src/html at 07f97737)

* craft-html: HTML documents on craft-flow, behind a flow feature

WordCraft's crates/formats/src/html (07f97737) as craft_html::doc: the
tag-soup builder, CSS subset, ruby and MathML reading, page decoding and
the writer, with its 44 tests. It reads tokens from this crate's
tokenizer through a small adapter (decoded text and attributes, names in
lower case) instead of its own lexer, and its character references are
this crate's. Lexer::set_pos is new: the ruby reader hands a token back.
The HTML parts of WordCraft's MathML and cross-format tests come along as
tests/doc.rs, the linear-time check as a ratio.

* craft-math: operator names take their LaTeX scripts directly again (restores #1019)

#1019 (22161d2d) wrote \sin^2\theta as \sin ^{2}{\theta } so it reads back as
the same tree. The squash merge of #1038, whose branch predated it, put
math_latex.rs and the corpus test back as they were before. This applies
#1019's change again, unchanged, with its test.

* Cargo.lock: craft-html's flow dependencies

---------

Co-authored-by: shuff57 <62350898+shuff57@users.noreply.github.com>
2026-11-11 21:33:47 -05:00

193 lines
7.7 KiB
Rust

use super::{Lexer, Limits, Options, Token};
/// As a mail reader sets it up.
const MAIL: Options<'static> =
Options { limits: Limits { max_attrs: 32, max_tag_name: 24 }, drop: &["script", "style", "title", "iframe"], raw_text: &["textarea", "xmp"] };
fn show<'a>(lexer: impl Iterator<Item = Token<'a>>) -> Vec<String> {
lexer
.map(|t| match t {
Token::Text(t) => format!("T:{t}"),
Token::Start(tag) => {
let attrs: Vec<String> = tag.attrs.iter().map(|(n, v)| format!("{n}={v}")).collect();
if attrs.is_empty() { format!("<{}>", tag.name) } else { format!("<{} {}>", tag.name, attrs.join(" ")) }
}
Token::End(n) => format!("</{n}>"),
})
.collect()
}
fn tokens(html: &str) -> Vec<String> {
show(Lexer::with(html, MAIL))
}
#[test]
fn text_and_tags() {
assert_eq!(tokens("Hi <B>there</b>!"), ["T:Hi ", "<b>", "T:there", "</b>", "T:!"]);
assert_eq!(tokens(""), Vec::<String>::new());
assert_eq!(tokens("plain"), ["T:plain"]);
assert_eq!(tokens("<br/><hr />"), ["<br>", "<hr>"]);
}
#[test]
fn attributes_in_every_quoting_style() {
assert_eq!(tokens(r#"<a href="x y" title='it"s' id=z data-k>"#), [r#"<a href=x y title=it"s id=z data-k=>"#]);
assert_eq!(tokens("<a href = \"u\"\n\tclass = c >t"), ["<a href=u class=c>", "T:t"]);
assert_eq!(tokens("<img src=a.png/>"), ["<img src=a.png/>"]);
assert_eq!(tokens("<p =x=y>"), ["<p =x=y>"]);
// A `>` inside a quoted value does not end the tag.
assert_eq!(tokens(r#"<a title="a>b">t</a>"#), ["<a title=a>b>", "T:t", "</a>"]);
}
#[test]
fn attribute_values_are_decoded_on_demand() {
let mut t = Lexer::new(r#"<a HREF="?a=1&amp;b=2&copy=3" href="second" Checked>"#);
let Some(Token::Start(tag)) = t.next() else { panic!("no tag") };
assert_eq!(tag.attr("href").as_deref(), Some("?a=1&b=2&copy=3"));
assert_eq!(tag.raw_attr("href"), Some("?a=1&amp;b=2&copy=3"));
assert_eq!(tag.attr("missing"), None);
assert!(tag.has("checked") && !tag.has("check"));
assert_eq!(tag.attributes().map(|(n, _)| n).collect::<Vec<_>>(), ["HREF", "href", "Checked"]);
}
#[test]
fn a_tag_that_closes_itself_says_so() {
let closes = |html: &str| match Lexer::new(html).next() {
Some(Token::Start(tag)) => tag.self_close,
other => panic!("{other:?}"),
};
for yes in ["<br/>", "<br />", "<img src=\"a\"/>", "<img src=a />", "<p\n/>"] {
assert!(closes(yes), "{yes}");
}
// A `/` that ends an unquoted value, or stands anywhere but before the `>`, is not it.
for no in ["<br>", "<img src=a.png/>", "<a / b>", "<a/ >", "<br/"] {
assert!(!closes(no), "{no}");
}
}
#[test]
fn a_lone_angle_bracket_is_text() {
assert_eq!(tokens("a < b and c > d"), ["T:a < b and c > d"]);
assert_eq!(tokens("1<2<3"), ["T:1<2<3"]);
assert_eq!(tokens("<"), ["T:<"]);
assert_eq!(tokens("x<"), ["T:x<"]);
assert_eq!(tokens("<3 <- </ x"), ["T:<3 <- </ x"]);
assert_eq!(tokens("a</>b"), ["T:a", "T:b"]);
}
#[test]
fn comments_and_declarations_vanish() {
assert_eq!(tokens("a<!-- x -- > y -->b"), ["T:a", "T:b"]);
assert_eq!(tokens("<!DOCTYPE html><p>x"), ["<p>", "T:x"]);
assert_eq!(tokens("<?xml version=\"1.0\"?>x"), ["T:x"]);
assert_eq!(tokens("a<!-->b<!--->c"), ["T:a", "T:b", "T:c"]);
assert_eq!(tokens("<!--[if mso]><p>only word</p><![endif]-->shown"), ["T:shown"]);
assert_eq!(tokens("<![if !mso]>shown<![endif]>"), ["T:shown"]);
// An unfinished comment swallows the rest.
assert_eq!(tokens("a<!-- b <p>c"), ["T:a"]);
}
#[test]
fn dropped_elements_never_leave_the_lexer() {
assert_eq!(tokens("a<script>if (a<b) alert('</p>')</script>b"), ["T:a", "T:b"]);
assert_eq!(tokens("a<STYLE type=text/css>p > b { color: red }</Style >b"), ["T:a", "T:b"]);
assert_eq!(tokens("<title>Hello</title>x"), ["T:x"]);
assert_eq!(tokens("a<script>never closed <p>x"), ["T:a"]);
// `</scripty>` is not the end tag.
assert_eq!(tokens("<script>x</scripty>y</script>z"), ["T:z"]);
// Dropping is an option: by default the tags come out and the content is markup like any other.
assert_eq!(show(Lexer::new("a<script>x<b>y</script>b")), ["T:a", "<script>", "T:x", "<b>", "T:y", "</script>", "T:b"]);
}
#[test]
fn raw_content_is_the_callers_to_take() {
let src = "<style media=print>p > b { color: red } </p></STYLE >after<script>x";
let mut l = Lexer::new(src);
let Some(Token::Start(tag)) = l.next() else { panic!("no tag") };
assert_eq!((tag.name.as_str(), tag.raw_attr("media")), ("style", Some("print")));
assert_eq!(l.raw_content("style"), "p > b { color: red } </p>");
assert_eq!(l.next(), Some(Token::Text("after")));
assert!(matches!(l.next(), Some(Token::Start(t)) if t.name == "script"));
// No end tag: to the end of the input.
assert_eq!(l.raw_content("script"), "x");
assert_eq!((l.next(), l.pos()), (None, src.len()));
}
#[test]
fn raw_text_content_is_text() {
assert_eq!(tokens("<textarea><b>x</b></textarea>y"), ["<textarea>", "T:<b>x</b>", "</textarea>", "T:y"]);
assert_eq!(tokens("<textarea></textarea>"), ["<textarea>", "</textarea>"]);
assert_eq!(tokens("<xmp>a<p>b"), ["<xmp>", "T:a<p>b"]);
assert_eq!(show(Lexer::new("<xmp>a<p>b")), ["<xmp>", "T:a", "<p>", "T:b"]);
}
#[test]
fn unfinished_tags_run_to_the_end() {
assert_eq!(tokens("a<b"), ["T:a", "<b>"]);
assert_eq!(tokens("a<a href=\"x"), ["T:a", "<a href=x>"]);
assert_eq!(tokens("a</b"), ["T:a", "</b>"]);
assert_eq!(tokens("a<b c"), ["T:a", "<b c=>"]);
assert_eq!(tokens("a<b c="), ["T:a", "<b c=>"]);
}
#[test]
fn non_ascii_everywhere() {
assert_eq!(tokens("é<b é=ü ñ>ß</b>ø"), ["T:é", "<b é=ü ñ=>", "T:ß", "</b>", "T:ø"]);
assert_eq!(tokens("<é>"), ["T:<é>"]);
assert_eq!(tokens("<bé>x"), ["<bé>", "T:x"]);
}
#[test]
fn long_names_and_many_attributes_are_capped() {
let name = |n: usize, opts: Options<'static>| show(Lexer::with(&format!("<{0}>x</{0}>", "a".repeat(n)), opts));
assert_eq!(name(25, MAIL), ["<>", "T:x", "</>"]);
assert_eq!(name(24, MAIL).first().map(String::len), Some(26));
assert_eq!(name(65, Options::default()), ["<>", "T:x", "</>"]);
assert_eq!(name(64, Options::default()).first().map(String::len), Some(66));
let many = format!("<p {}>", (0..100).map(|i| format!("a{i}=1")).collect::<Vec<_>>().join(" "));
for (opts, kept) in [(MAIL, 32), (Options::default(), 64)] {
let Some(Token::Start(tag)) = Lexer::with(&many, opts).next() else { panic!("no tag") };
assert_eq!(tag.attrs.len(), kept);
}
}
#[test]
fn hostile_runs_are_linear() {
for s in [
"<".repeat(300_000),
"<a ".repeat(100_000),
"<!--".repeat(100_000),
"</".repeat(150_000),
"<script>".repeat(50_000),
"<b x=\"".repeat(50_000),
] {
assert!(Lexer::with(&s, MAIL).count() <= s.len());
assert!(Lexer::new(&s).count() <= s.len());
}
}
proptest::proptest! {
#[test]
fn any_input_tokenizes_and_keeps_the_text(s in "[<>/!a-c =\"'&;é-]{0,80}") {
for opts in [MAIL, Options::default()] {
let total: usize = Lexer::with(&s, opts).map(|t| match t { Token::Text(t) => t.len(), _ => 0 }).sum();
proptest::prop_assert!(total <= s.len());
}
}
}
#[test]
fn a_token_can_be_read_again() {
let mut l = Lexer::new("é<b>x</b>");
assert_eq!(l.next(), Some(Token::Text("é")));
let at = l.pos();
assert!(matches!(l.next(), Some(Token::Start(t)) if t.name == "b"));
l.set_pos(at);
assert!(matches!(l.next(), Some(Token::Start(t)) if t.name == "b"));
// Inside a character or past the end: the next boundary.
l.set_pos(1);
assert!(matches!(l.next(), Some(Token::Start(t)) if t.name == "b"));
l.set_pos(1000);
assert_eq!(l.next(), None);
}