pauljd/craft-libs
1
0
Fork 0
mirror of https://github.com/storytold/craft-libs.git synced 2026-11-12 08:52:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

craft-html: HTML documents on craft-flow, behind a flow feature

Browse Source 
WordCraft's crates/formats/src/html (07f97737) as craft_html::doc: the
tag-soup builder, CSS subset, ruby and MathML reading, page decoding and
the writer, with its 44 tests. It reads tokens from this crate's
tokenizer through a small adapter (decoded text and attributes, names in
lower case) instead of its own lexer, and its character references are
this crate's. Lexer::set_pos is new: the ruby reader hands a token back.
The HTML parts of WordCraft's MathML and cross-format tests come along as
tests/doc.rs, the linear-time check as a ratio.
This commit is contained in:
Dylan M. Taylor
2026-11-12 01:26:44 +00:00
parent 3d9403037c
commit 04556167cd
16 changed files with 263 additions and 551 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.lock
Generated
+6
View File
@@ -1248,6 +1248,12 @@ dependencies = [
name = "craft-html"
version = "0.1.0"
dependencies = [
"craft-codepage",
"craft-flow",
"craft-math",
"craft-mime",
"craft-wordmodel",
"encoding_rs",
"proptest",
]
README.md
+1 -1
View File
@@ -137,7 +137,7 @@ craft-<name> = { git = "https://github.com/storytold/craft-libs", rev = "<commit
heic-rs reads the container and HEVC pictures, and this crate converts to 16-bit RGB to match
libheif (VUI range and matrix, chroma upsampling, `imir`, ICC on grid tiles). Also orientation,
alpha, EXIF/XMP and thumbnails. It never panics, and apps put it behind a non-default feature.
- [`craft-html`](crates/html/README.md): the bottom layer of reading and writing HTML: a tolerant tokenizer (text, start tags, end tags; elements to drop or to read as raw text are options), character references with the legacy names browsers accept without `;`, escaping, and the character set a page declares. No tree, no CSS, no rendering.
- [`craft-html`](crates/html/README.md): the bottom layer of reading and writing HTML: a tolerant tokenizer (text, start tags, end tags; elements to drop or to read as raw text are options), character references with the legacy names browsers accept without `;`, escaping, and the character set a page declares. With the `flow` feature, HTML documents read into and written from the craft-flow model.
- [`craft-i18n`](crates/i18n/README.md): translations over the embedded craft-i18n catalogs:
lookup with fallback (`pt-br` → `pt` → English), plurals, language names, the system locale and
a pseudo locale. The catalogs stay a fully optional build input; apps keep `tl!` and their
crates/html/Cargo.toml
+16
View File
@@ -10,5 +10,21 @@ repository.workspace = true
[lints]
workspace = true
[features]
# HTML documents: read into and written from the craft-flow model and a craft-wordmodel Document.
flow = ["dep:craft-codepage", "dep:craft-flow", "dep:craft-math", "dep:craft-mime", "dep:craft-wordmodel", "dep:encoding_rs"]
[dependencies]
craft-codepage = { workspace = true, optional = true }
craft-flow = { workspace = true, optional = true }
craft-math = { workspace = true, optional = true }
craft-mime = { workspace = true, features = ["detect"], optional = true }
craft-wordmodel = { workspace = true, optional = true }
encoding_rs = { workspace = true, optional = true }
[dev-dependencies]
proptest = { workspace = true }
[[test]]
name = "doc"
required-features = ["flow"]
crates/html/README.md
+17 -1
View File
@@ -52,12 +52,28 @@ linear.
`escape` writes `& < > " '` as references; `escape_dq` leaves `'`, for content and attribute
values in double quotes.
## HTML documents (feature `flow`)
`craft_html::doc` reads a page or a pasted fragment into the [craft-flow](../flow/README.md)
model and a craft-wordmodel `Document`, and writes one as clean HTML. The reader is a tolerant
tag-soup builder on this crate's tokenizer (a stack of open elements with HTML's implicit closes),
with a CSS subset from `style` attributes and `<style>` rules with simple selectors in cascade
order, tables with spans, lists, pictures (`data:` URIs, others through the caller's
`ImageLoader`), ruby and MathML equations ([craft-math](../math/README.md)). `import_page` opens a
web page as Word does (Times New Roman 12 pt, HTML's default spacing); `import` reads a fragment in
the document's own formatting. `decode` / `decode_as` turn a page's bytes into text (BOM, declared
charset, else detection). The writer gives semantic HTML with inline CSS, pictures as `data:`
URIs, equations as MathML and ruby as `<ruby>`.
The feature brings in craft-flow, craft-wordmodel, craft-math, craft-mime and craft-codepage; a
reader that only needs tokens (mail, clipboard tables) leaves it off.
## Not here
- The full HTML5 named reference table (about 2,200 names): only the names above.
- Decoding bytes to text. `declared_charset` gives the label a page names; a decoder for it is
the caller's (`craft-mime::charset` for the WHATWG labels).
- Tree construction, implied end tags, CSS.
- A DOM, and CSS beyond the subset the document reader applies.
## Provenance
crates/html/src/doc.rs
+8 -6
View File
@@ -1,4 +1,5 @@
//! HTML.
//! HTML documents (feature `flow`): read into and written from the craft-flow model and a
//! craft-wordmodel `Document`.
//!
//! Import is a tolerant tag-soup reader (no DOM): a tokenizer plus a stack of open elements with
//! HTML's implicit closes for `p`, `li`, `tr`, `td`/`th` and headings. It understands `p`, `div`,
@@ -18,12 +19,13 @@
use std::sync::Arc;
use wordcraft_doc::props::LineSpacing;
use wordcraft_doc::props::VAlign;
use wordcraft_doc::{Align, Document};
use craft_wordmodel::props::LineSpacing;
use craft_wordmodel::props::VAlign;
use craft_wordmodel::{Align, Document};
use crate::model::{
self, Cell, FBlock, FTable, Flow, Fmt, Inline, Kind, ListInfo, Meta, Para, Spacing, base64_encode, data_uri, make_img, mime_of, parse_color,
use craft_flow::{
self as model, Cell, FBlock, FTable, Flow, Fmt, Inline, Kind, ListInfo, Meta, Para, Spacing, base64_encode, data_uri, make_img, mime_of,
parse_color,
};
mod builder;
crates/html/src/doc/builder.rs
+7 -7
View File
@@ -459,8 +459,8 @@ fn important_last(css: &str) -> String {
/// The borders a cell's CSS gives it (`border`, `border-top`…: style, width and colour in any
/// order; `windowtext` is the automatic colour). `None` when it states none.
fn cell_borders(css: &str) -> Option<wordcraft_doc::props::Borders> {
use wordcraft_doc::props::{Border, BorderStyle, Borders};
fn cell_borders(css: &str) -> Option<craft_wordmodel::props::Borders> {
use craft_wordmodel::props::{Border, BorderStyle, Borders};
let mut b = Borders::default();
let mut any = false;
for (k, v) in decls(css) {
@@ -681,7 +681,7 @@ impl<'r> Builder<'r> {
// `<p>&nbsp;</p>`: an empty paragraph on purpose, whose line is as high as the
// space's formatting makes it.
if let Some(Inline::Text(_, f)) = p.inlines.first() {
p.mark = Some(Box::new(crate::model::char_props(f)));
p.mark = Some(Box::new(craft_flow::char_props(f)));
}
p.inlines.clear();
self.para_cont = false;
@@ -850,7 +850,7 @@ impl<'r> Builder<'r> {
let borderless = !t.bordered;
// Keep at most MAX_ROWS rows (the model drops the rest anyway): bounds the
// O(rows × rowspan) work of `insert_covered` on a table of millions of `<tr>`.
let rows: Vec<_> = t.rows.into_iter().filter(|r| !r.is_empty()).take(wordcraft_doc::table::MAX_ROWS).collect();
let rows: Vec<_> = t.rows.into_iter().filter(|r| !r.is_empty()).take(craft_wordmodel::table::MAX_ROWS).collect();
let mut ft = FTable { rows, widths: Vec::new(), borderless, extra: None, kept: None, wrap: None };
if !ft.rows.is_empty() {
ft.insert_covered();
@@ -1019,10 +1019,10 @@ impl<'r> Builder<'r> {
"sub" => fmt.sub = true,
"small" => fmt.size = Some(fmt.size.map_or(10.0, |s| (s * 0.833).max(1.0))),
"big" => fmt.size = Some(fmt.size.map_or(14.0, |s| (s * 1.2).min(1638.0))),
"mark" => fmt.background = Some(wordcraft_doc::Rgb(255, 255, 0)),
"mark" => fmt.background = Some(craft_wordmodel::Rgb(255, 255, 0)),
"font" => apply_font_attributes(attrs, &mut fmt),
"a" => {
if let Some(href) = attr(attrs, "href").filter(|h| wordcraft_doc::link::is_safe_href(h)) {
if let Some(href) = attr(attrs, "href").filter(|h| craft_wordmodel::link::is_safe_href(h)) {
fmt.link = model::kept_link(href.trim());
}
}
@@ -1046,7 +1046,7 @@ impl<'r> Builder<'r> {
// A page's headings are bold, black (unless they inherit a colour) and sized as
// browsers show them.
fmt.bold = true;
fmt.color.get_or_insert(wordcraft_doc::Rgb(0, 0, 0));
fmt.color.get_or_insert(craft_wordmodel::Rgb(0, 0, 0));
fmt.size = PAGE_HEADING_SIZES.get(usize::from(n.clamp(1, 6)) - 1).copied();
}
apply_style(css, &mut fmt, &mut align, &mut preserve, &mut page_break, &mut page_break_after);
crates/html/src/doc/entity.rs
+9 -333
View File
@@ -1,323 +1,11 @@
// ---------------------------------------------------------------------------------------------
// Entities and decoding
//! Character references (from the crate's core) and decoding a page's bytes to text.
/// Decode an entity body (`amp`, `#38`, `#x26`) to its text.
/// The text of an entity body (`amp`, `#38`, `#x26`).
pub fn decode_entity(name: &str) -> Option<String> {
decode_char(name).map(String::from)
crate::decode_entity(name).map(String::from)
}
fn decode_char(name: &str) -> Option<char> {
if let Some(num) = name.strip_prefix('#') {
let (digits, radix) = match num.strip_prefix(['x', 'X']) {
Some(h) => (h, 16),
None => (num, 10),
};
if !digits.chars().all(|c| c.is_digit(radix)) {
return None;
}
let v = u32::from_str_radix(digits, radix).ok()?;
let c = match v {
0 => '\u{FFFD}',
0x80..=0x9F => crate::txt::cp1252(v as u8),
_ => char::from_u32(v).unwrap_or('\u{FFFD}'),
};
return Some(c);
}
let c = match name {
"amp" | "AMP" => '&',
"lt" | "LT" => '<',
"gt" | "GT" => '>',
"quot" | "QUOT" => '"',
"apos" | "APOS" => '\'',
"nbsp" | "NBSP" => '\u{00A0}',
"ensp" => '\u{2002}',
"emsp" => '\u{2003}',
"thinsp" => '\u{2009}',
"shy" => '\u{00AD}',
"copy" | "COPY" => '©',
"reg" | "REG" => '®',
"trade" | "TRADE" => '™',
"hellip" => '…',
"mdash" => '—',
"ndash" => '–',
"minus" => '−',
"lsquo" => '‘',
"rsquo" => '’',
"sbquo" => '‚',
"ldquo" => '“',
"rdquo" => '”',
"bdquo" => '„',
"laquo" => '«',
"raquo" => '»',
"bull" => '•',
"middot" => '·',
"deg" => '°',
"plusmn" => '±',
"times" => '×',
"divide" => '÷',
"frac12" => '½',
"frac14" => '¼',
"frac34" => '¾',
"frac13" => '⅓',
"frac23" => '⅔',
"frac18" => '⅛',
"frac38" => '⅜',
"frac58" => '⅝',
"frac78" => '⅞',
"sup1" => '¹',
"sup2" => '²',
"sup3" => '³',
"micro" => 'µ',
"permil" => '‰',
"prime" => '′',
"Prime" => '″',
"oline" => '‾',
"frasl" => '⁄',
"sect" => '§',
"para" => '¶',
"euro" | "EURO" => '€',
"pound" | "POUND" => '£',
"yen" | "YEN" => '¥',
"cent" | "CENT" => '¢',
"curren" => '¤',
"dagger" => '†',
"Dagger" => '‡',
"larr" => '←',
"rarr" => '→',
"uarr" => '↑',
"darr" => '↓',
"harr" => '↔',
"lArr" => '⇐',
"rArr" => '⇒',
"uArr" => '⇑',
"dArr" => '⇓',
"hArr" => '⇔',
"crarr" => '↵',
"forall" => '∀',
"part" => '∂',
"exist" => '∃',
"empty" => '∅',
"nabla" => '∇',
"isin" => '∈',
"notin" => '∉',
"ni" => '∋',
"prod" => '∏',
"sum" => '∑',
"radic" => '√',
"prop" => '∝',
"ang" => '∠',
"and" => '∧',
"or" => '∨',
"cap" => '∩',
"cup" => '∪',
"int" => '∫',
"there4" => '∴',
"sim" => '∼',
"cong" => '≅',
"asymp" | "approx" => '≈',
"ne" => '≠',
"equiv" => '≡',
"le" => '≤',
"ge" => '≥',
"sub" => '⊂',
"sup" => '⊃',
"nsub" => '⊄',
"sube" => '⊆',
"supe" => '⊇',
"oplus" => '⊕',
"otimes" => '⊗',
"perp" => '⊥',
"sdot" => '⋅',
"infin" => '∞',
"loz" => '◊',
"spades" => '♠',
"clubs" => '♣',
"hearts" => '♥',
"diams" => '♦',
"check" => '✓',
"iexcl" => '¡',
"iquest" => '¿',
"szlig" => 'ß',
"auml" => 'ä',
"ouml" => 'ö',
"uuml" => 'ü',
"Auml" => 'Ä',
"Ouml" => 'Ö',
"Uuml" => 'Ü',
"eacute" => 'é',
"egrave" => 'è',
"ecirc" => 'ê',
"euml" => 'ë',
"Eacute" => 'É',
"Egrave" => 'È',
"Ecirc" => 'Ê',
"Euml" => 'Ë',
"aacute" => 'á',
"agrave" => 'à',
"acirc" => 'â',
"atilde" => 'ã',
"Agrave" => 'À',
"Aacute" => 'Á',
"Acirc" => 'Â',
"Atilde" => 'Ã',
"iacute" => 'í',
"igrave" => 'ì',
"icirc" => 'î',
"iuml" => 'ï',
"Iacute" => 'Í',
"Igrave" => 'Ì',
"Icirc" => 'Î',
"Iuml" => 'Ï',
"oacute" => 'ó',
"ograve" => 'ò',
"ocirc" => 'ô',
"otilde" => 'õ',
"Oacute" => 'Ó',
"Ograve" => 'Ò',
"Ocirc" => 'Ô',
"Otilde" => 'Õ',
"uacute" => 'ú',
"ugrave" => 'ù',
"ucirc" => 'û',
"Uacute" => 'Ú',
"Ugrave" => 'Ù',
"Ucirc" => 'Û',
"yacute" => 'ý',
"yuml" => 'ÿ',
"Yacute" => 'Ý',
"ntilde" => 'ñ',
"Ntilde" => 'Ñ',
"ccedil" => 'ç',
"Ccedil" => 'Ç',
"aring" => 'å',
"Aring" => 'Å',
"oslash" => 'ø',
"Oslash" => 'Ø',
"aelig" => 'æ',
"AElig" => 'Æ',
"alpha" => 'α',
"beta" => 'β',
"gamma" => 'γ',
"delta" => 'δ',
"epsilon" => 'ε',
"zeta" => 'ζ',
"eta" => 'η',
"theta" => 'θ',
"iota" => 'ι',
"kappa" => 'κ',
"lambda" => 'λ',
"mu" => 'μ',
"nu" => 'ν',
"xi" => 'ξ',
"pi" => 'π',
"rho" => 'ρ',
"sigma" => 'σ',
"tau" => 'τ',
"upsilon" => 'υ',
"phi" => 'φ',
"chi" => 'χ',
"psi" => 'ψ',
"omega" => 'ω',
"Gamma" => 'Γ',
"Delta" => 'Δ',
"Theta" => 'Θ',
"Lambda" => 'Λ',
"Xi" => 'Ξ',
"Pi" => 'Π',
"Sigma" => 'Σ',
"Upsilon" => 'Υ',
"Phi" => 'Φ',
"Psi" => 'Ψ',
"Omega" => 'Ω',
"zwj" => '\u{200D}',
"zwnj" => '\u{200C}',
_ => return None,
};
Some(c)
}
/// Legacy names HTML5 also decodes without a `;` (only those [`decode_char`] knows), sorted by bytes.
const BARE_NAMES: &[&str] = &[
"AElig", "AMP", "Aacute", "Acirc", "Agrave", "Aring", "Atilde", "Auml", "COPY", "Ccedil", "Eacute", "Ecirc", "Egrave", "Euml", "GT", "Iacute",
"Icirc", "Igrave", "Iuml", "LT", "Ntilde", "Oacute", "Ocirc", "Ograve", "Oslash", "Otilde", "Ouml", "QUOT", "REG", "Uacute", "Ucirc", "Ugrave",
"Uuml", "Yacute", "aacute", "acirc", "aelig", "agrave", "amp", "aring", "atilde", "auml", "ccedil", "cent", "copy", "curren", "deg", "divide",
"eacute", "ecirc", "egrave", "euml", "frac12", "frac14", "frac34", "gt", "iacute", "icirc", "iexcl", "igrave", "iquest", "iuml", "laquo", "lt",
"micro", "middot", "nbsp", "ntilde", "oacute", "ocirc", "ograve", "oslash", "otilde", "ouml", "para", "plusmn", "pound", "quot", "raquo", "reg",
"sect", "shy", "sup1", "sup2", "sup3", "szlig", "times", "uacute", "ucirc", "ugrave", "uuml", "yacute", "yen", "yuml",
];
/// A reference written without its `;` at the start of `after` (the text after `&`): a numeric one,
/// or the longest legacy name. In attribute values a name followed by `=` or an alphanumeric stays text.
fn bare_reference(after: &str, attr: bool) -> Option<(char, usize)> {
if let Some(num) = after.strip_prefix('#') {
let hex = num.starts_with(['x', 'X']);
let radix = if hex { 16 } else { 10 };
let skip = usize::from(hex);
let n = num.get(skip..)?.bytes().take_while(|b| char::from(*b).is_digit(radix)).count();
if n == 0 {
return None;
}
let len = 1 + skip + n;
return decode_char(after.get(..len)?).map(|c| (c, len));
}
// Legacy names are 2 to 6 ASCII alphanumerics: try the longest prefix first.
let run = after.bytes().take(6).take_while(u8::is_ascii_alphanumeric).count();
let len = (2..=run).rev().find(|&l| after.get(..l).is_some_and(|p| BARE_NAMES.binary_search(&p).is_ok()))?;
let next = after.as_bytes().get(len);
if attr && next.is_some_and(|b| b.is_ascii_alphanumeric() || *b == b'=') {
return None;
}
decode_char(after.get(..len)?).map(|c| (c, len))
}
/// Replace entity references in text `s`.
pub fn unescape(s: &str) -> String {
unescape_in(s, false)
}
/// Replace entity references in an attribute value `s`.
pub fn unescape_attr(s: &str) -> String {
unescape_in(s, true)
}
fn unescape_in(s: &str, attr: bool) -> String {
if !s.contains('&') {
return s.to_string();
}
let mut out = String::with_capacity(s.len());
let mut rest = s;
while let Some(k) = rest.find('&') {
out.push_str(&rest[..k]);
let after = &rest[k + 1..];
// An entity name is short; looking further would rescan the whole text for every stray `&`.
let mut cut = after.len().min(33);
while !after.is_char_boundary(cut) {
cut -= 1;
}
let end = after.get(..cut).and_then(|w| w.find(';'));
match end.and_then(|e| decode_char(&after[..e]).map(|c| (c, e + 1))).or_else(|| bare_reference(after, attr)) {
Some((c, e)) => {
out.push(c);
rest = &after[e..];
}
None => {
out.push('&');
rest = after;
}
}
}
out.push_str(rest);
out
}
fn sniff_charset(head: &str) -> Option<&str> {
let (i, n) = head.find("charset").map(|i| (i, 7)).or_else(|| head.find("encoding").map(|i| (i, 8)))?;
let after = head.get(i + n..)?.trim_start().strip_prefix('=')?;
let trimmed = after.trim_start().trim_start_matches(['"', '\'']);
let end = trimmed.find(|c: char| c.is_whitespace() || matches!(c, '"' | '\'' | ';' | '>' | '/')).unwrap_or(trimmed.len());
let label = trimmed[..end].trim();
if label.is_empty() { None } else { Some(label) }
}
pub use crate::unescape;
/// Decode HTML bytes: BOM, then a `charset` in the first 2 KB, else UTF-8 with a legacy fallback.
pub fn decode(b: &[u8]) -> String {
@@ -329,17 +17,16 @@ pub fn decode(b: &[u8]) -> String {
/// are decoded as such, other labels by their WHATWG decoders.
pub fn decode_as(b: &[u8], charset: Option<&str>) -> String {
if b.starts_with(&[0xFF, 0xFE]) || b.starts_with(&[0xFE, 0xFF]) || b.starts_with(&[0xEF, 0xBB, 0xBF]) {
return crate::txt::decode(b);
return craft_mime::detect::decode(b);
}
let head = String::from_utf8_lossy(b.get(..b.len().min(2048)).unwrap_or(b)).to_ascii_lowercase();
let named = charset.map(str::to_ascii_lowercase).or_else(|| sniff_charset(&head).map(str::to_string));
let named = charset.map(str::to_ascii_lowercase).or_else(|| crate::declared_charset(b));
let code_page = match named.as_deref().map(str::trim) {
Some("iso-8859-1" | "latin1" | "us-ascii") => Some(1252),
Some(cs) => cs
.strip_prefix("windows-")
.or_else(|| cs.strip_prefix("cp"))
.and_then(|n| n.parse::<u16>().ok())
.filter(|cp| wordcraft_doc::encoding::SINGLE_BYTE_CODE_PAGES.contains(cp)),
.filter(|cp| craft_codepage::SINGLE_BYTE_CODE_PAGES.contains(cp)),
None => None,
};
// Other labels (ISO 8859, KOI8, Shift_JIS, GBK, Big5, EUC-KR…) by their WHATWG decoders; a
@@ -349,22 +36,11 @@ pub fn decode_as(b: &[u8], charset: Option<&str>) -> String {
.and_then(|l| encoding_rs::Encoding::for_label(l.trim().as_bytes()))
.filter(|e| ![encoding_rs::UTF_8, encoding_rs::UTF_16LE, encoding_rs::UTF_16BE].contains(e));
match (code_page, other) {
(Some(cp), _) => b.iter().map(|&c| wordcraft_doc::encoding::windows_byte(cp, c)).collect(),
(Some(cp), _) => b.iter().map(|&c| craft_codepage::windows_byte(cp, c)).collect(),
(None, Some(enc)) => enc.decode_without_bom_handling(b).0.into_owned(),
(None, None) => match std::str::from_utf8(b) {
Ok(s) => s.to_string(),
Err(_) => crate::txt::decode(b),
Err(_) => craft_mime::detect::decode(b),
},
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn bare_names_are_sorted_and_known() {
assert!(BARE_NAMES.windows(2).all(|w| w[0] < w[1]));
assert!(BARE_NAMES.iter().all(|n| (2..=6).contains(&n.len()) && decode_char(n).is_some()));
}
}
crates/html/src/doc/export.rs
+3 -3
View File
@@ -71,7 +71,7 @@ fn fmt_tags(f: &Fmt) -> (String, Vec<&'static str>) {
let mut open = String::new();
let mut close: Vec<&str> = Vec::new();
// A script or other non-web target is written as plain text (`is_safe_href`).
if let Some(l) = f.link.as_deref().filter(|l| wordcraft_doc::link::is_safe_href(l)) {
if let Some(l) = f.link.as_deref().filter(|l| craft_wordmodel::link::is_safe_href(l)) {
open.push_str(&format!("<a href=\"{}\">", esc(l)));
close.push("</a>");
}
@@ -175,9 +175,9 @@ pub fn inlines_html(inlines: &[Inline]) -> String {
}
}
// No picture to show: the browser shows the alt text in its place.
Inline::Figure(alt) | Inline::Chart(crate::model::Chart { alt, .. }) => o.push_str(&format!("<img alt=\"{}\">", esc(alt))),
Inline::Figure(alt) | Inline::Chart(craft_flow::Chart { alt, .. }) => o.push_str(&format!("<img alt=\"{}\">", esc(alt))),
// MathML, which browsers, Word and LibreOffice show as an equation (#816).
Inline::Equation { linear, display } => o.push_str(&crate::mathml::linear_to_mathml(linear, *display)),
Inline::Equation { linear, display } => o.push_str(&craft_math::mathml::linear_to_mathml(linear, *display)),
}
}
o
crates/html/src/doc/import.rs
+5 -5
View File
@@ -29,7 +29,7 @@ pub(super) fn parse_in(s: &str, images: ImageLoader, page: bool) -> Flow {
};
// The lexer's offsets (a style element's text is cut out by them) are into the normalized text.
let s: &str = &normalized;
let mut lx = Lexer { s, pos: 0 };
let mut lx = Lexer::new(s);
let mut b = Builder::new(images, page);
let mut guard = 0usize;
while let Some(t) = lx.next() {
@@ -41,10 +41,10 @@ pub(super) fn parse_in(s: &str, images: ImageLoader, page: bool) -> Flow {
Tok::Text(t) => b.text(&t),
Tok::Start { name, attrs, self_close } => {
b.close_head_for_start(&name);
if crate::mathml::local(&name) == "math" && !self_close {
if craft_math::mathml::local(&name) == "math" && !self_close {
let el = read_math(&mut lx, name, attrs);
if !b.in_head()
&& let Some((linear, display)) = crate::mathml::read(&el)
&& let Some((linear, display)) = craft_math::mathml::read(&el)
{
b.start_para().inlines.push(Inline::Equation { linear, display });
}
@@ -58,7 +58,7 @@ pub(super) fn parse_in(s: &str, images: ImageLoader, page: bool) -> Flow {
continue;
}
if matches!(name.as_str(), "script" | "style" | "noscript" | "template" | "textarea") && !self_close {
let from = lx.pos;
let from = lx.pos();
let to = lx.skip_past_end_tag(&name);
if name == "style" {
parse_sheet(s.get(from..to).unwrap_or(""), &mut b.sheet);
@@ -114,7 +114,7 @@ pub fn import_page(s: &str, images: ImageLoader) -> Document {
/// [`import_page`] onto `page` (its size and margins), which tables size to: HTML gives no page
/// size, so the caller picks it (the region's paper, #1381).
pub fn import_on(s: &str, images: ImageLoader, page: &wordcraft_doc::SectionProps) -> Document {
pub fn import_on(s: &str, images: ImageLoader, page: &craft_wordmodel::SectionProps) -> Document {
model::begin_import();
model::to_doc_in(&parse_in(s, images, true), page.clone())
}
crates/html/src/doc/lexer.rs
+26 -152
View File
@@ -1,7 +1,7 @@
use super::entity::{unescape, unescape_attr};
//! The crate's tokenizer ([`crate::Lexer`]) as the document reader takes its tokens: text and
//! attribute values decoded, element and attribute names in lower case.
// ---------------------------------------------------------------------------------------------
// Tokenizer
use crate::{Token, unescape, unescape_attr};
#[derive(Debug)]
pub(super) enum Tok {
@@ -11,166 +11,40 @@ pub(super) enum Tok {
}
pub(super) struct Lexer<'a> {
pub(super) s: &'a str,
pub(super) pos: usize,
inner: crate::Lexer<'a, 'static>,
}
impl Lexer<'_> {
fn rest(&self) -> &str {
self.s.get(self.pos..).unwrap_or("")
impl<'a> Lexer<'a> {
pub(super) fn new(s: &'a str) -> Lexer<'a> {
Lexer { inner: crate::Lexer::new(s) }
}
/// Skip past the next occurrence of `pat` (ASCII case-insensitive), or to the end.
fn skip_past(&mut self, pat: &str) -> usize {
let start = self.pos;
let hay = self.rest().as_bytes();
let p = pat.as_bytes();
let found = hay.windows(p.len().max(1)).position(|w| w.eq_ignore_ascii_case(p));
match found {
Some(k) => self.pos += k + p.len(),
None => self.pos = self.s.len(),
}
found.map(|k| start + k).unwrap_or(self.s.len())
/// Where the next token starts.
pub(super) fn pos(&self) -> usize {
self.inner.pos()
}
/// Read again from `pos` (an earlier [`Lexer::pos`]).
pub(super) fn set_pos(&mut self, pos: usize) {
self.inner.set_pos(pos);
}
/// Skip past the next matching end tag `</tag...>` (ASCII case-insensitive), or to the end.
/// Skip past the end tag of raw text element `tag`; returns where that end tag starts (the
/// end of the input when there is none).
pub(super) fn skip_past_end_tag(&mut self, tag: &str) -> usize {
let prefix = format!("</{tag}");
let p = prefix.as_bytes();
while self.pos < self.s.len() {
let hay = self.rest().as_bytes();
let found = hay.windows(p.len().max(1)).position(|w| w.eq_ignore_ascii_case(p));
let Some(k) = found else {
self.pos = self.s.len();
return self.pos;
};
let after_tag = k + p.len();
let next_byte = hay.get(after_tag);
if next_byte.is_none_or(|b| b.is_ascii_whitespace() || *b == b'>' || *b == b'/') {
let at = self.pos + k;
self.pos += after_tag;
self.skip_past(">");
return at;
}
self.pos += k + 2;
}
self.s.len()
let from = self.inner.pos();
from + self.inner.raw_content(tag).len()
}
pub(super) fn next(&mut self) -> Option<Tok> {
let rest = self.rest();
if rest.is_empty() {
return None;
}
if !rest.starts_with('<') {
let end = rest.find('<').unwrap_or(rest.len());
let t = unescape(&rest[..end]);
self.pos += end;
return Some(Tok::Text(t));
}
let b = rest.as_bytes();
if rest.starts_with("<!--") {
if rest.starts_with("<!-->") {
self.pos += 5;
} else if rest.starts_with("<!--->") {
self.pos += 6;
} else {
self.pos += 4;
self.skip_past("-->");
}
return Some(Tok::Text(String::new()));
}
if rest.starts_with("<!") || rest.starts_with("<?") {
self.skip_past(">");
return Some(Tok::Text(String::new()));
}
if rest.starts_with("</") {
let name: String = rest
.strip_prefix("</")
.unwrap_or("")
.chars()
.take_while(|c| c.is_ascii_alphanumeric() || *c == ':' || *c == '-')
.map(|c| c.to_ascii_lowercase())
.collect();
if name.is_empty() {
self.pos += 2;
return Some(Tok::Text("</".into()));
}
self.skip_past(">");
return Some(Tok::End(name));
}
if !b.get(1).is_some_and(u8::is_ascii_alphabetic) {
self.pos += 1;
return Some(Tok::Text("<".into()));
}
// Start tag.
let name: String =
rest[1..].chars().take_while(|c| c.is_ascii_alphanumeric() || *c == ':' || *c == '-').map(|c| c.to_ascii_lowercase()).collect();
let mut i = 1 + name.len();
let mut attrs = Vec::new();
let mut self_close = false;
loop {
while b.get(i).is_some_and(u8::is_ascii_whitespace) {
i += 1;
}
match b.get(i) {
None => break,
Some(b'>') => {
i += 1;
break;
}
Some(b'/') => {
if b.get(i + 1) == Some(&b'>') {
self_close = true;
i += 2;
break;
}
i += 1;
continue;
}
_ => {}
}
let a0 = i;
while b.get(i).is_some_and(|c| !c.is_ascii_whitespace() && !matches!(c, b'=' | b'>' | b'/')) {
i += 1;
}
let an = rest.get(a0..i).unwrap_or("").to_ascii_lowercase();
if an.is_empty() {
i += 1;
continue;
}
while b.get(i).is_some_and(u8::is_ascii_whitespace) {
i += 1;
}
let mut val = String::new();
if b.get(i) == Some(&b'=') {
i += 1;
while b.get(i).is_some_and(u8::is_ascii_whitespace) {
i += 1;
}
match b.get(i) {
Some(&q) if q == b'"' || q == b'\'' => {
let v0 = i + 1;
let e = rest.get(v0..).and_then(|r| r.find(q as char)).map(|k| v0 + k).unwrap_or(rest.len());
val = unescape_attr(rest.get(v0..e).unwrap_or(""));
i = e + 1;
}
_ => {
let v0 = i;
while b.get(i).is_some_and(|c| !c.is_ascii_whitespace() && *c != b'>') {
i += 1;
}
val = unescape_attr(rest.get(v0..i).unwrap_or(""));
}
}
}
if attrs.len() < 64 {
attrs.push((an, val));
}
}
self.pos += i.min(rest.len());
Some(Tok::Start { name, attrs, self_close })
Some(match self.inner.next()? {
Token::Text(t) => Tok::Text(unescape(t)),
Token::Start(tag) => Tok::Start {
attrs: tag.attrs.iter().map(|(n, v)| (n.to_ascii_lowercase(), unescape_attr(v))).collect(),
name: tag.name,
self_close: tag.self_close,
},
Token::End(name) => Tok::End(name),
})
}
}
crates/html/src/doc/math_ruby.rs
+6 -6
View File
@@ -3,8 +3,8 @@ use super::lexer::{Lexer, Tok};
/// A `<math>` element (#816): its tokens up to the matching end tag, as a tree. Nesting and size
/// are bounded; an unclosed element ends with the input.
pub(super) fn read_math(lx: &mut Lexer, name: String, attrs: Vec<(String, String)>) -> crate::mathml::El {
use crate::mathml::{El, Item, MAX_NODES, local};
pub(super) fn read_math(lx: &mut Lexer, name: String, attrs: Vec<(String, String)>) -> craft_math::mathml::El {
use craft_math::mathml::{El, Item, MAX_NODES, local};
let mut stack: Vec<El> = vec![El { name, attrs, children: Vec::new() }];
let mut count = 0usize;
let close = |stack: &mut Vec<El>| {
@@ -32,7 +32,7 @@ pub(super) fn read_math(lx: &mut Lexer, name: String, attrs: Vec<(String, String
}
Tok::Start { name, attrs, self_close } => {
let el = El { name, attrs, children: Vec::new() };
if self_close || stack.len() > wordcraft_doc::math::MAX_DEPTH {
if self_close || stack.len() > craft_wordmodel::math::MAX_DEPTH {
if let Some(top) = stack.last_mut() {
top.children.push(Item::El(el));
}
@@ -63,7 +63,7 @@ pub(super) fn read_math(lx: &mut Lexer, name: String, attrs: Vec<(String, String
/// Skip tokens (starting with `first`) up to and including the `</math>` that closes the element
/// being read; nested `<math>` elements are counted. An unclosed element ends with the input.
fn skip_math(lx: &mut Lexer, first: Tok) {
use crate::mathml::local;
use craft_math::mathml::local;
let mut depth = 0usize;
let mut t = Some(first);
while let Some(tok) = t {
@@ -122,11 +122,11 @@ pub(super) fn read_ruby(lx: &mut Lexer) -> RubyText {
let mut count = 0usize;
while count < MAX_TOKENS {
count += 1;
let start = lx.pos;
let start = lx.pos();
let Some(t) = lx.next() else { break };
if nested == 0 && ends_ruby(&t) {
// The boundary ends the element: give the token back to the main reader.
lx.pos = start;
lx.set_pos(start);
break;
}
match t {
crates/html/src/doc/tests.rs
+37 -37
View File
@@ -10,8 +10,8 @@ fn numeric_entity_rejects_sign() {
#[test]
fn important_declarations_win_the_cascade() {
let color = |html: &str| fmt_of(paras(&parse(html))[0]).color;
let red = Some(wordcraft_doc::Rgb(255, 0, 0));
let blue = Some(wordcraft_doc::Rgb(0, 0, 255));
let red = Some(craft_wordmodel::Rgb(255, 0, 0));
let blue = Some(craft_wordmodel::Rgb(0, 0, 255));
assert_eq!(color("<style>p { color: red }</style><p style='color: blue'>x</p>"), blue);
assert_eq!(color("<style>p { color: red !important }</style><p style='color: blue'>x</p>"), red);
assert_eq!(color("<style>p { color: red !important }</style><p style='color: blue !important'>x</p>"), blue);
@@ -39,9 +39,9 @@ fn texts(f: &Flow) -> Vec<String> {
}
fn table_style(html: &str) -> Option<String> {
let doc = crate::model::to_doc(&parse(html));
let doc = craft_flow::to_doc(&parse(html));
doc.body.iter().find_map(|b| match &**b {
wordcraft_doc::Block::Table(t) => Some(t.props.style.clone().unwrap_or_default()),
craft_wordmodel::Block::Table(t) => Some(t.props.style.clone().unwrap_or_default()),
_ => None,
})
}
@@ -67,7 +67,7 @@ fn borderless_tables_export_without_cell_borders() {
("<table><tr><td>A<td>B</table>", false),
("<table border=1><tr><td>A<td>B</table>", true),
] {
let out = export(&crate::model::to_doc(&parse(src)));
let out = export(&craft_flow::to_doc(&parse(src)));
assert_eq!(out.contains("border:1px"), grid, "{src}: {out}");
assert_eq!(table_style(&out).as_deref(), Some(if grid { "TableGrid" } else { "" }), "{src}");
assert!(out.contains(">A</td>") && out.contains(">B</td>"), "{out}");
@@ -157,7 +157,7 @@ fn styles_and_links() {
let f = parse("<p style='text-align:center'><span style=\"color:#ff0000;font-weight:700\">red</span> <a href=\"http://x\">l</a><br>n</p>");
let FBlock::Para(p) = &f.blocks[0] else { panic!() };
assert_eq!(p.align, Some(Align::Center));
assert!(matches!(&p.inlines[0], Inline::Text(t, f) if t == "red" && f.bold && f.color == Some(wordcraft_doc::Rgb(255,0,0))));
assert!(matches!(&p.inlines[0], Inline::Text(t, f) if t == "red" && f.bold && f.color == Some(craft_wordmodel::Rgb(255,0,0))));
assert!(p.inlines.iter().any(|i| matches!(i, Inline::Text(t, f) if t == "l" && f.link.as_deref() == Some("http://x"))));
assert!(p.text().ends_with("l\nn"));
}
@@ -369,14 +369,14 @@ fn font_names_are_written_as_css_strings() {
/// A document that reads "I read " then a phonetic guide over `base`, then " daily.".
fn ruby_doc(base: &str, ruby: &str) -> Document {
let props = wordcraft_doc::CharProps::default();
let mut p = wordcraft_doc::Paragraph::with_text("I read ", props.clone());
let props = craft_wordmodel::CharProps::default();
let mut p = craft_wordmodel::Paragraph::with_text("I read ", props.clone());
let n = p.len();
p.insert_object(n, crate::model::ruby_object(base.into(), ruby.into(), 20.0), &props).unwrap();
p.insert_object(n, craft_flow::ruby_object(base.into(), ruby.into(), 20.0), &props).unwrap();
let n = p.len();
p.insert_text(n, " daily.", &props).unwrap();
let mut d = Document::new();
d.body = vec![wordcraft_doc::para_block(p)];
d.body = vec![craft_wordmodel::para_block(p)];
d
}
@@ -394,8 +394,8 @@ fn ruby_of(f: &Flow) -> Option<(String, String)> {
/// The first document ruby: its base text and reading.
fn doc_ruby(d: &Document) -> Option<(String, String)> {
d.body.iter().find_map(|b| match &**b {
wordcraft_doc::Block::Para(p) => p.objects.iter().find_map(|o| match o {
wordcraft_doc::InlineObject::Ruby { base, ruby, .. } => Some((base.clone(), ruby.clone())),
craft_wordmodel::Block::Para(p) => p.objects.iter().find_map(|o| match o {
craft_wordmodel::InlineObject::Ruby { base, ruby, .. } => Some((base.clone(), ruby.clone())),
_ => None,
}),
_ => None,
@@ -403,10 +403,10 @@ fn doc_ruby(d: &Document) -> Option<(String, String)> {
}
/// The first document ruby: base size, ruby size, raise and alignment.
fn doc_ruby_attrs(d: &Document) -> Option<(f32, f32, f32, wordcraft_doc::para::RubyAlign)> {
fn doc_ruby_attrs(d: &Document) -> Option<(f32, f32, f32, craft_wordmodel::para::RubyAlign)> {
d.body.iter().find_map(|b| match &**b {
wordcraft_doc::Block::Para(p) => p.objects.iter().find_map(|o| match o {
wordcraft_doc::InlineObject::Ruby { base_size, size, raise, align, .. } => Some((*base_size, *size, *raise, *align)),
craft_wordmodel::Block::Para(p) => p.objects.iter().find_map(|o| match o {
craft_wordmodel::InlineObject::Ruby { base_size, size, raise, align, .. } => Some((*base_size, *size, *raise, *align)),
_ => None,
}),
_ => None,
@@ -432,7 +432,7 @@ fn ruby_imports_with_its_surrounding_text() {
assert!(matches!(&p.inlines[0], Inline::Text(t, _) if t == "I read "), "{:?}", p.inlines);
assert!(matches!(p.inlines.last(), Some(Inline::Text(t, _)) if t == " daily."), "{:?}", p.inlines);
// One document object, holding the base text and the reading.
assert_eq!(doc_ruby(&crate::model::to_doc(&f)), Some(("漢字".into(), "かんじ".into())));
assert_eq!(doc_ruby(&craft_flow::to_doc(&f)), Some(("漢字".into(), "かんじ".into())));
}
#[test]
@@ -442,19 +442,19 @@ fn ruby_rb_and_rp_fallbacks_import_as_one_guide() {
let f = parse("<p><ruby><rb>漢</rb><rb>字</rb><rp>(</rp><rt>かんじ</rt><rp>)</rp></ruby></p>");
assert_eq!(ruby_of(&f), Some(("漢字".into(), "かんじ".into())));
assert_eq!(texts(&f), vec!["NormalNone:漢字"]);
assert_eq!(doc_ruby(&crate::model::to_doc(&f)), Some(("漢字".into(), "かんじ".into())));
assert_eq!(doc_ruby(&craft_flow::to_doc(&f)), Some(("漢字".into(), "かんじ".into())));
}
#[test]
fn ruby_import_uses_the_phonetic_guide_defaults() {
// HTML states no alignment, size or raise: the Phonetic Guide's own defaults over a
// 20 pt run (half the base size, raised clear of the base text, centred)…
let d = crate::html::import("<p style='font-size:20pt'><ruby>漢字<rt>かんじ</rt></ruby></p>");
let d = crate::doc::import("<p style='font-size:20pt'><ruby>漢字<rt>かんじ</rt></ruby></p>");
let Some((base_size, size, raise, align)) = doc_ruby_attrs(&d) else { panic!("no ruby") };
assert_eq!((base_size, size, align.ooxml()), (20.0, 10.0, "center"));
assert!((raise - 18.0).abs() < 0.001, "{raise}");
// …and the guide's 10.5 pt fallback base size when the source states none.
let d = crate::html::import("<p><ruby>漢字<rt>かんじ</rt></ruby></p>");
let d = crate::doc::import("<p><ruby>漢字<rt>かんじ</rt></ruby></p>");
let Some((base_size, size, raise, _)) = doc_ruby_attrs(&d) else { panic!("no ruby") };
assert_eq!((base_size, size), (10.5, 5.0));
assert!((raise - 9.4).abs() < 0.001, "{raise}");
@@ -464,19 +464,19 @@ fn ruby_import_uses_the_phonetic_guide_defaults() {
fn ruby_survives_an_html_round_trip() {
let html = export(&ruby_doc("漢字", "かんじ"));
assert_eq!(ruby_of(&parse(&html)), Some(("漢字".into(), "かんじ".into())));
let back = crate::html::import(&html);
let back = crate::doc::import(&html);
assert_eq!(doc_ruby(&back), Some(("漢字".into(), "かんじ".into())));
assert_eq!(back.plain_text(Default::default()), "I read 漢字 daily.");
}
#[test]
fn ruby_keeps_the_run_formatting() {
let props = wordcraft_doc::CharProps { bold: Some(true), ..Default::default() };
let mut p = wordcraft_doc::Paragraph::with_text("I read ", props.clone());
let props = craft_wordmodel::CharProps { bold: Some(true), ..Default::default() };
let mut p = craft_wordmodel::Paragraph::with_text("I read ", props.clone());
let n = p.len();
p.insert_object(n, crate::model::ruby_object("漢字".into(), "かんじ".into(), 20.0), &props).unwrap();
p.insert_object(n, craft_flow::ruby_object("漢字".into(), "かんじ".into(), 20.0), &props).unwrap();
let mut d = Document::new();
d.body = vec![wordcraft_doc::para_block(p)];
d.body = vec![craft_wordmodel::para_block(p)];
assert!(export(&d).contains("<strong><ruby>漢字<rt>かんじ</rt></ruby></strong>"), "{}", export(&d));
// Importing it back keeps the formatting for the base text and the reading.
let f = parse("<p><strong><ruby>漢字<rt>かんじ</rt></ruby></strong></p>");
@@ -533,7 +533,7 @@ fn css_important_casing_and_spacing() {
assert_eq!(p.align, Some(Align::Center));
match &p.inlines[0] {
Inline::Text(_, fmt) => {
assert_eq!(fmt.color, Some(wordcraft_doc::props::Rgb(0xFF, 0x00, 0x00)));
assert_eq!(fmt.color, Some(craft_wordmodel::props::Rgb(0xFF, 0x00, 0x00)));
assert_eq!(fmt.size, Some(16.0));
assert!(fmt.bold);
}
@@ -548,8 +548,8 @@ fn table_cell_vertical_alignment() {
assert_eq!(t.rows[0][0].valign, Some(VAlign::Top));
assert_eq!(t.rows[0][1].valign, Some(VAlign::Center));
assert_eq!(t.rows[0][2].valign, Some(VAlign::Bottom));
let doc = crate::model::to_doc(&f);
let wordcraft_doc::Block::Table(dt) = &*doc.body[0] else { panic!() };
let doc = craft_flow::to_doc(&f);
let craft_wordmodel::Block::Table(dt) = &*doc.body[0] else { panic!() };
assert_eq!(dt.rows[0].cells[0].props.valign, Some(VAlign::Top));
assert_eq!(dt.rows[0].cells[1].props.valign, Some(VAlign::Center));
assert_eq!(dt.rows[0].cells[2].props.valign, Some(VAlign::Bottom));
@@ -595,7 +595,7 @@ fn font_size_at_the_integer_limits_does_not_overflow() {
#[test]
fn empty_comments_do_not_swallow_the_rest() {
for html in ["<p>a<!-->b</p><p>c</p>", "<p>a<!--->b</p><p>c</p>"] {
let doc = crate::html::import(html);
let doc = crate::doc::import(html);
assert_eq!(doc.plain_text(Default::default()), "ab\nc", "{html}");
}
}
@@ -616,7 +616,7 @@ fn style_sheet_rules_apply_in_cascade_order() {
);
let ps = paras(&f);
let (plain, big, inline, digits) = (fmt_of(ps[0]), fmt_of(ps[1]), fmt_of(ps[2]), fmt_of(ps[3]));
assert_eq!((plain.color, plain.size, plain.bold, plain.italic), (Some(wordcraft_doc::Rgb(0, 255, 0)), Some(10.0), false, false));
assert_eq!((plain.color, plain.size, plain.bold, plain.italic), (Some(craft_wordmodel::Rgb(0, 255, 0)), Some(10.0), false, false));
assert_eq!((big.size, big.bold), (Some(20.0), true));
assert_eq!(inline.size, Some(8.0));
assert_eq!(ps[3].align, Some(Align::Center));
@@ -662,7 +662,7 @@ fn a_page_opens_as_word_opens_one() {
let FBlock::Para(cp) = &t.rows[0][0].blocks[0] else { panic!() };
assert_eq!(cp.spacing.line, Some(LineSpacing::Multiple(1.0)));
let head = fmt_of(ps[1]);
assert_eq!((head.bold, head.size, head.color), (true, Some(24.0), Some(wordcraft_doc::Rgb(0, 0, 0))));
assert_eq!((head.bold, head.size, head.color), (true, Some(24.0), Some(craft_wordmodel::Rgb(0, 0, 0))));
// A colour the heading inherits stays.
// A generic family names a face on a page; pasted text keeps the document's font.
let generic = parse_in("<p style='font-family: sans-serif'>a</p><p style='font-family: serif'>b</p>", &|_| None, true);
@@ -670,7 +670,7 @@ fn a_page_opens_as_word_opens_one() {
assert_eq!(fonts, [Some("Arial".to_string()), Some(PAGE_FONT.to_string())]);
assert_eq!(fmt_of(paras(&parse("<p style='font-family: sans-serif'>a</p>"))[0]).font, None);
let inherited = parse_in("<body style='color: #000080'><h1>Head</h1></body>", &|_| None, true);
assert_eq!(fmt_of(paras(&inherited)[0]).color, Some(wordcraft_doc::Rgb(0, 0, 0x80)));
assert_eq!(fmt_of(paras(&inherited)[0]).color, Some(craft_wordmodel::Rgb(0, 0, 0x80)));
// Automatic spacing where a p or heading leaves its margins out (layout drops it at the
// top of the page and of a cell).
let auto = |p: &Para| (p.spacing.before_auto, p.spacing.after_auto, p.spacing.before, p.spacing.after);
@@ -684,7 +684,7 @@ fn a_page_opens_as_word_opens_one() {
// The document says so, and the empty paragraph's mark keeps its size.
let d = model::to_doc(&f);
let props = |i: usize| match &*d.body[i] {
wordcraft_doc::Block::Para(p) => p.clone(),
craft_wordmodel::Block::Para(p) => p.clone(),
b => panic!("{b:?}"),
};
assert_eq!((props(0).props.space_before_auto, props(0).props.space_after_auto), (Some(true), Some(true)));
@@ -719,16 +719,16 @@ fn cells_give_their_columns_widths() {
#[test]
fn cells_take_their_css_borders() {
use wordcraft_doc::props::{Border, BorderStyle};
use craft_wordmodel::props::{Border, BorderStyle};
let f = parse(
"<style>td.b { border-top: solid windowtext 1px; border-bottom: 2pt double #ff0000; border-left: none; border-right: 0px solid; vertical-align: bottom }</style><table><tr><td class=b>a</td><td>b</td></tr></table>",
);
let FBlock::Table(t) = &f.blocks[0] else { panic!() };
// The style sheet's other cell properties apply too.
assert_eq!(t.rows[0][0].valign, Some(wordcraft_doc::props::VAlign::Bottom));
assert_eq!(t.rows[0][0].valign, Some(craft_wordmodel::props::VAlign::Bottom));
let b = t.rows[0][0].borders.expect("borders");
assert_eq!(b.top, Some(Border::single(0.75)));
assert_eq!(b.bottom, Some(Border { style: BorderStyle::Double, width: 2.0, color: Some(wordcraft_doc::Rgb(255, 0, 0)), space: 0.0 }));
assert_eq!(b.bottom, Some(Border { style: BorderStyle::Double, width: 2.0, color: Some(craft_wordmodel::Rgb(255, 0, 0)), space: 0.0 }));
assert_eq!(b.left.map(|l| l.style), Some(BorderStyle::None));
// A border of no width is none (OOXML has no zero-width line).
assert_eq!(b.right.map(|r| r.style), Some(BorderStyle::None));
@@ -745,7 +745,7 @@ fn style_sheet_edge_cases() {
);
let ps = paras(&f);
// A rule right after statement at-rules still applies.
assert_eq!(fmt_of(ps[0]).color, Some(wordcraft_doc::Rgb(0, 0, 255)));
assert_eq!(fmt_of(ps[0]).color, Some(craft_wordmodel::Rgb(0, 0, 255)));
// The body's margins aren't the paragraphs' indents.
assert_eq!((ps[1].spacing.left, ps[1].spacing.before), (None, None));
// Bare numbers, percentages and keywords state no margin: automatic spacing stays.
@@ -753,7 +753,7 @@ fn style_sheet_edge_cases() {
// The `style` attribute wins even over a sheet with more declarations than are read.
let many = (0..600).map(|i| format!("x-{i}: 1")).collect::<Vec<_>>().join(";");
let f = parse(&format!("<style>p {{ {many}; color: #ff0000 }}</style><p style='color: #00ff00'>c</p>"));
assert_eq!(fmt_of(paras(&f)[0]).color, Some(wordcraft_doc::Rgb(0, 255, 0)));
assert_eq!(fmt_of(paras(&f)[0]).color, Some(craft_wordmodel::Rgb(0, 255, 0)));
}
#[test]
crates/html/src/lex.rs
+12
View File
@@ -110,6 +110,18 @@ impl<'a, 'o> Lexer<'a, 'o> {
self.pos
}
/// Goes back (or forward) to `pos`, a byte offset an earlier [`Lexer::pos`] gave, to read
/// from there again: for a reader that looked at a token and hands it back. An offset past the
/// end, or inside a character, moves to the next character boundary.
pub fn set_pos(&mut self, pos: usize) {
let mut pos = pos.min(self.src.len());
while !self.src.is_char_boundary(pos) {
pos += 1;
}
self.pos = pos;
self.raw = None;
}
/// The raw content of element `name`, whose start tag was the last token: everything up to
/// its end tag (any letter case), or to the end of the input when it has none. The end tag
/// is consumed. For `style` and `script`, whose content is not markup.
crates/html/src/lex/tests.rs
+15
View File
@@ -175,3 +175,18 @@ proptest::proptest! {
}
}
}
#[test]
fn a_token_can_be_read_again() {
let mut l = Lexer::new("é<b>x</b>");
assert_eq!(l.next(), Some(Token::Text("é")));
let at = l.pos();
assert!(matches!(l.next(), Some(Token::Start(t)) if t.name == "b"));
l.set_pos(at);
assert!(matches!(l.next(), Some(Token::Start(t)) if t.name == "b"));
// Inside a character or past the end: the next boundary.
l.set_pos(1);
assert!(matches!(l.next(), Some(Token::Start(t)) if t.name == "b"));
l.set_pos(1000);
assert_eq!(l.next(), None);
}
crates/html/src/lib.rs
+2
View File
@@ -34,6 +34,8 @@
#[doc = include_str!("../README.md")]
struct ReadmeExamples;
#[cfg(feature = "flow")]
pub mod doc;
mod entity;
mod lex;
crates/html/tests/doc.rs
+93
View File
@@ -0,0 +1,93 @@
//! HTML documents (feature `flow`): the HTML parts of WordCraft's cross-format and MathML tests
//! (`crates/formats/src/tests.rs`, `mathml.rs`): equations through HTML, hostile input, entities.
use craft_flow::{FBlock, Inline};
use craft_html::doc::{import, inlines_html, parse};
use std::time::{Duration, Instant};
fn equations(html: &str) -> Vec<(String, bool)> {
let mut out = Vec::new();
for b in &parse(html).blocks {
if let FBlock::Para(p) = b {
for i in &p.inlines {
if let Inline::Equation { linear, display } = i {
out.push((linear.clone(), *display));
}
}
}
}
out
}
#[test]
fn html_export_round_trips_equations() {
for (linear, display) in [("y=2x_s", false), ("x=(-b±√(b^2-4ac))/2a", true), ("∑_(k=1)^n▒k", false), ("■(a&b@c&d)", false)] {
let html = inlines_html(&[Inline::Equation { linear: linear.into(), display }]);
assert!(html.starts_with("<math"), "{html}");
assert_eq!(equations(&format!("<p>Before {html} after</p>")), vec![(linear.to_string(), display)], "{html}");
}
// Text after the equation still arrives; an unclosed element ends with the input.
assert!(import("<p><math><mi>x</mi></math> tail</p>").plain_text(Default::default()).contains("tail"));
assert!(!equations("<p><math><mfrac><mi>x").is_empty());
let deep = format!("<math>{}<mi>x</mi>{}</math>", "<mrow>".repeat(5000), "</mrow>".repeat(5000));
assert!(equations(&deep).len() <= 1);
}
#[test]
fn math_over_the_token_cap_is_skipped_to_its_end() {
// Past the cap, the rest of the element is dropped, not read as body text.
let html = format!("<p><math>{}<mrow><mi>z</mi></mrow></math> tail</p>", "<mi>y</mi>".repeat(craft_math::mathml::MAX_NODES * 2));
let mut text = String::new();
for b in &parse(&html).blocks {
if let FBlock::Para(p) = b {
for i in &p.inlines {
if let Inline::Text(t, _) = i {
text.push_str(t);
}
}
}
}
assert_eq!(text.trim(), "tail");
}
#[test]
fn deep_nesting_is_bounded() {
let html = "<div><table><tr><td>".repeat(5000) + "x";
assert!(import(&html).plain_text(Default::default()).contains('x'));
}
#[test]
fn entities_in_text() {
let d = import("<p>&lt;&amp;&#65;&eacute; &nosuchentity; &#x41;</p>");
assert_eq!(d.plain_text(Default::default()).trim(), "<&Aé &nosuchentity; A");
// An entity name longer than the scan window is left alone.
let long = format!("<p>&{};</p>", "a".repeat(40));
assert!(import(&long).plain_text(Default::default()).contains(&"a".repeat(40)));
}
/// A run of `&#` is read in linear time: four times as much takes well under the sixteen times
/// as long a quadratic read would (a ratio, so a busy machine does not fail it).
#[test]
fn a_run_of_references_takes_linear_time() {
let fastest = |n: usize| {
let text = "&#".repeat(n);
(0..3)
.map(|_| {
let t = Instant::now();
let _ = import(&text).plain_text(Default::default());
t.elapsed()
})
.min()
.unwrap_or_default()
};
let (small, large) = (fastest(125_000), fastest(500_000));
assert!(large < small * 8 + Duration::from_millis(50), "{small:?} for 125,000, {large:?} for 500,000");
}
proptest::proptest! {
#[test]
fn fuzz_html(s in "(<(/)?(p|b|i|ul|ol|li|table|tr|td|th|pre|blockquote|a href=x|img src=data:,|br|h2|span style='color:red')>|[a-z &;#<>\"'=]){0,120}") {
let d = import(&s);
let _ = craft_html::doc::export(&d);
}
}