Files
craft-libs/crates/latex/src/parser.rs
T
Dylan M. Taylorandshuff57 bdf06f1029 Add craft-latex: LaTeX documents read and written, on craft-flow (stacked) (#1038)
* Add craft-wordmodel: WordCraft's document model without its editing code

WordCraft's crates/doc (wordcraft-doc, at 07f97737) moved as it is, so the DOCX,
RTF, ODT and other text-format crates can read into and write from one model.
Module paths, type names and serde attributes are unchanged: an app that
re-exports the crate keeps its native file format.

Left in the app: range editing (edit.rs), IME text and equation editing. Not
model, and already in craft-libs: code pages (craft-rtf), the ZIP entry-count
check (craft-opc), the compound-file guard (craft-cfb). The paragraph walks
moved to walk.rs and the unit helpers of wordcraft-geom to geom.rs.

* craft-wordmodel: math_tree, the arguments of an equation node and a walk over its runs

The DOCX reader and writer walk an equation's runs. The walk and the child
accessors it uses were in WordCraft's equation editor (math_edit.rs), which
stays in the app; they are structure, not editing.

* craft-mime: detect, for bytes that name no character set

is_text, decode (BOM, unmarked UTF-16, UTF-8, else the legacy encoding
the bytes look like), legacy and legacy_label, behind a detect feature
that brings in chardetng. Moved from WordCraft's crates/formats/src/txt.rs
at 06c2ae6f with its text-sniffing tests. Also charset::is_known, so a
caller can tell a label decode would read as UTF-8 from one it knows.

* craft-wordmodel: keep the document operations (edit.rs) with the model

The DOCX and RTF readers use them: alternative-format chunks are merged into
a document with adopt_blocks, and a reader formats ranges with format_range
and inserts objects. They operate on the document, not on a view, so they
belong beside it. The tests that drove them come back with them; the
paragraph walks stay in walk.rs and edit re-exports them.

base64 0.23, as WordCraft and craft-offcrypto use.

* Add craft-math: the equation tree with OMML, MathML, LaTeX and UnicodeMath

The math tree, LaTeX, linear-format, symbol and tree-walk modules come
from craft-wordmodel (storytold/craft-libs#963, WordCraft's crates/doc at
07f97737) unchanged, with the run colour (Rgb) and the symbol-font tables
they use. MathML is WordCraft's crates/formats/src/mathml.rs and OMML is
WordCraft's crates/docx/src/read/math.rs and write/math.rs (06c2ae6f /
07f97737), moved onto craft-xml. The HTML-specific MathML tests stay in
WordCraft; the rest moved with the code.

mathml::parse and omml::parse/read_str/write_str are new: string entry
points for callers that have no tree of their own.

* craft-math: read MathLive's and other spellings of LaTeX math commands

\abs{...} (an absolute value), \lvert \rvert \lVert \rVert, \nparallel,
\backprime, \infinity, \cross, \differentialD, \exponentialE and
\imaginaryI (MathLive's d, upright e and i), \emph, \textsf, \hbox,
\mathnormal and \underleftarrow / \underrightarrow /
\underleftrightarrow were unknown and read as literal text. They are read
as equaSHion's typed-input parser reads them (storytold/craft-libs#29).

* craft-math: equaSHion's equation corpus as test data

corpus/equations.tsv, corpus/latex.tsv and the OMML samples of
storytold/craft-libs#29 (7a6ef6b9), unmodified.

* craft-math: corpus round trips, GridCraft's OMML cases, entry points and limits tested; README

The corpus equations go through OMML, MathML and LaTeX and back; the OMML
samples are written back as kept and from the tree. GridCraft's OMML to
LaTeX test cases (crates/xlsx/src/omml.rs) give LaTeX that reads back the
same through the tree. omml::parse and read_str are tested against other
prefixes, the strict namespace, kept OMML, junk, the element cap and deep
nesting.

* Add craft-flow: the flow model the text formats map to, on craft-wordmodel

WordCraft's crates/formats/src/model (07f97737) moved as it is: Flow,
FBlock, Para, Inline, Fmt, tables and cells, pictures, to_doc and
from_doc, picture sniffing, colours, base64 and the Dropped report. Only
the paths changed (wordcraft_doc is craft_wordmodel); the report helpers
are public so the format crates can call them; the image re-encoding
helper stays with WordCraft's RTF writer.

* Add craft-markdown: Markdown read and written, on craft-flow

WordCraft's crates/formats/src/markdown (07f97737) with its 35 tests,
moved onto craft-flow and craft-wordmodel. Character references come from
craft-html. The table cell budget lives in craft-flow, which names it in
its warning. Two test lines that wrote plain text through WordCraft's txt
module stay in WordCraft.

* WIP: craft-latex sources from WordCraft (unmodified)

* Add craft-latex: LaTeX documents read and written, on craft-flow

WordCraft's crates/formats/src/latex (07f97737) with its 21 tests, moved
onto craft-flow and craft-wordmodel; bytes are decoded by craft-mime's
detect. The LaTeX parts of WordCraft's cross-format tests (nesting,
column specs, labels, no TeX commands from the document, the fuzz and
round-trip properties) come along as tests/hostile.rs. The ligature test
builds its three-font paragraph as a flow instead of reading HTML.

* craft-math: latex_text, LaTeX math and the linear format converted as text

WordCraft's crates/formats/src/latex/math.rs (07f97737): the converter its
LaTeX document reader and writer use, which keeps the source's spacing.
It moves here so craft-math is the one place that converts LaTeX math;
craft-latex calls it, and its output does not change. Its tests (from
WordCraft's latex tests and cross-format tests) come along.

* craft-math: latex_text's group finders are public, for craft-latex's length parser

* craft-latex: math through craft_math::latex_text; label timing test by ratio

The document math converter moved to craft-math (#1040) unchanged; this
crate calls it there (and its length parser uses its group finders), so
export output is the same. The label test compared one read with a
ten-second wall clock and failed on a loaded machine; it now compares the
fastest of three reads of 25,000 and 100,000 labels, which a linear read
keeps near 4x and a quadratic one near 16x.

---------

Co-authored-by: shuff57 <62350898+shuff57@users.noreply.github.com>
2026-11-11 20:58:36 -05:00

915 lines
33 KiB
Rust

use super::lex::{Tok, accent_mark, compose, ligatures, text_symbol, tokenize};
use super::limits::{MAX_ANCHOR_RUN, MAX_NEST, MAX_SOURCE, MAX_SPEC};
use super::table::{raw_text, source_text, table};
use super::units::{length_pt, parse_color, size_of};
use super::{Align, Document, FBlock, Flow, Fmt, Inline, Kind, ListInfo, Para, model};
use craft_math::latex_text::latex_to_linear;
// =============================================================================================
// Import
/// Import a LaTeX source file.
pub fn import(bytes: &[u8]) -> Document {
model::begin_import();
let mut d = model::to_doc(&parse(&craft_mime::detect::decode(bytes.get(..MAX_SOURCE).unwrap_or(bytes))));
d.ensure_nonempty();
d
}
/// Commands dropped together with their arguments (layout and definitions, not text).
const DROP_WITH_ARGS: &[&str] = &[
"documentclass",
"usepackage",
"RequirePackage",
"newcommand",
"renewcommand",
"providecommand",
"newcommand*",
"renewcommand*",
"providecommand*",
"newenvironment",
"renewenvironment",
"DeclareMathOperator",
"DeclareMathOperator*",
"def",
"let",
"setlength",
"addtolength",
"setcounter",
"addtocounter",
"stepcounter",
"refstepcounter",
"vspace",
"vspace*",
"hspace",
"hspace*",
"vskip",
"hskip",
"pagestyle",
"thispagestyle",
"pagenumbering",
"bibliographystyle",
"bibliography",
"addbibresource",
"printbibliography",
"input",
"include",
"includeonly",
"graphicspath",
"hypersetup",
"definecolor",
"geometry",
"numberwithin",
"newtheorem",
"newtheorem*",
"theoremstyle",
"phantom",
"hphantom",
"vphantom",
"index",
"nocite",
"addcontentsline",
"markboth",
"markright",
"linespread",
"setstretch",
"fancyhf",
"fancyhead",
"fancyfoot",
"lstset",
"captionsetup",
"setmainfont",
"setsansfont",
"setmonofont",
"newfontfamily",
"date",
"thanks",
"maketitle*",
"tableofcontents",
"listoffigures",
"listoftables",
"appendix",
"frontmatter",
"mainmatter",
"backmatter",
"cline",
"hline",
"toprule",
"midrule",
"bottomrule",
"cmidrule",
"addlinespace",
"endhead",
"endfirsthead",
"endfoot",
"endlastfoot",
"rowcolor",
"arraystretch",
"smallskip",
"medskip",
"bigskip",
"today",
"and",
];
/// Paragraph-level context from the enclosing environments.
#[derive(Clone, Copy, Default)]
struct Ctx {
kind: Kind,
align: Option<Align>,
}
pub(super) struct Parser<'a> {
pub(super) t: &'a [Tok],
pub(super) pos: usize,
blocks: Vec<FBlock>,
para: Para,
/// Character formatting in effect (groups save and restore it).
fmt: Fmt,
/// Paragraph context of the enclosing environments.
ctx: Ctx,
/// Text that followed an optional argument's `]` in the same token, emitted next.
pending: Option<String>,
/// Open lists: ordered?
lists: Vec<bool>,
page_break: bool,
depth: usize,
meta: model::Meta,
/// Sectioning levels in use: `\part`/`\chapter` push `\section` down.
top: u8,
}
impl<'a> Parser<'a> {
pub(super) fn new(t: &'a [Tok], depth: usize) -> Parser<'a> {
Parser {
t,
pos: 0,
blocks: Vec::new(),
para: Para::default(),
fmt: Fmt::default(),
ctx: Ctx::default(),
pending: None,
lists: Vec::new(),
page_break: false,
depth,
meta: Default::default(),
top: 0,
}
}
fn fmt(&self) -> Fmt {
self.fmt.clone()
}
fn fmt_mut(&mut self) -> &mut Fmt {
&mut self.fmt
}
fn ctx(&self) -> Ctx {
self.ctx
}
/// A bookmark here. `text` looks back past the bookmarks that end the paragraph to see what
/// the last text was, so a run of them with nothing between is kept short: after
/// [`MAX_ANCHOR_RUN`] in a row further ones are left out (thousands of labels with no text
/// between them made reading a file take time quadratic in its size).
fn anchor(&mut self, name: String) {
let run = self.para.inlines.iter().rev().take(MAX_ANCHOR_RUN).take_while(|i| matches!(i, Inline::Anchor(_))).count();
if run < MAX_ANCHOR_RUN {
self.para.inlines.push(Inline::Anchor(name));
}
}
fn text(&mut self, s: &str) {
// Spaces collapse, and none start a paragraph (as in TeX).
let after_space = match self.para.inlines.iter().rev().find(|i| !matches!(i, Inline::Anchor(_))) {
Some(Inline::Text(t, _)) => t.is_empty() || t.ends_with([' ', '\n']),
Some(_) => false,
None => true,
};
let s = if after_space { s.trim_start_matches(' ') } else { s };
if s.is_empty() {
return;
}
let f = self.fmt();
self.para.push_text(s, &f);
}
/// End the current paragraph (if it has content).
fn flush(&mut self) {
let mut p = std::mem::take(&mut self.para);
p.trim();
if p.is_empty() && !p.inlines.iter().any(|i| matches!(i, Inline::Image(_) | Inline::Equation { .. })) {
// Keep a list item's position even when empty.
if p.list.is_none() {
return;
}
}
let c = self.ctx();
if p.kind == Kind::Normal {
p.kind = c.kind;
}
if p.align.is_none() {
p.align = c.align;
}
if self.page_break {
p.page_break = true;
self.page_break = false;
}
self.blocks.push(FBlock::Para(p));
}
pub(super) fn skip_spaces(&mut self) {
while let Some(Tok::Text(s)) = self.t.get(self.pos) {
if !s.trim().is_empty() {
break;
}
self.pos += 1;
}
}
/// The tokens of an optional `[…]` argument, if one follows.
pub(super) fn opt_arg(&mut self) -> Option<String> {
let save = self.pos;
self.skip_spaces();
let Some(Tok::Text(s)) = self.t.get(self.pos) else {
self.pos = save;
return None;
};
if !s.starts_with('[') {
self.pos = save;
return None;
}
// Collect raw text up to the matching `]` (no nested brackets across tokens).
let mut out = String::new();
let mut first = true;
while let Some(tok) = self.t.get(self.pos) {
match tok {
Tok::Text(s) => {
let s = if first { s.get(1..).unwrap_or("") } else { s.as_str() };
first = false;
if let Some(end) = s.find(']') {
out.push_str(s.get(..end).unwrap_or(""));
let rest = s.get(end + 1..).unwrap_or("").to_string();
self.pos += 1;
if !rest.trim().is_empty() {
// The text after `]` comes next (tokens are shared, so it waits here).
self.pending = Some(rest);
}
return Some(out);
}
out.push_str(s);
}
Tok::Open => out.push('{'),
Tok::Close => out.push('}'),
Tok::Cmd(c) => {
out.push('\\');
out.push_str(c);
}
Tok::Par => break,
_ => {}
}
self.pos += 1;
}
Some(out)
}
/// The tokens of a `{…}` argument (or a single token), consumed.
pub(super) fn arg_tokens(&mut self) -> &'a [Tok] {
self.skip_spaces();
match self.t.get(self.pos) {
Some(Tok::Open) => {
let start = self.pos + 1;
let mut depth = 0usize;
let mut i = self.pos;
while let Some(tok) = self.t.get(i) {
match tok {
Tok::Open => depth += 1,
Tok::Close => {
depth -= 1;
if depth == 0 {
self.pos = i + 1;
return self.t.get(start..i).unwrap_or_default();
}
}
_ => {}
}
i += 1;
}
self.pos = self.t.len();
self.t.get(start..).unwrap_or_default()
}
Some(Tok::Text(_)) => {
// A single character argument (`\textbf x`) is rare: take the whole token.
let i = self.pos;
self.pos += 1;
self.t.get(i..i + 1).unwrap_or_default()
}
Some(Tok::Cmd(_)) => {
let i = self.pos;
self.pos += 1;
self.t.get(i..i + 1).unwrap_or_default()
}
_ => &[],
}
}
/// An argument as plain text (no formatting): labels, URLs, colour names.
pub(super) fn arg_raw(&mut self) -> String {
raw_text(self.arg_tokens())
}
/// An argument as LaTeX source (braces kept): column specs, lengths.
fn arg_source(&mut self) -> String {
source_text(self.arg_tokens())
}
/// An argument's text as it would be shown (formatting dropped): titles and authors.
fn arg_text(&mut self) -> String {
let toks = self.arg_tokens();
let mut sub = Parser::new(toks, self.depth + 1);
if self.depth < MAX_NEST {
sub.run(None);
}
let (blocks, _) = sub.finish();
let text: Vec<String> = blocks.iter().filter_map(|b| if let FBlock::Para(p) = b { Some(p.text()) } else { None }).collect();
squash(&text.join(" "))
}
/// Parse `toks` inline into the current paragraph with formatting `f` applied.
fn inline_with(&mut self, toks: &'a [Tok], f: impl FnOnce(&mut Fmt)) {
if self.depth >= MAX_NEST {
let s = raw_text(toks);
self.text(&s);
return;
}
let saved_fmt = self.fmt.clone();
f(&mut self.fmt);
let (saved_t, saved_pos) = (self.t, self.pos);
self.t = toks;
self.pos = 0;
self.depth += 1;
self.run(None);
self.flush_pending();
self.depth -= 1;
self.t = saved_t;
self.pos = saved_pos;
self.fmt = saved_fmt;
}
/// Skip every `[…]` and `{…}` argument that follows (and, for definitions, the macro name).
fn drop_args(&mut self, name: &str) {
if matches!(name, "newcommand" | "renewcommand" | "providecommand" | "newcommand*" | "renewcommand*" | "providecommand*" | "def" | "let") {
self.skip_spaces();
if matches!(self.t.get(self.pos), Some(Tok::Cmd(_))) {
self.pos += 1;
}
if name == "let" {
self.skip_spaces();
if matches!(self.t.get(self.pos), Some(Tok::Cmd(_))) {
self.pos += 1;
}
return;
}
if name == "def" {
// `\def\x#1{…}`: the parameter text, then the body.
while matches!(self.t.get(self.pos), Some(Tok::Text(_))) {
self.pos += 1;
}
}
}
loop {
let save = self.pos;
self.skip_spaces();
match self.t.get(self.pos) {
Some(Tok::Open) => {
let _ = self.arg_tokens();
}
Some(Tok::Text(s)) if s.starts_with('[') => {
let _ = self.opt_arg();
if self.pending.is_some() {
return;
}
}
_ => {
self.pos = save;
return;
}
}
}
}
fn heading(&mut self, level: u8) {
let _ = self.opt_arg();
let toks = self.arg_tokens();
self.flush();
self.para = Para::new(Kind::Heading(level.clamp(1, 6)));
self.inline_with(toks, |_| {});
self.flush();
}
/// Parse until `\end{env}` (when given), a closing brace at this level, or the end.
fn flush_pending(&mut self) {
if let Some(p) = self.pending.take() {
self.text(&ligatures(&p));
}
}
pub(super) fn run(&mut self, env: Option<&str>) {
loop {
self.flush_pending();
let Some(tok) = self.t.get(self.pos) else { break };
self.pos += 1;
match tok {
Tok::Text(s) => {
let s = ligatures(s);
self.text(&s);
}
Tok::Par => self.flush(),
Tok::Tie => self.text("\u{A0}"),
Tok::Amp => self.text(" "),
Tok::Open => {
// A group scopes declarations (`{\bfseries …}`, `{\centering …\par}`):
// character formatting and paragraph alignment come back at its end.
let toks = {
self.pos -= 1;
self.arg_tokens()
};
let saved_ctx = self.ctx;
self.inline_with(toks, |_| {});
self.ctx = saved_ctx;
}
Tok::Close => {}
Tok::Math(m, display) => {
let linear = latex_to_linear(m);
if linear.is_empty() {
continue;
}
if *display {
self.flush();
self.para.inlines.push(Inline::Equation { linear, display: true });
self.flush();
} else {
self.para.inlines.push(Inline::Equation { linear, display: false });
}
}
Tok::Verbatim(v) => {
self.flush();
let v = v.strip_prefix('\n').unwrap_or(v);
let v = v.strip_suffix('\n').unwrap_or(v);
for line in v.lines() {
let mut p = Para::new(Kind::Code);
p.push_text(line.trim_end_matches('\r'), &Fmt::default());
if line.trim().is_empty() {
p.inlines.push(Inline::Text(String::new(), Fmt::default()));
}
self.blocks.push(FBlock::Para(p));
}
}
Tok::Verb(v) => {
let mut f = self.fmt();
f.code = true;
self.para.push_text(v, &f);
}
// Too deep: the environment's content goes on at this level.
Tok::Begin(_) if self.depth >= MAX_NEST => {}
Tok::Begin(name) => self.environment(name),
Tok::End(name) => {
if env == Some(name.as_str()) {
return;
}
// A stray `\end`: ignore.
}
Tok::Cmd(name) => self.command(name),
}
}
}
fn command(&mut self, name: &str) {
if let Some(sym) = text_symbol(name) {
if name == " " {
// A control space is never collapsed into the space before it.
let f = self.fmt();
self.para.push_text(sym, &f);
} else if name == "item" {
self.item();
} else if matches!(name, "centering" | "raggedleft" | "raggedright") {
let a = match name {
"centering" => Align::Center,
"raggedleft" => Align::Right,
_ => Align::Left,
};
self.ctx.align = Some(a);
} else {
self.text(sym);
}
return;
}
if let Some(mark) = accent_mark(name) {
// `\"o` takes one letter of the text after it; `\"{o}` the group.
self.skip_spaces();
let arg = match self.t.get(self.pos) {
Some(Tok::Text(t)) => {
self.pos += 1;
let mut chars = t.chars();
let first = chars.next().map(String::from).unwrap_or_default();
if !chars.as_str().is_empty() {
self.pending = Some(chars.as_str().to_string());
}
first
}
_ => self.arg_raw(),
};
let mut chars = arg.chars();
let s = match chars.next() {
Some(b) => format!("{}{}", compose(b, mark), chars.as_str()),
// An accent over nothing prints the accent alone: the spacing form of the mark.
None => match name {
"`" | "^" | "~" => name.to_string(),
"'" => "\u{B4}".to_string(),
"\"" => "\u{A8}".to_string(),
_ => mark.to_string(),
},
};
self.text(&s);
return;
}
if let Some(size) = size_of(name) {
self.fmt_mut().size = (size > 0.0).then_some(size);
return;
}
match name {
"\\" | "newline" | "linebreak" | "break" => {
let _ = self.opt_arg();
self.text("\n");
}
"part" | "part*" => self.heading(1),
"chapter" | "chapter*" => self.heading(self.top.max(1)),
"section" | "section*" => self.heading(1 + self.top),
"subsection" | "subsection*" => self.heading(2 + self.top),
"subsubsection" | "subsubsection*" => self.heading(3 + self.top),
"paragraph" | "paragraph*" => self.heading(4 + self.top),
"subparagraph" | "subparagraph*" => self.heading(5 + self.top),
"title" => self.meta.title = self.arg_text(),
"author" => self.meta.author = self.arg_text(),
"maketitle" => {
self.flush();
if !self.meta.title.is_empty() {
let mut p = Para::new(Kind::Title);
p.push_text(&self.meta.title, &Fmt::default());
self.blocks.push(FBlock::Para(p));
}
if !self.meta.author.is_empty() {
let mut p = Para { align: Some(Align::Center), ..Default::default() };
p.push_text(&self.meta.author, &Fmt::default());
self.blocks.push(FBlock::Para(p));
}
}
"newpage" | "clearpage" | "cleardoublepage" | "pagebreak" | "eject" => {
let _ = self.opt_arg();
self.flush();
self.page_break = true;
}
"par" => self.flush(),
"hrule" | "hrulefill" | "rule" => {
let _ = self.opt_arg();
let full = if name == "rule" {
let w = self.arg_source();
let _ = self.arg_raw();
w.contains("linewidth") || w.contains("textwidth") || w.contains("columnwidth")
} else {
true
};
if full {
self.flush();
self.blocks.push(FBlock::Para(Para::new(Kind::Rule)));
}
}
"textbf" => self.with_arg(|f| f.bold = true),
"textit" | "textsl" => self.with_arg(|f| f.italic = true),
"emph" => self.with_arg(|f| f.italic = !f.italic),
"underline" | "uline" | "ul" | "uuline" => self.with_arg(|f| f.underline = true),
"sout" | "st" | "xout" | "cancel" => self.with_arg(|f| f.strike = true),
"texttt" | "code" | "path" => self.with_arg(|f| f.code = true),
"textsuperscript" => self.with_arg(|f| f.sup = true),
"textsubscript" => self.with_arg(|f| f.sub = true),
"textmd" | "textup" | "textrm" | "textsf" | "textsc" | "textnormal" | "mbox" | "makebox" | "fbox" | "framebox" | "hbox" | "text"
| "textls" | "foreignlanguage" | "enquote" | "MakeUppercase" | "MakeLowercase" | "uppercase" | "lowercase" => {
if name == "foreignlanguage" {
let _ = self.arg_raw();
}
if matches!(name, "makebox" | "framebox") {
let _ = self.opt_arg();
let _ = self.opt_arg();
}
let quote = name == "enquote";
if quote {
self.text("“");
}
self.with_arg(|f| {
if name == "textmd" {
f.bold = false;
}
if name == "textup" {
f.italic = false;
}
if name == "textnormal" {
*f = Fmt { link: f.link.clone(), ..Default::default() };
}
});
if quote {
self.text("”");
}
}
"bfseries" | "bf" => self.fmt_mut().bold = true,
"itshape" | "it" | "slshape" | "sl" | "em" => self.fmt_mut().italic = true,
"ttfamily" | "tt" => self.fmt_mut().code = true,
"mdseries" => self.fmt_mut().bold = false,
"upshape" => self.fmt_mut().italic = false,
"rmfamily" | "sffamily" | "scshape" | "sc" | "rm" | "sf" | "normalfont*" => {}
"fontsize" => {
let s = self.arg_source();
let _ = self.arg_raw();
self.fmt_mut().size = length_pt(&s).or_else(|| s.trim().parse().ok()).filter(|v: &f32| *v >= 1.0 && *v <= 1638.0);
}
"textcolor" | "color" | "colorbox" | "highlight" | "hl" | "cellcolor" => {
let model = self.opt_arg();
let spec = if matches!(name, "hl" | "highlight") { "yellow".to_string() } else { self.arg_raw() };
let c = parse_color(model.as_deref(), &spec);
match name {
"color" => self.fmt_mut().color = c,
"cellcolor" => {}
"textcolor" => self.with_arg(move |f| f.color = c.or(f.color)),
_ => self.with_arg(move |f| f.background = c.or(f.background)),
}
}
"href" => {
let url = model::kept_link(&self.arg_raw());
self.with_arg(move |f| f.link = url);
}
"url" | "nolinkurl" => {
let url = self.arg_raw();
let mut f = self.fmt();
if name == "url" {
f.link = model::kept_link(&url);
}
self.para.push_text(&url, &f);
}
"hyperlink" => {
let target = self.arg_raw();
let link = model::kept_link(&format!("#{target}"));
self.with_arg(move |f| f.link = link);
}
"hypertarget" => {
let target = self.arg_raw();
self.anchor(target);
self.with_arg(|_| {});
}
"label" => {
let l = self.arg_raw();
self.anchor(l);
}
"ref" | "eqref" | "pageref" | "autoref" | "cref" | "Cref" | "nameref" => {
let l = self.arg_raw();
let mut f = self.fmt();
f.link = model::kept_link(&format!("#{l}"));
self.para.push_text(&l, &f);
}
"cite" | "citep" | "citet" | "parencite" | "textcite" | "autocite" => {
let _ = self.opt_arg();
let _ = self.opt_arg();
let keys = self.arg_raw();
let keys: Vec<&str> = keys.split(',').map(str::trim).filter(|k| !k.is_empty()).collect();
self.text(&format!("[{}]", keys.join(", ")));
}
"footnote" | "footnotetext" | "marginpar" => {
let _ = self.opt_arg();
self.text(" (");
self.with_arg(|_| {});
self.text(")");
}
"includegraphics" => {
let _ = self.opt_arg();
let file = self.arg_raw();
let mut f = self.fmt();
f.italic = true;
self.para.push_text(&format!("[Picture: {}]", file.trim()), &f);
}
"caption" | "caption*" => {
let _ = self.opt_arg();
let toks = self.arg_tokens();
self.flush();
self.para.align = Some(Align::Center);
self.inline_with(toks, |f| f.italic = true);
self.flush();
}
"bibitem" => {
let label = self.opt_arg();
let key = self.arg_raw();
self.flush();
self.text(&format!("[{}] ", label.unwrap_or(key)));
}
"multicolumn" | "multirow" => {
// Outside a tabular: keep the content.
let _ = self.arg_raw();
let _ = self.opt_arg();
let _ = self.arg_raw();
self.with_arg(|_| {});
}
"begin" | "end" => {}
n if DROP_WITH_ARGS.contains(&n) => self.drop_args(n),
_ => {
// Unknown: drop the command, keep its arguments' text.
let _ = self.opt_arg();
}
}
}
fn with_arg(&mut self, f: impl FnOnce(&mut Fmt)) {
let toks = self.arg_tokens();
self.inline_with(toks, f);
}
fn item(&mut self) {
self.flush();
let level = self.lists.len().saturating_sub(1).min(8) as u8;
self.para = Para::default();
if let Some(&ordered) = self.lists.last() {
self.para.list = Some(ListInfo { ordered, level, start: 1, ..Default::default() });
}
let label = self.opt_arg();
if let Some(l) = label {
let mut f = self.fmt();
f.bold = true;
self.para.push_text(&format!("{} ", l.trim()), &f);
}
}
fn environment(&mut self, name: &str) {
self.depth += 1;
match name {
"document" => {
self.run(Some(name));
}
"itemize" | "enumerate" | "description" | "list" | "compactitem" | "compactenum" | "inparaenum" => {
let _ = self.opt_arg();
if name == "list" {
let _ = self.arg_raw();
let _ = self.arg_raw();
}
self.flush();
self.lists.push(matches!(name, "enumerate" | "compactenum" | "inparaenum"));
self.run(Some(name));
self.flush();
self.lists.pop();
}
"quote" | "quotation" | "verse" | "abstract" => {
self.flush();
if name == "abstract" {
let mut p = Para { align: Some(Align::Center), ..Default::default() };
p.push_text("Abstract", &Fmt { bold: true, ..Default::default() });
self.blocks.push(FBlock::Para(p));
}
let saved = self.ctx;
self.ctx.kind = Kind::Quote;
self.run(Some(name));
self.flush();
self.ctx = saved;
}
"center" | "flushleft" | "flushright" | "centering" => {
self.flush();
let align = match name {
"flushleft" => Align::Left,
"flushright" => Align::Right,
_ => Align::Center,
};
let saved = self.ctx;
self.ctx.align = Some(align);
self.run(Some(name));
self.flush();
self.ctx = saved;
}
"tabular" | "tabular*" | "tabularx" | "tabulary" | "longtable" | "longtable*" | "array" | "tblr" | "longtblr" => {
let _ = self.opt_arg();
if matches!(name, "tabular*" | "tabularx" | "tabulary") {
let _ = self.arg_raw();
}
// A real column spec is short; capping it bounds the work on hostile input.
let spec: String = self.arg_source().chars().take(MAX_SPEC).collect();
let toks = self.until_end(name);
self.flush();
let t = table(toks, &spec, self.depth);
if !t.rows.is_empty() {
self.blocks.push(FBlock::Table(t));
}
}
"thebibliography" => {
let _ = self.arg_raw();
self.flush();
let mut p = Para::new(Kind::Heading(1 + self.top));
p.push_text("References", &Fmt::default());
self.blocks.push(FBlock::Para(p));
self.run(Some(name));
self.flush();
}
"minipage" | "figure" | "figure*" | "table" | "table*" | "wrapfigure" | "subfigure" | "titlepage" | "multicols" | "multicols*"
| "small" | "footnotesize" | "large" | "theorem" | "lemma" | "proof" | "definition" | "example" | "remark" | "corollary"
| "proposition" => {
// Floats and boxes: their content, in order.
let _ = self.opt_arg();
if matches!(name, "minipage" | "subfigure" | "multicols" | "multicols*") {
let _ = self.arg_raw();
}
if name == "wrapfigure" {
let _ = self.arg_raw();
let _ = self.arg_raw();
}
self.flush();
if matches!(name, "theorem" | "lemma" | "proof" | "definition" | "example" | "remark" | "corollary" | "proposition") {
let mut label: String = name.chars().take(1).flat_map(char::to_uppercase).chain(name.chars().skip(1)).collect();
label.push_str(". ");
self.para.push_text(&label, &Fmt { bold: name != "proof", italic: name == "proof", ..Default::default() });
}
let (saved_fmt, saved_ctx) = (self.fmt.clone(), self.ctx);
if let Some(s) = size_of(name) {
self.fmt.size = (s > 0.0).then_some(s);
}
self.run(Some(name));
self.flush();
self.fmt = saved_fmt;
self.ctx = saved_ctx;
}
_ => {
// Unknown environment: its content.
let _ = self.opt_arg();
let (saved_fmt, saved_ctx) = (self.fmt.clone(), self.ctx);
self.run(Some(name));
self.fmt = saved_fmt;
self.ctx = saved_ctx;
}
}
self.depth -= 1;
}
/// The tokens up to the matching `\end{name}` (consumed).
fn until_end(&mut self, name: &str) -> &'a [Tok] {
let start = self.pos;
let mut depth = 0usize;
while let Some(tok) = self.t.get(self.pos) {
match tok {
Tok::Begin(n) if n == name => depth += 1,
Tok::End(n) if n == name => {
if depth == 0 {
let toks = self.t.get(start..self.pos).unwrap_or_default();
self.pos += 1;
return toks;
}
depth -= 1;
}
_ => {}
}
self.pos += 1;
}
self.t.get(start..).unwrap_or_default()
}
pub(super) fn finish(mut self) -> (Vec<FBlock>, model::Meta) {
self.flush_pending();
self.flush();
(self.blocks, self.meta)
}
}
/// Parse LaTeX source into a flow.
pub fn parse(src: &str) -> Flow {
let toks = tokenize(src);
// The preamble only contributes metadata; without `\begin{document}` it is all body.
let body_at = toks.iter().position(|t| matches!(t, Tok::Begin(n) if n == "document"));
let mut meta = model::Meta::default();
let (body, preamble): (&[Tok], &[Tok]) = match body_at {
Some(i) => (toks.get(i + 1..).unwrap_or_default(), toks.get(..i).unwrap_or_default()),
None => (&toks[..], &[]),
};
let mut pre = Parser::new(preamble, 0);
while let Some(t) = pre.t.get(pre.pos) {
pre.pos += 1;
if let Tok::Cmd(c) = t {
match c.as_str() {
"title" => meta.title = pre.arg_text(),
"author" => meta.author = pre.arg_text(),
_ => {}
}
}
}
let mut p = Parser::new(body, 0);
p.meta = meta;
// `\part` and `\chapter` put `\section` one level down each.
let uses = |names: [&str; 2]| body.iter().any(|t| matches!(t, Tok::Cmd(c) if names.contains(&c.as_str())));
p.top = u8::from(uses(["part", "part*"])) + u8::from(uses(["chapter", "chapter*"]));
p.run(Some("document"));
let (blocks, meta) = p.finish();
Flow { blocks, meta }
}
fn squash(s: &str) -> String {
s.split_whitespace().collect::<Vec<_>>().join(" ")
}