mirror of
https://github.com/storytold/vectorcraft.git
synced 2026-11-09 00:17:42 +00:00
Variables: Import Data Sets reads every column as a variable and replaces the data sets
Squashed from #5151 onto main.
This commit is contained in:
dotCooCoo
@@ -4,11 +4,11 @@
|
||||
//! variable, anything to a `visibility` variable, a linked image to an `image`
|
||||
//! variable, or a graph root to a `graph` variable (CSV data), put a row of values in
|
||||
//! each dataset, and `dataset.select` swaps the whole document to that row in one undo
|
||||
//! step. `dataset.import` / `dataset.export` move those rows as CSV (header: `name`
|
||||
//! then variable names, which export writes with `#` before a visibility variable and `@`
|
||||
//! before an image variable); import also reads tab-separated files, files without the
|
||||
//! `name` column, and header cells that define new variables (`#` for visibility, `@` for
|
||||
//! image, no prefix for text). `variable.loadLibrary` replaces the variables and data sets with the
|
||||
//! step. `dataset.import` / `dataset.export` move those rows as CSV (header: one
|
||||
//! variable name per column, which export writes with `#` before a visibility variable and `@`
|
||||
//! before an image variable); import replaces the datasets, names the rows Data Set 1, 2…,
|
||||
//! also reads tab-separated files, and reads header cells that define new variables (`#` for
|
||||
//! visibility, `@` for image, no prefix for text). `variable.loadLibrary` replaces the variables and data sets with the
|
||||
//! text variables and data sets of a variable library, the XML file of the reference app's Save
|
||||
//! Variable Library, and `variable.saveLibrary` writes the text variables and data sets as one.
|
||||
//! Agents reach every command through `run_command`, like all engine commands.
|
||||
@@ -838,7 +838,7 @@ pub fn specs() -> Vec<CommandSpec> {
|
||||
"Import Data Sets…",
|
||||
["Window", "Variables"],
|
||||
None,
|
||||
"{path? | data? | dataBase64? | csv?} load datasets from CSV, comma- or tab-separated, in UTF-16 with a byte-order mark, UTF-8 or else Windows-1252 (header: name, then variable names; one row per dataset; without the name column every header is a variable and the rows take the first free names Data Set 1, 2…). A header that names no variable defines one, unbound: `#Name` defines a visibility variable, `@Name` defines an image variable, and a header without a prefix defines a text variable. A prefixed header whose name is a variable of another kind, and a `%` header that isn't a variable's exact name (graph data files), are refused, and so are two headers that name one variable and a header that would define a variable while the variables are locked. Rows whose name already exists are replaced → {imported, updated, names, defined}",
|
||||
"{path? | data? | dataBase64? | csv?} replace every dataset with the rows of a CSV, comma- or tab-separated, in UTF-16 with a byte-order mark, UTF-8 or else Windows-1252 (header: one variable name per column, whatever its case; one row per dataset, named Data Set 1, 2… in file order). The document's variables and their bindings stay. A header that names no variable defines one, unbound: `#Name` defines a visibility variable, `@Name` defines an image variable, and a header without a prefix defines a text variable. A prefixed header whose name is a variable of another kind, and a `%` header that isn't a variable's exact name (graph data files), are refused, and so are two headers that name one variable and a header that would define a variable while the variables are locked. An import that leaves the variables and data sets as they were adds no undo step → {imported, names, defined, deletedDataSets}",
|
||||
has_doc,
|
||||
dataset_import
|
||||
),
|
||||
@@ -847,7 +847,7 @@ pub fn specs() -> Vec<CommandSpec> {
|
||||
"Export Data Sets…",
|
||||
["Window", "Variables"],
|
||||
None,
|
||||
"{path?} write every dataset as CSV (header: name, then one column per variable: `#Name` for a visibility variable, `@Name` for an image variable, and the name alone for a text or graph variable, as dataset.import reads them). Refused when the document has more than 1000 data sets or 256 variables, the most dataset.import reads, or when a variable's header is another variable's name (a visibility variable `Show` next to a text variable `#Show`) → {path, datasets, warning?}; without path → {csv, datasets, warning?}. `warning` names the variables whose header or values make a document without them refuse the file (a % header, a # header whose values aren't true or false); otherwise it names the variables such a document doesn't read back with their kind (text variables whose names start with # or @, graph variables)",
|
||||
"{path?} write every dataset as CSV (header: one column per variable and no dataset names: `#Name` for a visibility variable, `@Name` for an image variable, and the name alone for a text or graph variable, as dataset.import reads them). Refused when the document has more than 1000 data sets or 256 variables, the most dataset.import reads, or when a variable's header is another variable's name (a visibility variable `Show` next to a text variable `#Show`) → {path, datasets, warning?}; without path → {csv, datasets, warning?}. `warning` names the variables whose header or values make a document without them refuse the file (a % header, a # header whose values aren't true or false); otherwise it names the variables such a document doesn't read back with their kind (text variables whose names start with # or @, graph variables)",
|
||||
has_doc,
|
||||
dataset_export
|
||||
),
|
||||
|
||||
@@ -102,7 +102,8 @@ fn tab_separated(text: &str) -> bool {
|
||||
/// by commas, or by tabs when [`tab_separated`] says so. `"` quotes a cell (doubled inside),
|
||||
/// which may then hold separators and line breaks, as [`csv_escape`] writes them. An unquoted
|
||||
/// cell is trimmed. A quoted cell keeps what its quotes hold as written, spaces included, and
|
||||
/// drops white space after the closing quote at the end of the cell. Blank lines are skipped.
|
||||
/// drops white space after the closing quote at the end of the cell. A line whose cells are all
|
||||
/// blank and unquoted is skipped; a quoted empty cell keeps its line as a row.
|
||||
fn parse_datasets_csv(text: &str, cmd: &str) -> Result<Vec<CsvRow>> {
|
||||
let text = text.strip_prefix('\u{feff}').unwrap_or(text);
|
||||
let sep = if tab_separated(text) { '\t' } else { ',' };
|
||||
@@ -132,7 +133,10 @@ fn parse_datasets_csv(text: &str, cmd: &str) -> Result<Vec<CsvRow>> {
|
||||
Some('\n') | None if !in_quotes => {
|
||||
quoted.push(closed.is_some());
|
||||
row.push(end_csv_cell(&mut cur, &mut closed));
|
||||
if row.iter().any(|c| !c.trim().is_empty()) {
|
||||
if row.len() > MAX_CSV_COLS {
|
||||
return Err(bad(cmd, format!("line {line}: more than {MAX_CSV_COLS} columns")));
|
||||
}
|
||||
if row.iter().any(|c| !c.trim().is_empty()) || quoted.contains(&true) {
|
||||
rows.push((start, std::mem::take(&mut row), std::mem::take(&mut quoted)));
|
||||
// +1 for the header.
|
||||
if rows.len() > MAX_CSV_ROWS + 1 {
|
||||
@@ -174,8 +178,10 @@ fn end_csv_cell(cur: &mut String, closed: &mut Option<usize>) -> String {
|
||||
}
|
||||
|
||||
fn csv_escape(s: &str) -> String {
|
||||
// An empty string is quoted so that it reads back as an empty value, not as a missing one.
|
||||
if s.is_empty() || s.contains([',', '"', '\n', '\r']) || s != s.trim() { format!("\"{}\"", s.replace('"', "\"\"")) } else { s.to_string() }
|
||||
// An empty string is quoted so that it reads back as an empty value, not as a missing one. A
|
||||
// tab is quoted so that a header of one variable whose name holds a tab isn't read as
|
||||
// tab-separated.
|
||||
if s.is_empty() || s.contains([',', '"', '\n', '\r', '\t']) || s != s.trim() { format!("\"{}\"", s.replace('"', "\"\"")) } else { s.to_string() }
|
||||
}
|
||||
|
||||
/// A visibility cell: true/false, 1/0, yes/no (ASCII, case-insensitive). An error names the
|
||||
@@ -240,8 +246,7 @@ fn kinds_a_new_document_loses(vars: &Variables) -> (Vec<&str>, Vec<&str>) {
|
||||
for (i, v) in vars.variables.iter().enumerate() {
|
||||
let header = export_header(v);
|
||||
let cell = header.trim();
|
||||
// Column 1 holds the dataset names.
|
||||
let read = if cell.is_empty() { None } else { header_variable(cell, i + 2, &none, &mut defined, C).ok() };
|
||||
let read = if cell.is_empty() { None } else { header_variable(cell, i + 1, &none, &mut defined, C).ok() };
|
||||
// The import refuses a header that names a variable an earlier column named.
|
||||
let Some((name, kind)) = read.filter(|(name, _)| seen.insert(name.clone())) else {
|
||||
refused.push(v.name.as_str());
|
||||
@@ -260,30 +265,36 @@ fn kinds_a_new_document_loses(vars: &Variables) -> (Vec<&str>, Vec<&str>) {
|
||||
(refused, other)
|
||||
}
|
||||
|
||||
/// Build the CSV of every dataset (header `name` then each variable's [`export_header`] in
|
||||
/// definition order). Refused when a variable's header is another variable's name, which the
|
||||
/// import reads as that variable (a visibility variable `Show` next to a text variable `#Show`).
|
||||
/// Build the CSV of every dataset: a header of each variable's [`export_header`] in definition
|
||||
/// order, then one row per dataset. The dataset names are not written. Refused when a variable's
|
||||
/// header is another variable's name, which the import reads as that variable (a visibility
|
||||
/// variable `Show` next to a text variable `#Show`). A dataset with no value for any variable would
|
||||
/// be a blank line, which the import skips, so its row gets a quoted empty cell: in the first column
|
||||
/// whose variable isn't a text variable, where it reads back as no value, or else in the first
|
||||
/// column, where it reads back as an empty text.
|
||||
fn datasets_to_csv(vars: &Variables, cmd: &str) -> Result<String> {
|
||||
let names: BTreeSet<&str> = vars.variables.iter().map(|v| v.name.as_str()).collect();
|
||||
let mut out = String::from("name");
|
||||
let mut header = Vec::with_capacity(vars.variables.len());
|
||||
for v in &vars.variables {
|
||||
let header = export_header(v);
|
||||
if header != v.name && names.contains(header.as_str()) {
|
||||
let (name, other) = (&v.name, &header);
|
||||
return Err(bad(cmd, format!("the header `{header}` for `{name}` would read back as the variable `{other}`: rename one of them")));
|
||||
let cell = export_header(v);
|
||||
if cell != v.name && names.contains(cell.as_str()) {
|
||||
let (name, other) = (&v.name, &cell);
|
||||
return Err(bad(cmd, format!("the header `{cell}` for `{name}` would read back as the variable `{other}`: rename one of them")));
|
||||
}
|
||||
out.push(',');
|
||||
out.push_str(&csv_escape(&header));
|
||||
header.push(csv_escape(&cell));
|
||||
}
|
||||
let mut out = header.join(",");
|
||||
out.push('\n');
|
||||
let keep = vars.variables.iter().position(|v| v.kind != VariableKind::Text).unwrap_or(0);
|
||||
for ds in &vars.datasets {
|
||||
out.push_str(&csv_escape(&ds.name));
|
||||
for v in &vars.variables {
|
||||
out.push(',');
|
||||
if let Some(val) = ds.values.get(&v.name) {
|
||||
out.push_str(&csv_escape(&value_to_cell(val)));
|
||||
}
|
||||
let mut row: Vec<String> =
|
||||
vars.variables.iter().map(|v| ds.values.get(&v.name).map(|val| csv_escape(&value_to_cell(val))).unwrap_or_default()).collect();
|
||||
if row.iter().all(String::is_empty)
|
||||
&& let Some(cell) = row.get_mut(keep)
|
||||
{
|
||||
*cell = csv_escape("");
|
||||
}
|
||||
out.push_str(&row.join(","));
|
||||
out.push('\n');
|
||||
}
|
||||
Ok(out)
|
||||
@@ -329,12 +340,11 @@ fn header_variable(
|
||||
}
|
||||
}
|
||||
|
||||
/// Import datasets from CSV. A first header cell `name` makes the first column the dataset
|
||||
/// names; the other headers name variables, as [`header_variable`] reads them, and the
|
||||
/// variables the document doesn't define yet are defined, unbound. Without the `name` column,
|
||||
/// every header names a variable and the rows take the first free names "Data Set 1", "Data
|
||||
/// Set 2" and so on. Two headers that name one variable are refused. Existing rows of the same
|
||||
/// name are replaced; new names are appended. One undo step.
|
||||
/// Import datasets from CSV. Every header cell names a variable, as [`header_variable`] reads it,
|
||||
/// and the variables the document doesn't define yet are defined, unbound. Two headers that name
|
||||
/// one variable are refused. The rows replace every dataset of the document and are named "Data
|
||||
/// Set 1", "Data Set 2" and so on, in file order. The document's variables and their bindings
|
||||
/// stay. One undo step, and none when the variables and datasets stay as they were.
|
||||
pub(super) fn dataset_import(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
const C: &str = "dataset.import";
|
||||
let (bytes, given_as_text) = datasets_csv_bytes(p, C)?;
|
||||
@@ -344,16 +354,10 @@ pub(super) fn dataset_import(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
let Some((_, header, _)) = rows.first() else {
|
||||
return Err(bad(C, "CSV is empty"));
|
||||
};
|
||||
let name_column = header.first().is_some_and(|h| h.trim().eq_ignore_ascii_case("name"));
|
||||
let (kinds, existing_names) = {
|
||||
let st = s.doc()?;
|
||||
let kinds: BTreeMap<String, VariableKind> = st.doc.variables.variables.iter().map(|v| (v.name.clone(), v.kind)).collect();
|
||||
let existing: BTreeSet<String> = st.doc.variables.datasets.iter().map(|d| d.name.clone()).collect();
|
||||
(kinds, existing)
|
||||
};
|
||||
let kinds: BTreeMap<String, VariableKind> = s.doc()?.doc.variables.variables.iter().map(|v| (v.name.clone(), v.kind)).collect();
|
||||
let mut columns: Vec<(usize, String, VariableKind)> = Vec::new();
|
||||
let mut defined: Vec<Variable> = Vec::new();
|
||||
for (i, h) in header.iter().enumerate().skip(usize::from(name_column)) {
|
||||
for (i, h) in header.iter().enumerate() {
|
||||
let cell = h.trim();
|
||||
if cell.is_empty() {
|
||||
return Err(bad(C, format!("empty header in column {}", i + 1)));
|
||||
@@ -373,27 +377,8 @@ pub(super) fn dataset_import(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
return Err(bad(C, "CSV has a header but no data rows"));
|
||||
}
|
||||
let mut parsed: Vec<DataSet> = Vec::new();
|
||||
// The names of the rows read so far, and the last number given to a row without a name.
|
||||
let (mut seen, mut number) = (BTreeSet::new(), 0usize);
|
||||
for &(line, ref row, ref quoted) in rows.iter().skip(1) {
|
||||
let raw_name = if name_column { row.first().map_or("", |c| c.trim()) } else { "" };
|
||||
let name = if raw_name.is_empty() {
|
||||
// Same numbering as Capture Data Set when a row leaves the name blank or the file has
|
||||
// no name column. The search starts after the last number given.
|
||||
let (n, name) = (number + 1..)
|
||||
.map(|n| (n, format!("Data Set {n}")))
|
||||
.find(|(_, n)| !seen.contains(n) && !existing_names.contains(n))
|
||||
.unwrap_or_else(|| (line, format!("Data Set {line}")));
|
||||
number = n;
|
||||
name
|
||||
} else if raw_name.chars().count() > MAX_NAME {
|
||||
return Err(bad(C, format!("line {line}: `name` is longer than {MAX_NAME} characters")));
|
||||
} else {
|
||||
raw_name.to_string()
|
||||
};
|
||||
if !seen.insert(name.clone()) {
|
||||
return Err(bad(C, format!("line {line}: dataset `{name}` appears twice")));
|
||||
}
|
||||
for (n, &(line, ref row, ref quoted)) in rows.iter().skip(1).enumerate() {
|
||||
let name = format!("Data Set {}", n + 1);
|
||||
let mut values = BTreeMap::new();
|
||||
for &(col, ref var, kind) in &columns {
|
||||
let cell = row.get(col).map(String::as_str).unwrap_or("");
|
||||
@@ -406,26 +391,23 @@ pub(super) fn dataset_import(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
}
|
||||
let names: Vec<String> = parsed.iter().map(|d| d.name.clone()).collect();
|
||||
let new_variables: Vec<String> = defined.iter().map(|v| v.name.clone()).collect();
|
||||
let deleted: Vec<String> = s.doc()?.doc.variables.datasets.iter().map(|d| d.name.clone()).collect();
|
||||
// The import is one undo step, and an import that leaves the variables and data sets as they
|
||||
// were adds none.
|
||||
let (imported, updated) = super::edit_variables(s, "Import Data Sets", |vars| {
|
||||
vars.variables.extend(defined);
|
||||
let (mut imported, mut updated) = (0usize, 0usize);
|
||||
for row in parsed {
|
||||
if let Some(existing) = vars.datasets.iter_mut().find(|x| x.name == row.name) {
|
||||
*existing = row;
|
||||
updated += 1;
|
||||
} else {
|
||||
vars.datasets.push(row);
|
||||
imported += 1;
|
||||
}
|
||||
// were adds none, so the active data set stays when nothing changes.
|
||||
super::edit_variables(s, "Import Data Sets", |vars| {
|
||||
if defined.is_empty() && vars.datasets == parsed {
|
||||
return Ok(());
|
||||
}
|
||||
Ok((imported, updated))
|
||||
vars.variables.extend(defined);
|
||||
vars.datasets = parsed;
|
||||
vars.active_dataset = None;
|
||||
Ok(())
|
||||
})?;
|
||||
Ok(json!({"imported": imported, "updated": updated, "names": names, "defined": new_variables}))
|
||||
Ok(json!({"imported": names.len(), "names": names, "defined": new_variables, "deletedDataSets": deleted}))
|
||||
}
|
||||
|
||||
/// Export every dataset as CSV ([`datasets_to_csv`]). With `path`, write the file; else return
|
||||
/// Export every dataset as CSV ([`datasets_to_csv`]: one column per variable, no dataset names).
|
||||
/// With `path`, write the file; else return
|
||||
/// `{csv}`. The reply's `warning` names the variables whose column makes a document without them
|
||||
/// refuse the file, or else the variables it doesn't read back with their kind
|
||||
/// ([`kinds_a_new_document_loses`]); the app shows it in its status bar.
|
||||
@@ -439,7 +421,6 @@ pub(super) fn dataset_export(s: &mut Session, p: &Value) -> Result<Value> {
|
||||
if st.doc.variables.datasets.len() > MAX_CSV_ROWS {
|
||||
return Err(bad(C, format!("more than {MAX_CSV_ROWS} data sets: Import Data Sets reads at most {MAX_CSV_ROWS}")));
|
||||
}
|
||||
// The name column comes first, so the import's column cap leaves room for this many variables.
|
||||
if st.doc.variables.variables.len() > MAX_CSV_COLS {
|
||||
return Err(bad(C, format!("more than {MAX_CSV_COLS} variables: Import Data Sets reads at most {MAX_CSV_COLS}")));
|
||||
}
|
||||
|
||||
@@ -245,25 +245,26 @@ fn mismatched_and_missing_bindings_are_skipped_and_counted() {
|
||||
fn a_row_with_no_value_for_a_bound_variable_warns_that_nothing_was_applied() {
|
||||
let mut s = session();
|
||||
let t = text(&mut s, "x");
|
||||
s.execute("dataset.import", &json!({"csv": "name,Name,Title\nA,Ann,Boss\nB,Bob,Chief\n"})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": "Name,Title\nAnn,Boss\nBob,Chief\n"})).unwrap();
|
||||
let (a, b) = ("Data Set 1", "Data Set 2");
|
||||
let warning = |row: &str| format!("nothing in data set {row} was applied: no object is bound to a variable it has a value for");
|
||||
let v = s.execute("dataset.select", &json!({"name": "A"})).unwrap();
|
||||
assert_eq!((v["applied"].as_u64(), v["skipped"].as_u64(), &v["warning"]), (Some(0), Some(0), &json!(warning("A"))), "{v}");
|
||||
let v = s.execute("dataset.select", &json!({"name": a})).unwrap();
|
||||
assert_eq!((v["applied"].as_u64(), v["skipped"].as_u64(), &v["warning"]), (Some(0), Some(0), &json!(warning(a))), "{v}");
|
||||
let v = s.execute("dataset.next", &json!({})).unwrap();
|
||||
assert_eq!(v["warning"], json!(warning("B")), "{v}");
|
||||
assert_eq!(v["warning"], json!(warning(b)), "{v}");
|
||||
let v = s.execute("dataset.prev", &json!({})).unwrap();
|
||||
assert_eq!(v["warning"], json!(warning("A")), "{v}");
|
||||
assert_eq!(v["warning"], json!(warning(a)), "{v}");
|
||||
assert_eq!(plain(&s, t), "x");
|
||||
|
||||
// A bound variable that the row has no value for still applies nothing.
|
||||
s.execute("variable.define", &json!({"name": "Other", "kind": "text"})).unwrap();
|
||||
s.execute("variable.bind", &json!({"variable": "Other", "ids": [t.0]})).unwrap();
|
||||
let v = s.execute("dataset.select", &json!({"name": "B"})).unwrap();
|
||||
assert_eq!((v["applied"].as_u64(), v["skipped"].as_u64(), &v["warning"]), (Some(0), Some(1), &json!(warning("B"))), "{v}");
|
||||
let v = s.execute("dataset.select", &json!({"name": b})).unwrap();
|
||||
assert_eq!((v["applied"].as_u64(), v["skipped"].as_u64(), &v["warning"]), (Some(0), Some(1), &json!(warning(b))), "{v}");
|
||||
|
||||
// Bound to a variable with a value, the row applies it and gives no warning.
|
||||
s.execute("variable.bind", &json!({"variable": "Title", "ids": [t.0]})).unwrap();
|
||||
let v = s.execute("dataset.select", &json!({"name": "A"})).unwrap();
|
||||
let v = s.execute("dataset.select", &json!({"name": a})).unwrap();
|
||||
assert_eq!(v["applied"].as_u64(), Some(1), "{v}");
|
||||
assert!(v.get("warning").is_none(), "{v}");
|
||||
assert_eq!(plain(&s, t), "Boss");
|
||||
@@ -271,7 +272,7 @@ fn a_row_with_no_value_for_a_bound_variable_warns_that_nothing_was_applied() {
|
||||
// Locked art is skipped, and its binding still counts as bound.
|
||||
s.execute("select.set", &json!({"ids": [t.0]})).unwrap();
|
||||
s.execute("object.lock", &json!({})).unwrap();
|
||||
let v = s.execute("dataset.select", &json!({"name": "B"})).unwrap();
|
||||
let v = s.execute("dataset.select", &json!({"name": b})).unwrap();
|
||||
assert_eq!(v["applied"].as_u64(), Some(0), "{v}");
|
||||
assert!(v.get("warning").is_none(), "{v}");
|
||||
}
|
||||
@@ -613,8 +614,8 @@ fn locked_variables_refuse_new_deleted_and_renamed_variables() {
|
||||
s.execute("variable.unbind", &json!({"ids": [r.0], "variable": "Show"})).unwrap();
|
||||
s.execute("dataset.capture", &json!({})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "Bob", "values": {"Name": "Bob"}})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": "name,Name\nCarol,Carol\n"})).unwrap();
|
||||
s.execute("dataset.select", &json!({"name": "Carol"})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": "Name\nCarol\n"})).unwrap();
|
||||
s.execute("dataset.select", &json!({"name": "Data Set 1"})).unwrap();
|
||||
assert_eq!(plain(&s, t), "Carol");
|
||||
|
||||
// Unlocking lets a variable be defined again; `locked` sets the state rather than toggling.
|
||||
@@ -1395,12 +1396,11 @@ fn datasets_csv_keeps_line_breaks_in_quoted_cells() {
|
||||
let value = "Line one, \"quoted\"\nLine two";
|
||||
s.execute("dataset.new", &json!({"name": "A", "values": {"Name": value}})).unwrap();
|
||||
let csv = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
s.execute("dataset.delete", &json!({"names": ["A"]})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
let sets = &s.doc().unwrap().doc.variables.datasets;
|
||||
assert_eq!(sets.len(), 1);
|
||||
assert_eq!(sets[0].values.get("Name"), Some(&vectorcraft_doc::DataValue::Text(value.into())));
|
||||
assert!(s.execute("dataset.import", &json!({"csv": "name,Name\nX,\"open"})).is_err(), "unclosed quote");
|
||||
assert!(s.execute("dataset.import", &json!({"csv": "Name\n\"open"})).is_err(), "unclosed quote");
|
||||
}
|
||||
|
||||
/// An empty text value is not a missing value: it comes back from the exported CSV as an empty
|
||||
@@ -1413,23 +1413,45 @@ fn datasets_csv_keeps_an_empty_text_value() {
|
||||
s.execute("variable.bind", &json!({"variable": "Name", "ids": [t.0]})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "A", "values": {"Name": ""}})).unwrap();
|
||||
let csv = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
s.execute("dataset.delete", &json!({"names": ["A"]})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
let value = s.doc().unwrap().doc.variables.dataset("A").and_then(|d| d.values.get("Name")).cloned();
|
||||
assert_eq!(value, Some(vectorcraft_doc::DataValue::Text(String::new())));
|
||||
s.execute("dataset.import", &json!({"csv": "name,Name\nB,\nC,\"\"\n"})).unwrap();
|
||||
let value = |name: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get("Name")).cloned();
|
||||
assert_eq!(value("C"), Some(vectorcraft_doc::DataValue::Text(String::new())));
|
||||
let value = |s: &Session, name: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get("Name")).cloned();
|
||||
assert_eq!(value(&s, "Data Set 1"), Some(vectorcraft_doc::DataValue::Text(String::new())));
|
||||
s.execute("variable.define", &json!({"name": "Other", "kind": "text"})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": "Name,Other\n,x\n\"\",y\n"})).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 2"), Some(vectorcraft_doc::DataValue::Text(String::new())));
|
||||
// Compatibility rule for files written before empty text was quoted: a bare empty cell stays "no value".
|
||||
assert_eq!(value("B"), None);
|
||||
assert_eq!(value(&s, "Data Set 1"), None);
|
||||
// A data set with no value for the variable exports a bare cell and stays without a value.
|
||||
s.execute("dataset.new", &json!({"name": "D", "values": {}})).unwrap();
|
||||
let csv = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
assert!(csv.contains("\nD,\n"), "{csv:?}");
|
||||
s.execute("dataset.delete", &json!({"names": ["D"]})).unwrap();
|
||||
assert_eq!(csv, "Name,Other\n,x\n\"\",y\n");
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
let value = |name: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get("Name")).cloned();
|
||||
assert_eq!(value("D"), None);
|
||||
assert_eq!(value(&s, "Data Set 1"), None);
|
||||
}
|
||||
|
||||
/// A data set with no value for any variable exports as a row that imports back: a quoted empty
|
||||
/// cell in the first column that isn't a text variable reads back as no value, and with only text
|
||||
/// variables the first column reads back as an empty text.
|
||||
#[test]
|
||||
fn datasets_csv_keeps_a_data_set_without_values() {
|
||||
let mut s = session();
|
||||
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "A", "values": {}})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "B", "values": {"Title": "x"}})).unwrap();
|
||||
let export = |s: &mut Session| s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
let csv = export(&mut s);
|
||||
assert_eq!(csv, "Title\n\"\"\nx\n");
|
||||
let imp = s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!(imp["names"], json!(["Data Set 1", "Data Set 2"]));
|
||||
let values = |s: &Session, name: &str| s.doc().unwrap().doc.variables.dataset(name).map(|d| d.values.clone()).unwrap();
|
||||
assert_eq!(values(&s, "Data Set 1").get("Title"), Some(&vectorcraft_doc::DataValue::Text(String::new())));
|
||||
|
||||
s.execute("variable.define", &json!({"name": "Show", "kind": "visibility"})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "C", "values": {}})).unwrap();
|
||||
let csv = export(&mut s);
|
||||
assert!(csv.ends_with("\n,\"\"\n"), "{csv:?}");
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!(s.doc().unwrap().doc.variables.datasets.len(), 3);
|
||||
assert!(values(&s, "Data Set 3").is_empty());
|
||||
}
|
||||
|
||||
/// A quoted text cell keeps its leading and trailing spaces, so a padded value comes back from
|
||||
@@ -1442,25 +1464,28 @@ fn datasets_csv_keeps_spaces_in_quoted_cells() {
|
||||
s.execute("variable.bind", &json!({"variable": "Name", "ids": [t.0]})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "A", "values": {"Name": " padded "}})).unwrap();
|
||||
let csv = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
s.execute("dataset.delete", &json!({"names": ["A"]})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": "name,Name\nB, typed \nC,\" quoted \" \n"})).unwrap();
|
||||
let value = |name: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get("Name")).cloned();
|
||||
let value = |s: &Session, name: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get("Name")).cloned();
|
||||
let text_value = |v: &str| Some(vectorcraft_doc::DataValue::Text(v.into()));
|
||||
assert_eq!(value("A"), text_value(" padded "));
|
||||
assert_eq!(value("B"), text_value("typed"));
|
||||
assert_eq!(value("C"), text_value(" quoted "));
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value(" padded "));
|
||||
s.execute("dataset.import", &json!({"csv": "Name\n typed \n\" quoted \" \n"})).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("typed"));
|
||||
assert_eq!(value(&s, "Data Set 2"), text_value(" quoted "));
|
||||
}
|
||||
|
||||
/// Export refuses a document with more data sets or variables than `dataset.import` reads, so
|
||||
/// every file it writes imports back: 1,000 data sets and 256 variables export and import, and
|
||||
/// one more of either is refused.
|
||||
/// one more of either is refused. The import refuses a line of 257 columns, the last one included.
|
||||
#[test]
|
||||
fn datasets_export_refuses_more_than_import_reads() {
|
||||
let mut s = session();
|
||||
let header: Vec<String> = (1..=256).map(|i| format!("V{i}")).collect();
|
||||
for wide in [format!("{},V257\nx\n", header.join(",")), format!("{}\nx{}\n", header.join(","), ",".repeat(256))] {
|
||||
let err = s.execute("dataset.import", &json!({"csv": wide})).unwrap_err().to_string();
|
||||
assert!(err.contains("more than 256 columns"), "{err}");
|
||||
}
|
||||
let rows: String = (1..=1000).map(|i| format!("Set {i},x\n")).collect();
|
||||
s.execute("dataset.import", &json!({"csv": format!("name,{}\n{rows}", header.join(","))})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": format!("{}\n{rows}", header.join(","))})).unwrap();
|
||||
let csv = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
|
||||
@@ -1488,59 +1513,102 @@ fn datasets_csv_reads_tab_separated_files() {
|
||||
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
|
||||
s.execute("variable.define", &json!({"name": "Show", "kind": "visibility"})).unwrap();
|
||||
s.execute("variable.define", &json!({"name": "Last, First", "kind": "text"})).unwrap();
|
||||
let tsv = "name\tTitle\tShow\r\nA\tTwo words\ttrue\r\nB\t\"a, b\tc\"\tfalse\r\n";
|
||||
let tsv = "Title\tShow\r\nTwo words\ttrue\r\n\"a, b\tc\"\tfalse\r\n";
|
||||
let imp = s.execute("dataset.import", &json!({"csv": tsv})).unwrap();
|
||||
assert_eq!(imp["names"], json!(["A", "B"]));
|
||||
s.execute("dataset.import", &json!({"csv": "name,Title\nC,one\ttwo"})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": "name\t\"Last, First\"\nD\tSmith, Jo"})).unwrap();
|
||||
let value = |name: &str, var: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get(var)).cloned();
|
||||
assert_eq!(imp["names"], json!(["Data Set 1", "Data Set 2"]));
|
||||
let value = |s: &Session, name: &str, var: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get(var)).cloned();
|
||||
let text_value = |v: &str| Some(vectorcraft_doc::DataValue::Text(v.into()));
|
||||
assert_eq!(value("A", "Title"), text_value("Two words"));
|
||||
assert_eq!(value("A", "Show"), Some(vectorcraft_doc::DataValue::Visible(true)));
|
||||
assert_eq!(value("B", "Title"), text_value("a, b\tc"));
|
||||
assert_eq!(value("B", "Show"), Some(vectorcraft_doc::DataValue::Visible(false)));
|
||||
assert_eq!(value("C", "Title"), text_value("one\ttwo"));
|
||||
assert_eq!(value("D", "Last, First"), text_value("Smith, Jo"));
|
||||
let unclosed = s.execute("dataset.import", &json!({"csv": "name\tTitle\t\"Show\nA\tx\ttrue"})).unwrap_err().to_string();
|
||||
assert_eq!(value(&s, "Data Set 1", "Title"), text_value("Two words"));
|
||||
assert_eq!(value(&s, "Data Set 1", "Show"), Some(vectorcraft_doc::DataValue::Visible(true)));
|
||||
assert_eq!(value(&s, "Data Set 2", "Title"), text_value("a, b\tc"));
|
||||
assert_eq!(value(&s, "Data Set 2", "Show"), Some(vectorcraft_doc::DataValue::Visible(false)));
|
||||
s.execute("dataset.import", &json!({"csv": "Title,Show\none\ttwo,true"})).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 1", "Title"), text_value("one\ttwo"));
|
||||
s.execute("dataset.import", &json!({"csv": "\"Last, First\"\tShow\nSmith, Jo\ttrue"})).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 1", "Last, First"), text_value("Smith, Jo"));
|
||||
// A file of one variable whose name holds a tab is exported with the name quoted, and reads back as one column.
|
||||
let mut one = session();
|
||||
one.execute("variable.define", &json!({"name": "Tab\tName", "kind": "text"})).unwrap();
|
||||
one.execute("dataset.new", &json!({"name": "A", "values": {"Tab\tName": "v"}})).unwrap();
|
||||
let csv = one.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
assert_eq!(csv, "\"Tab\tName\"\nv\n");
|
||||
let imp = one.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!(imp["defined"], json!([]), "{imp}");
|
||||
assert_eq!(value(&one, "Data Set 1", "Tab\tName"), text_value("v"));
|
||||
let unclosed = s.execute("dataset.import", &json!({"csv": "Title\t\"Show\nx\ttrue"})).unwrap_err().to_string();
|
||||
assert!(unclosed.contains("unclosed quote"), "{unclosed}");
|
||||
}
|
||||
|
||||
/// Without a `name` column every header cell is a variable, and the rows take the first free
|
||||
/// names Data Set 1, 2 and so on. A first header cell `name`, in any case, still holds the names.
|
||||
/// Every header cell is a variable, whatever its case: a file laid out as the set-up page shows
|
||||
/// (`Name,Age`) defines and fills `Name`, and `name` and `NAME` are variables too. The rows are
|
||||
/// named Data Set 1, 2 and so on, in file order.
|
||||
#[test]
|
||||
fn datasets_csv_without_a_name_column_numbers_the_rows() {
|
||||
fn datasets_csv_reads_every_header_cell_as_a_variable() {
|
||||
let mut s = session();
|
||||
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "Data Set 2", "values": {"Title": "kept"}})).unwrap();
|
||||
let imp = s.execute("dataset.import", &json!({"csv": "Title\nx\ny\n"})).unwrap();
|
||||
assert_eq!((imp["imported"].as_u64(), imp["updated"].as_u64()), (Some(2), Some(0)));
|
||||
assert_eq!(imp["names"], json!(["Data Set 1", "Data Set 3"]));
|
||||
let value = |s: &Session, name: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get("Title")).cloned();
|
||||
let text_value = |v: &str| Some(vectorcraft_doc::DataValue::Text(v.into()));
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("x"));
|
||||
assert_eq!(value(&s, "Data Set 2"), text_value("kept"));
|
||||
assert_eq!(value(&s, "Data Set 3"), text_value("y"));
|
||||
let named = s.execute("dataset.import", &json!({"csv": "NAME,Title\nData Set 1,z"})).unwrap();
|
||||
assert_eq!((named["imported"].as_u64(), named["updated"].as_u64()), (Some(0), Some(1)));
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("z"));
|
||||
let t = text(&mut s, "x");
|
||||
let imp = s.execute("dataset.import", &json!({"csv": "Name,Age\nBill Tucker,36\nJo Lee,41\n"})).unwrap();
|
||||
assert_eq!((imp["imported"].as_u64(), &imp["defined"]), (Some(2), &json!(["Name", "Age"])), "{imp}");
|
||||
assert_eq!(imp["names"], json!(["Data Set 1", "Data Set 2"]));
|
||||
s.execute("variable.bind", &json!({"variable": "Name", "ids": [t.0]})).unwrap();
|
||||
s.execute("dataset.select", &json!({"name": "Data Set 2"})).unwrap();
|
||||
assert_eq!(plain(&s, t), "Jo Lee");
|
||||
for header in ["name", "NAME"] {
|
||||
let imp = s.execute("dataset.import", &json!({"csv": format!("{header},Age\nx,1\n")})).unwrap();
|
||||
assert_eq!((&imp["defined"], &imp["names"]), (&json!([header]), &json!(["Data Set 1"])), "{header}: {imp}");
|
||||
}
|
||||
let value = |s: &Session, var: &str| s.doc().unwrap().doc.variables.dataset("Data Set 1").and_then(|d| d.values.get(var)).cloned();
|
||||
assert_eq!(value(&s, "NAME"), Some(vectorcraft_doc::DataValue::Text("x".into())));
|
||||
}
|
||||
|
||||
/// An error about a row's values or name gives the file line the row starts on, counting blank
|
||||
/// lines and line breaks inside quoted cells.
|
||||
/// An import replaces every data set of the document, as Load Variable Library does, and clears
|
||||
/// the active data set. The document's variables stay with their bindings, including those the
|
||||
/// file has no column for, which then have no value in any data set. Undo brings the data sets back.
|
||||
#[test]
|
||||
fn datasets_csv_import_replaces_the_data_sets() {
|
||||
let mut s = session();
|
||||
let t = text(&mut s, "x");
|
||||
let r = rect(&mut s);
|
||||
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
|
||||
s.execute("variable.define", &json!({"name": "Show", "kind": "visibility"})).unwrap();
|
||||
s.execute("variable.bind", &json!({"variable": "Title", "ids": [t.0]})).unwrap();
|
||||
s.execute("variable.bind", &json!({"variable": "Show", "ids": [r.0]})).unwrap();
|
||||
for (name, title) in [("First", "a"), ("Data Set 2", "b"), ("Data Set 3", "c")] {
|
||||
s.execute("dataset.new", &json!({"name": name, "values": {"Title": title, "Show": true}})).unwrap();
|
||||
}
|
||||
s.execute("dataset.select", &json!({"name": "First"})).unwrap();
|
||||
let imp = s.execute("dataset.import", &json!({"csv": "Title\ny\nz\n"})).unwrap();
|
||||
assert_eq!(imp["names"], json!(["Data Set 1", "Data Set 2"]));
|
||||
assert_eq!(imp["deletedDataSets"], json!(["First", "Data Set 2", "Data Set 3"]));
|
||||
let vars = &s.doc().unwrap().doc.variables;
|
||||
let sets: Vec<_> = vars.datasets.iter().map(|d| (d.name.as_str(), d.values.get("Title").cloned(), d.values.get("Show").cloned())).collect();
|
||||
let text_value = |v: &str| Some(vectorcraft_doc::DataValue::Text(v.into()));
|
||||
assert_eq!(sets, [("Data Set 1", text_value("y"), None), ("Data Set 2", text_value("z"), None)]);
|
||||
assert_eq!(vars.active_dataset, None);
|
||||
let listed = s.execute("variable.list", &json!({})).unwrap();
|
||||
assert_eq!(listed["variables"], json!([{"name": "Title", "kind": "text", "bindings": 1}, {"name": "Show", "kind": "visibility", "bindings": 1}]));
|
||||
s.execute("dataset.select", &json!({"name": "Data Set 2"})).unwrap();
|
||||
assert_eq!(plain(&s, t), "z");
|
||||
|
||||
// The first undo takes back the selection, the second the import.
|
||||
s.execute("edit.undo", &json!({})).unwrap();
|
||||
s.execute("edit.undo", &json!({})).unwrap();
|
||||
let vars = &s.doc().unwrap().doc.variables;
|
||||
let names: Vec<&str> = vars.datasets.iter().map(|d| d.name.as_str()).collect();
|
||||
assert_eq!((names, vars.active_dataset.as_deref()), (vec!["First", "Data Set 2", "Data Set 3"], Some("First")));
|
||||
}
|
||||
|
||||
/// An error about a row's values gives the file line the row starts on, counting blank lines and
|
||||
/// line breaks inside quoted cells.
|
||||
#[test]
|
||||
fn datasets_csv_row_errors_name_the_file_line() {
|
||||
let mut s = session();
|
||||
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
|
||||
s.execute("variable.define", &json!({"name": "Show", "kind": "visibility"})).unwrap();
|
||||
let error = |s: &mut Session, csv: &str| s.execute("dataset.import", &json!({"csv": csv})).unwrap_err().to_string();
|
||||
let blank_line = error(&mut s, "name,Show\n\nX,maybe");
|
||||
let blank_line = error(&mut s, "Show\n\nmaybe");
|
||||
assert!(blank_line.contains("line 3:"), "{blank_line}");
|
||||
let line_break = error(&mut s, "name,Title,Show\nA,\"one\ntwo\",true\nB,x,maybe");
|
||||
let line_break = error(&mut s, "Title,Show\n\"one\ntwo\",true\nx,maybe");
|
||||
assert!(line_break.contains("line 4:"), "{line_break}");
|
||||
let twice = error(&mut s, "name,Title\nA,x\n\nA,y");
|
||||
assert!(twice.contains("line 4:"), "{twice}");
|
||||
let long = error(&mut s, &format!("name,Title\n\n{},x", "n".repeat(256)));
|
||||
assert!(long.contains("line 3:") && long.contains("longer than 255"), "{long}");
|
||||
}
|
||||
|
||||
/// A refused cell's error gives its line and why the value was refused: a visibility value that
|
||||
@@ -1551,10 +1619,10 @@ fn datasets_csv_cell_errors_say_why() {
|
||||
s.execute("variable.define", &json!({"name": "Show", "kind": "visibility"})).unwrap();
|
||||
s.execute("variable.define", &json!({"name": "Sales", "kind": "graph"})).unwrap();
|
||||
let error = |s: &mut Session, csv: &str| s.execute("dataset.import", &json!({"csv": csv})).unwrap_err();
|
||||
let maybe = error(&mut s, "name,Show\nX,maybe");
|
||||
let maybe = error(&mut s, "Show\nmaybe");
|
||||
let want = "line 2: value for `Show` must be true or false";
|
||||
assert!(matches!(&maybe, EngineError::BadParams { msg, .. } if msg == want), "{maybe}");
|
||||
let long = error(&mut s, &format!("name,Sales\nX,\"{}\"", "1,".repeat(140_000)));
|
||||
let long = error(&mut s, &format!("Sales\n\"{}\"", "1,".repeat(140_000)));
|
||||
let want = "line 2: value for `Sales` is longer than 262144 bytes";
|
||||
assert!(matches!(&long, EngineError::BadParams { msg, .. } if msg == want), "{long}");
|
||||
assert!(s.doc().unwrap().doc.variables.datasets.is_empty());
|
||||
@@ -1567,10 +1635,10 @@ fn datasets_csv_cell_errors_say_why() {
|
||||
#[test]
|
||||
fn datasets_csv_header_prefixes_define_variables() {
|
||||
let mut s = session();
|
||||
let csv = "name,#Shown,@Photo,Title\nA,true,a.png,x";
|
||||
let csv = "#Shown,@Photo,Title\ntrue,a.png,x";
|
||||
let imp = s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!(imp["defined"], json!(["Shown", "Photo", "Title"]));
|
||||
assert_eq!((imp["imported"].as_u64(), imp["names"].clone()), (Some(1), json!(["A"])));
|
||||
assert_eq!((imp["imported"].as_u64(), imp["names"].clone()), (Some(1), json!(["Data Set 1"])));
|
||||
let listed = s.execute("variable.list", &json!({})).unwrap();
|
||||
assert_eq!(
|
||||
listed["variables"],
|
||||
@@ -1580,23 +1648,23 @@ fn datasets_csv_header_prefixes_define_variables() {
|
||||
{"name": "Title", "kind": "text", "bindings": 0},
|
||||
])
|
||||
);
|
||||
let value = |s: &Session, var: &str| s.doc().unwrap().doc.variables.dataset("A").and_then(|d| d.values.get(var)).cloned();
|
||||
let value = |s: &Session, var: &str| s.doc().unwrap().doc.variables.dataset("Data Set 1").and_then(|d| d.values.get(var)).cloned();
|
||||
assert_eq!(value(&s, "Shown"), Some(vectorcraft_doc::DataValue::Visible(true)));
|
||||
assert_eq!(value(&s, "Photo"), Some(vectorcraft_doc::DataValue::Image("a.png".into())));
|
||||
assert_eq!(value(&s, "Title"), Some(vectorcraft_doc::DataValue::Text("x".into())));
|
||||
|
||||
let again = s.execute("dataset.import", &json!({"csv": "name\t#Shown\t@Photo\tTitle\nA\tfalse\tb.png\ty"})).unwrap();
|
||||
assert_eq!((again["updated"].as_u64(), again["defined"].clone()), (Some(1), json!([])));
|
||||
let again = s.execute("dataset.import", &json!({"csv": "#Shown\t@Photo\tTitle\nfalse\tb.png\ty"})).unwrap();
|
||||
assert_eq!((again["imported"].as_u64(), again["defined"].clone()), (Some(1), json!([])));
|
||||
assert_eq!(value(&s, "Shown"), Some(vectorcraft_doc::DataValue::Visible(false)));
|
||||
|
||||
let error = |s: &mut Session, csv: &str| s.execute("dataset.import", &json!({"csv": csv})).unwrap_err().to_string();
|
||||
let other_kind = error(&mut s, "name,New,#Title\nB,n,true");
|
||||
let other_kind = error(&mut s, "Shown,New,#Title\ntrue,n,true");
|
||||
assert!(other_kind.contains("column 3:") && other_kind.contains("text variable"), "{other_kind}");
|
||||
let twice = error(&mut s, "Fresh,@Fresh\nn,f.png");
|
||||
assert!(twice.contains("column 2:") && twice.contains("text variable"), "{twice}");
|
||||
let bare = error(&mut s, "name,#\nB,true");
|
||||
let bare = error(&mut s, "Title,#\nx,true");
|
||||
assert!(bare.contains("column 2:") && bare.contains("names no variable"), "{bare}");
|
||||
let long = error(&mut s, &format!("name,@{}\nB,b.png", "n".repeat(256)));
|
||||
let long = error(&mut s, &format!("Title,@{}\nx,b.png", "n".repeat(256)));
|
||||
assert!(long.contains("column 2:") && long.contains("longer than 255"), "{long}");
|
||||
assert_eq!(s.doc().unwrap().doc.variables.variables.len(), 3, "a failed import defines nothing");
|
||||
|
||||
@@ -1620,11 +1688,10 @@ fn datasets_csv_exact_names_win_over_prefixes() {
|
||||
let values = json!({"#Shown": "hello", "@Photo": true, "%Chart": ",Q\nX,3", "Shown": "s.png"});
|
||||
s.execute("dataset.new", &json!({"name": "A", "values": values})).unwrap();
|
||||
let csv = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
assert!(csv.starts_with("name,#Shown,#@Photo,%Chart,@Shown\n"), "{csv}");
|
||||
s.execute("dataset.delete", &json!({"names": ["A"]})).unwrap();
|
||||
assert!(csv.starts_with("#Shown,#@Photo,%Chart,@Shown\n"), "{csv}");
|
||||
let imp = s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!((imp["imported"].as_u64(), imp["defined"].clone()), (Some(1), json!([])));
|
||||
let value = |var: &str| s.doc().unwrap().doc.variables.dataset("A").and_then(|d| d.values.get(var)).cloned();
|
||||
let value = |var: &str| s.doc().unwrap().doc.variables.dataset("Data Set 1").and_then(|d| d.values.get(var)).cloned();
|
||||
assert_eq!(value("#Shown"), Some(vectorcraft_doc::DataValue::Text("hello".into())));
|
||||
assert_eq!(value("@Photo"), Some(vectorcraft_doc::DataValue::Visible(true)));
|
||||
assert_eq!(value("%Chart"), Some(vectorcraft_doc::DataValue::Graph(",Q\nX,3".into())));
|
||||
@@ -1644,7 +1711,7 @@ fn datasets_csv_export_marks_visibility_and_image_columns() {
|
||||
}
|
||||
s.execute("dataset.new", &json!({"name": "A", "values": {"Name": "Ann", "Show": false, "Photo": "a.png"}})).unwrap();
|
||||
let out = s.execute("dataset.export", &json!({})).unwrap();
|
||||
assert_eq!(out["csv"], "name,Name,#Show,@Photo\nA,Ann,false,a.png\n");
|
||||
assert_eq!(out["csv"], "Name,#Show,@Photo\nAnn,false,a.png\n");
|
||||
assert!(out.get("warning").is_none(), "{out}");
|
||||
|
||||
let mut fresh = session();
|
||||
@@ -1653,7 +1720,8 @@ fn datasets_csv_export_marks_visibility_and_image_columns() {
|
||||
let listed = fresh.execute("variable.list", &json!({})).unwrap();
|
||||
let kinds: Vec<_> = listed["variables"].as_array().unwrap().iter().map(|v| v["kind"].clone()).collect();
|
||||
assert_eq!(kinds, [json!("text"), json!("visibility"), json!("image")]);
|
||||
let value = |var: &str| fresh.doc().unwrap().doc.variables.dataset("A").and_then(|d| d.values.get(var)).cloned();
|
||||
let value = |var: &str| fresh.doc().unwrap().doc.variables.dataset("Data Set 1").and_then(|d| d.values.get(var)).cloned();
|
||||
assert_eq!(value("Name"), Some(vectorcraft_doc::DataValue::Text("Ann".into())));
|
||||
assert_eq!(value("Show"), Some(vectorcraft_doc::DataValue::Visible(false)));
|
||||
assert_eq!(value("Photo"), Some(vectorcraft_doc::DataValue::Image("a.png".into())));
|
||||
|
||||
@@ -1662,7 +1730,48 @@ fn datasets_csv_export_marks_visibility_and_image_columns() {
|
||||
}
|
||||
let out = s.execute("dataset.export", &json!({})).unwrap();
|
||||
assert_eq!(out["warning"], "a new document won't read these variables back with their kind: `#Tag`, `Chart`, `@Note`");
|
||||
assert!(out["csv"].as_str().unwrap().starts_with("name,Name,#Show,@Photo,#Tag,Chart,@Note\n"), "{out}");
|
||||
assert!(out["csv"].as_str().unwrap().starts_with("Name,#Show,@Photo,#Tag,Chart,@Note\n"), "{out}");
|
||||
}
|
||||
|
||||
/// A file `dataset.export` writes has no column of data set names, and `dataset.import` reads
|
||||
/// every column as a variable, so the file imports back to the same data sets: the same variables
|
||||
/// with their kinds, and each data set's values in order, a data set without values included, named
|
||||
/// Data Set 1, 2 and so on. In a document whose data sets already have those names and values, the
|
||||
/// import changes nothing.
|
||||
#[test]
|
||||
fn datasets_csv_export_imports_back_to_the_same_data_sets() {
|
||||
let mut s = session();
|
||||
for (name, kind) in [("Name", "text"), ("Show", "visibility"), ("Photo", "image")] {
|
||||
s.execute("variable.define", &json!({"name": name, "kind": kind})).unwrap();
|
||||
}
|
||||
s.execute("dataset.new", &json!({"name": "A", "values": {"Name": "Ann, Jr.", "Show": true, "Photo": "a.png"}})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "B", "values": {"Show": false}})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "C", "values": {}})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "D", "values": {"Name": "", "Photo": "d.png"}})).unwrap();
|
||||
let csv = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
assert!(csv.starts_with("Name,#Show,@Photo\n"), "{csv}");
|
||||
let before = s.doc().unwrap().doc.variables.clone();
|
||||
|
||||
let mut fresh = session();
|
||||
fresh.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
let vars = fresh.doc().unwrap().doc.variables.clone();
|
||||
assert_eq!(vars.variables, before.variables, "the same variables, with their kinds");
|
||||
let values = |v: &vectorcraft_doc::Variables| v.datasets.iter().map(|d| d.values.clone()).collect::<Vec<_>>();
|
||||
assert_eq!(values(&vars), values(&before), "the same values in every data set");
|
||||
let names: Vec<&str> = vars.datasets.iter().map(|d| d.name.as_str()).collect();
|
||||
assert_eq!(names, ["Data Set 1", "Data Set 2", "Data Set 3", "Data Set 4"]);
|
||||
|
||||
// Exporting the imported data sets writes the same file, and importing it changes nothing: no
|
||||
// undo step, and the active data set stays.
|
||||
let again = fresh.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
assert_eq!(again, csv);
|
||||
fresh.execute("dataset.select", &json!({"name": "Data Set 1"})).unwrap();
|
||||
let vars = fresh.doc().unwrap().doc.variables.clone();
|
||||
assert!(vars.active_dataset.is_some());
|
||||
let undo = fresh.doc().unwrap().history.undo.len();
|
||||
fresh.execute("dataset.import", &json!({"csv": again})).unwrap();
|
||||
assert_eq!(fresh.doc().unwrap().history.undo.len(), undo, "no undo step");
|
||||
assert_eq!(fresh.doc().unwrap().doc.variables, vars);
|
||||
}
|
||||
|
||||
/// The export's `warning` names the variables whose column a document without the variables
|
||||
@@ -1719,7 +1828,7 @@ fn datasets_csv_export_refuses_a_header_that_names_another_variable() {
|
||||
fn datasets_csv_refuses_graph_data_file_columns() {
|
||||
let mut s = session();
|
||||
let error = |s: &mut Session, csv: &str| s.execute("dataset.import", &json!({"csv": csv})).unwrap_err().to_string();
|
||||
let named = error(&mut s, "name,Title,%Sales\nA,x,sales.txt");
|
||||
let named = error(&mut s, "Name,Title,%Sales\nA,x,sales.txt");
|
||||
assert!(named.contains("column 3:") && named.contains("`%Sales`"), "{named}");
|
||||
let unnamed = error(&mut s, "%Sales\tTitle\nsales.txt\tx");
|
||||
assert!(unnamed.contains("column 1:") && unnamed.contains("`%Sales`"), "{unnamed}");
|
||||
@@ -1735,10 +1844,10 @@ fn datasets_csv_refuses_two_columns_for_one_variable() {
|
||||
s.execute("variable.define", &json!({"name": "Shown", "kind": "visibility"})).unwrap();
|
||||
let error = |s: &mut Session, csv: &str| s.execute("dataset.import", &json!({"csv": csv})).unwrap_err().to_string();
|
||||
for (csv, first, second, var) in [
|
||||
("name,City,City\nR,Paris,Rome", 2, 3, "City"),
|
||||
("name,City,City\nR,Paris,", 2, 3, "City"),
|
||||
("Title,City,City\nR,Paris,Rome", 2, 3, "City"),
|
||||
("Title,City,City\nR,Paris,", 2, 3, "City"),
|
||||
("City,Title,City\nParis,x,Rome", 1, 3, "City"),
|
||||
("name,Shown,#Shown\nR,true,false", 2, 3, "Shown"),
|
||||
("Title,Shown,#Shown\nR,true,false", 2, 3, "Shown"),
|
||||
("Title\t@Photo\t@ Photo\nx\ta.png\tb.png", 2, 3, "Photo"),
|
||||
] {
|
||||
let err = error(&mut s, csv);
|
||||
@@ -1757,21 +1866,22 @@ fn locked_variables_refuse_header_cells_that_define_variables() {
|
||||
s.execute("variable.define", &json!({"name": "Shown", "kind": "visibility"})).unwrap();
|
||||
s.execute("variable.define", &json!({"name": "Title", "kind": "text"})).unwrap();
|
||||
s.execute("variable.lock", &json!({"locked": true})).unwrap();
|
||||
for csv in ["name,#Shown,Title,New\nA,true,x,n", "@Photo\nb.png", "name,#Fresh\nA,true"] {
|
||||
for csv in ["#Shown,Title,New\ntrue,x,n", "@Photo\nb.png", "Title,#Fresh\nx,true"] {
|
||||
let e = s.execute("dataset.import", &json!({"csv": csv})).unwrap_err();
|
||||
assert!(matches!(&e, EngineError::Disabled(cmd, why) if cmd == "dataset.import" && why == "variables are locked"), "{csv}: {e}");
|
||||
}
|
||||
let vars = &s.doc().unwrap().doc.variables;
|
||||
assert!(vars.variables.len() == 2 && vars.datasets.is_empty(), "a refused import defines and imports nothing");
|
||||
|
||||
let imp = s.execute("dataset.import", &json!({"csv": "name,#Shown,Title\nA,true,x"})).unwrap();
|
||||
let imp = s.execute("dataset.import", &json!({"csv": "#Shown,Title\ntrue,x"})).unwrap();
|
||||
assert_eq!((imp["imported"].as_u64(), imp["defined"].clone()), (Some(1), json!([])));
|
||||
s.execute("variable.lock", &json!({"locked": false})).unwrap();
|
||||
let imp = s.execute("dataset.import", &json!({"csv": "name,New\nB,n"})).unwrap();
|
||||
let imp = s.execute("dataset.import", &json!({"csv": "New\nn"})).unwrap();
|
||||
assert_eq!(imp["defined"], json!(["New"]));
|
||||
}
|
||||
|
||||
/// Import / export datasets as CSV: header `name` then variables; round-trip keeps values.
|
||||
/// Import / export datasets as CSV: one column per variable and no dataset names; round-trip keeps
|
||||
/// the values in order, and the rows come back as Data Set 1, 2.
|
||||
#[test]
|
||||
fn datasets_csv_round_trips_and_rejects_bad_input() {
|
||||
let dir = Folder::new("vars-csv");
|
||||
@@ -1795,28 +1905,27 @@ fn datasets_csv_round_trips_and_rejects_bad_input() {
|
||||
|
||||
let exported = s.execute("dataset.export", &json!({})).unwrap();
|
||||
let csv = exported["csv"].as_str().unwrap().to_string();
|
||||
assert!(csv.starts_with("name,Name,#Show,@Photo\n"), "{csv}");
|
||||
assert!(csv.starts_with("Name,#Show,@Photo\n"), "{csv}");
|
||||
assert!(csv.contains("Alice") && csv.contains("Bob") && csv.contains("false"), "{csv}");
|
||||
|
||||
// Clear rows, import again, apply B.
|
||||
s.execute("dataset.delete", &json!({"names": ["A", "B"]})).unwrap();
|
||||
// Import again, which replaces A and B, and apply the second row.
|
||||
let imp = s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!((imp["imported"].as_u64(), imp["updated"].as_u64()), (Some(2), Some(0)));
|
||||
s.execute("dataset.select", &json!({"name": "B"})).unwrap();
|
||||
assert_eq!((imp["imported"].as_u64(), &imp["deletedDataSets"]), (Some(2), &json!(["A", "B"])));
|
||||
s.execute("dataset.select", &json!({"name": "Data Set 2"})).unwrap();
|
||||
assert_eq!(plain(&s, t), "Bob");
|
||||
assert!(!s.doc().unwrap().doc.node(r).unwrap().visible);
|
||||
assert_eq!(image(&s, img).link.as_ref().map(|l| l.path.as_str()), Some(blue_path.as_str()));
|
||||
|
||||
// Re-import updates in place and defines nothing; a bad visibility value is an error.
|
||||
// Re-import replaces the rows and defines nothing; a bad visibility value is an error.
|
||||
let again = s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!((again["imported"].as_u64(), again["updated"].as_u64()), (Some(0), Some(2)));
|
||||
assert_eq!((again["imported"].as_u64(), &again["deletedDataSets"]), (Some(2), &json!(["Data Set 1", "Data Set 2"])));
|
||||
assert_eq!(again["defined"], json!([]));
|
||||
assert!(s.execute("dataset.import", &json!({"csv": "name,Show\nX,maybe"})).is_err());
|
||||
assert!(s.execute("dataset.import", &json!({"csv": "Show\nmaybe"})).is_err());
|
||||
|
||||
let out = dir.file("sets.csv");
|
||||
s.execute("dataset.export", &json!({"path": out})).unwrap();
|
||||
let from_file = s.execute("dataset.import", &json!({"path": out})).unwrap();
|
||||
assert_eq!(from_file["updated"].as_u64(), Some(2));
|
||||
assert_eq!(from_file["names"], json!(["Data Set 1", "Data Set 2"]));
|
||||
|
||||
// One undo undoes the last import's replacements.
|
||||
s.execute("edit.undo", &json!({})).unwrap();
|
||||
@@ -1831,17 +1940,17 @@ fn datasets_csv_reads_windows_1252_files() {
|
||||
let import = |s: &mut Session, bytes: &[u8]| s.execute("dataset.import", &json!({"dataBase64": vectorcraft_format::base64_encode(bytes)}));
|
||||
let value = |s: &Session, name: &str| s.doc().unwrap().doc.variables.dataset(name).and_then(|d| d.values.get("City")).cloned();
|
||||
let text_value = |v: &str| Some(vectorcraft_doc::DataValue::Text(v.into()));
|
||||
let imp = import(&mut s, b"name,City\r\nA,Jos\xe9\r\nB,\x80 5 \x96 \x93Caf\xe9\x94\r\n").unwrap();
|
||||
assert_eq!(imp["names"], json!(["A", "B"]));
|
||||
assert_eq!(value(&s, "A"), text_value("José"));
|
||||
assert_eq!(value(&s, "B"), text_value("€ 5 – “Café”"));
|
||||
import(&mut s, "name,City\nC,José\n".as_bytes()).unwrap();
|
||||
assert_eq!(value(&s, "C"), text_value("José"));
|
||||
import(&mut s, "\u{feff}name,City\nD,Zoë\n".as_bytes()).unwrap();
|
||||
assert_eq!(value(&s, "D"), text_value("Zoë"));
|
||||
let e = import(&mut s, b"\xef\xbb\xbfname,City\nE,Jos\xe9\n").unwrap_err().to_string();
|
||||
let imp = import(&mut s, b"City\r\nJos\xe9\r\n\x80 5 \x96 \x93Caf\xe9\x94\r\n").unwrap();
|
||||
assert_eq!(imp["names"], json!(["Data Set 1", "Data Set 2"]));
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("José"));
|
||||
assert_eq!(value(&s, "Data Set 2"), text_value("€ 5 – “Café”"));
|
||||
import(&mut s, "City\nJosé\n".as_bytes()).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("José"));
|
||||
import(&mut s, "\u{feff}City\nZoë\n".as_bytes()).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("Zoë"));
|
||||
let e = import(&mut s, b"\xef\xbb\xbfCity\nJos\xe9\n").unwrap_err().to_string();
|
||||
assert!(e.contains("CSV is not UTF-8"), "{e}");
|
||||
assert_eq!(value(&s, "E"), None);
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("Zoë"));
|
||||
}
|
||||
|
||||
/// A CSV that starts with a UTF-16 byte-order mark is read as UTF-16, and
|
||||
@@ -1858,24 +1967,24 @@ fn datasets_csv_reads_utf16_files() {
|
||||
let units = text.encode_utf16().flat_map(|u| if le { u.to_le_bytes() } else { u.to_be_bytes() });
|
||||
bom.iter().copied().chain(units).collect()
|
||||
};
|
||||
import(&mut s, &utf16(b"\xff\xfe", "name\tCity\r\nA\tZoë 😀\r\n", true)).unwrap();
|
||||
assert_eq!(value(&s, "A"), text_value("Zoë 😀"));
|
||||
import(&mut s, &utf16(b"\xfe\xff", "name,City\nB,José\n", false)).unwrap();
|
||||
assert_eq!(value(&s, "B"), text_value("José"));
|
||||
let mut odd = utf16(b"\xff\xfe", "name,City\nC,x\n", true);
|
||||
import(&mut s, &utf16(b"\xff\xfe", "City\r\nZoë 😀\r\n", true)).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("Zoë 😀"));
|
||||
import(&mut s, &utf16(b"\xfe\xff", "City\nJosé\n", false)).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("José"));
|
||||
let mut odd = utf16(b"\xff\xfe", "City\nx\n", true);
|
||||
odd.push(b'y');
|
||||
let e = import(&mut s, &odd).unwrap_err().to_string();
|
||||
assert!(e.contains("CSV is not UTF-16"), "{e}");
|
||||
let e = import(&mut s, &[b"\xff\xfe".as_slice(), "name,City\nD,".as_bytes(), b"\x00\xd8"].concat()).unwrap_err().to_string();
|
||||
let e = import(&mut s, &[b"\xff\xfe".as_slice(), "City\nAB,".as_bytes(), b"\x00\xd8"].concat()).unwrap_err().to_string();
|
||||
assert!(e.contains("CSV is not UTF-16"), "{e}");
|
||||
let e = import(&mut s, &utf16(b"", "name,City\nE,x\n", true)).unwrap_err().to_string();
|
||||
let e = import(&mut s, &utf16(b"", "City\nx\n", true)).unwrap_err().to_string();
|
||||
assert!(e.contains("CSV holds NUL bytes"), "{e}");
|
||||
assert_eq!((value(&s, "C"), value(&s, "D"), value(&s, "E")), (None, None, None));
|
||||
s.execute("dataset.new", &json!({"name": "F", "values": {"City": "a\u{0}b"}})).unwrap();
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("José"), "a refused file changes nothing");
|
||||
s.execute("dataset.set", &json!({"name": "Data Set 1", "values": {"City": "a\u{0}b"}})).unwrap();
|
||||
let csv = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
s.execute("dataset.delete", &json!({"names": ["F"]})).unwrap();
|
||||
s.execute("dataset.delete", &json!({"names": ["Data Set 1"]})).unwrap();
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!(value(&s, "F"), text_value("a\u{0}b"));
|
||||
assert_eq!(value(&s, "Data Set 1"), text_value("a\u{0}b"));
|
||||
}
|
||||
|
||||
/// A data set of [`library`]: its name, and each variable's name with its paragraphs.
|
||||
@@ -1993,31 +2102,30 @@ fn a_variable_library_replaces_every_variable_and_data_set() {
|
||||
}
|
||||
|
||||
/// An import or a library load that leaves the variables and data sets as they were (the same CSV
|
||||
/// imported again, a row rewritten with its own values, the same library loaded again) records no
|
||||
/// undo step: the step to redo stays, and the document stays unmodified. An import or load that
|
||||
/// changes something is still one undo step.
|
||||
/// imported again, the same rows tab-separated, the file Export Data Sets writes, the same library
|
||||
/// loaded again) records no undo step: the step to redo stays, and the document stays unmodified.
|
||||
/// An import or load that changes something is still one undo step.
|
||||
#[test]
|
||||
fn an_import_or_library_load_that_changes_nothing_records_no_undo_step() {
|
||||
let mut s = session();
|
||||
let t = text(&mut s, "Alice");
|
||||
let csv = "name,Name,#Shown\nA,Ann,true\nB,Bob,false\n";
|
||||
let csv = "Name,#Shown\nAnn,true\nBob,false\n";
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
for csv in [csv, "name\tName\t#Shown\nB\tBob\tfalse\n", "Name,#Shown\nAnn,true\n"] {
|
||||
let exported = s.execute("dataset.export", &json!({})).unwrap()["csv"].as_str().unwrap().to_string();
|
||||
for csv in [csv, "Name\t#Shown\nAnn\ttrue\nBob\tfalse\n", exported.as_str()] {
|
||||
super::select_with_a_step_to_redo(&mut s, &[t]);
|
||||
let before = super::history(&s);
|
||||
assert_eq!((before.1, before.2), (1, false), "a step to redo, and nothing unsaved");
|
||||
let r = s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
if csv.starts_with("name") {
|
||||
assert_eq!((super::history(&s), r["imported"].clone()), (before, json!(0)), "{csv}");
|
||||
} else {
|
||||
// Without a name column the row is a new data set, which is a change.
|
||||
assert_eq!((super::history(&s), r["imported"].clone()), ((before.0 + 1, 0, true), json!(1)), "{csv}");
|
||||
}
|
||||
assert_eq!((super::history(&s), r["imported"].clone()), (before, json!(2)), "{csv}");
|
||||
}
|
||||
// A file with one of the two rows replaces both data sets with one, which is a change.
|
||||
for csv in ["Name,#Shown\nAnn,true\n", "Name\nCy\n"] {
|
||||
super::select_with_a_step_to_redo(&mut s, &[t]);
|
||||
let undo = super::history(&s).0;
|
||||
s.execute("dataset.import", &json!({"csv": csv})).unwrap();
|
||||
assert_eq!(super::history(&s), (undo + 1, 0, true), "{csv}");
|
||||
}
|
||||
super::select_with_a_step_to_redo(&mut s, &[t]);
|
||||
let undo = super::history(&s).0;
|
||||
s.execute("dataset.import", &json!({"csv": "name,Name\nB,Cy\n"})).unwrap();
|
||||
assert_eq!(super::history(&s), (undo + 1, 0, true));
|
||||
|
||||
let xml = library(&[("Name", "textcontent")], &[("A", &[("Name", &["Ann"])]), ("B", &[("Name", &["Bob"])])]);
|
||||
s.execute("variable.loadLibrary", &json!({"xml": xml})).unwrap();
|
||||
|
||||
@@ -1792,8 +1792,8 @@ fn dataset_import_repeated_column_error_is_translated() {
|
||||
use vectorcraft_engine::{EngineError, Session};
|
||||
let mut s = Session::new();
|
||||
s.execute("file.new", &json!({})).unwrap();
|
||||
let e = s.execute("dataset.import", &json!({"csv": "name,City,City\nR,Paris,Rome"})).unwrap_err();
|
||||
let want = "columns 2 and 3 both name the variable `City`";
|
||||
let e = s.execute("dataset.import", &json!({"csv": "City,City\nParis,Rome"})).unwrap_err();
|
||||
let want = "columns 1 and 2 both name the variable `City`";
|
||||
assert!(matches!(&e, EngineError::BadParams { msg, .. } if msg == want), "{e}");
|
||||
let e = e.to_string();
|
||||
for code in COMPLETE_MESSAGES {
|
||||
@@ -1842,8 +1842,8 @@ fn dataset_export_limit_errors_are_translated() {
|
||||
use vectorcraft_engine::{EngineError, Session};
|
||||
let mut s = Session::new();
|
||||
s.execute("file.new", &json!({})).unwrap();
|
||||
let rows: String = (1..=1000).map(|i| format!("Set {i},x\n")).collect();
|
||||
s.execute("dataset.import", &json!({"csv": format!("name,V1\n{rows}")})).unwrap();
|
||||
let rows = "x\n".repeat(1000);
|
||||
s.execute("dataset.import", &json!({"csv": format!("V1\n{rows}")})).unwrap();
|
||||
s.execute("dataset.new", &json!({"name": "Set 1001"})).unwrap();
|
||||
let mut errors = vec![];
|
||||
let e = s.execute("dataset.export", &json!({})).unwrap_err();
|
||||
@@ -1906,8 +1906,8 @@ fn dataset_import_cell_errors_are_translated() {
|
||||
s.execute("variable.define", &json!({"name": "Sales", "kind": "graph"})).unwrap();
|
||||
let mut errors = vec![];
|
||||
for (csv, want) in [
|
||||
("name,Show\nX,maybe".to_string(), "line 2: value for `Show` must be true or false"),
|
||||
(format!("name,Sales\nX,\"{}\"", "1,".repeat(140_000)), "line 2: value for `Sales` is longer than 262144 bytes"),
|
||||
("Show\nmaybe".to_string(), "line 2: value for `Show` must be true or false"),
|
||||
(format!("Sales\n\"{}\"", "1,".repeat(140_000)), "line 2: value for `Sales` is longer than 262144 bytes"),
|
||||
] {
|
||||
let e = s.execute("dataset.import", &json!({"csv": csv})).unwrap_err();
|
||||
assert!(matches!(&e, EngineError::BadParams { msg, .. } if msg == want), "{e}");
|
||||
|
||||
@@ -429,23 +429,24 @@ mod tests {
|
||||
let mut app = VectorcraftApp::new(vectorcraft_engine::Session::new(), Default::default());
|
||||
app.session.execute("file.new", &json!({"width": 300, "height": 200})).unwrap();
|
||||
let t = app.session.execute("text.create", &json!({"x": 10, "y": 40, "text": "x"})).unwrap()["id"].as_u64().unwrap();
|
||||
app.run("dataset.import", json!({"csv": "name,Name\nA,Ann\nB,Bob\n"})).unwrap();
|
||||
app.run("dataset.import", json!({"csv": "Name\nAnn\nBob\n"})).unwrap();
|
||||
let (a, b) = ("Data Set 1", "Data Set 2");
|
||||
let message = |row: &str| format!("nothing in data set {row} was applied: no object is bound to a variable it has a value for");
|
||||
app.run("dataset.select", json!({"name": "A"})).unwrap();
|
||||
assert_eq!(app.ui.status, message("A"));
|
||||
app.run("dataset.select", json!({"name": a})).unwrap();
|
||||
assert_eq!(app.ui.status, message(a));
|
||||
app.run("dataset.next", json!({})).unwrap();
|
||||
assert_eq!(app.ui.status, message("B"));
|
||||
assert_eq!(app.ui.status, message(b));
|
||||
app.run("dataset.prev", json!({})).unwrap();
|
||||
assert_eq!(app.ui.status, message("A"));
|
||||
assert_eq!(app.ui.status, message(a));
|
||||
|
||||
app.run("variable.bind", json!({"variable": "Name", "ids": [t]})).unwrap();
|
||||
app.run("dataset.select", json!({"name": "B"})).unwrap();
|
||||
app.run("dataset.select", json!({"name": b})).unwrap();
|
||||
assert_eq!(app.ui.status, "");
|
||||
// Stepping to a data set that applies clears the message as well.
|
||||
for step in ["dataset.next", "dataset.prev"] {
|
||||
app.run("variable.unbind", json!({"variable": "Name", "ids": [t]})).unwrap();
|
||||
app.run("dataset.select", json!({"name": "A"})).unwrap();
|
||||
assert_eq!(app.ui.status, message("A"));
|
||||
app.run("dataset.select", json!({"name": a})).unwrap();
|
||||
assert_eq!(app.ui.status, message(a));
|
||||
app.run("variable.bind", json!({"variable": "Name", "ids": [t]})).unwrap();
|
||||
app.run(step, json!({})).unwrap();
|
||||
assert_eq!(app.ui.status, "", "{step}");
|
||||
|
||||
@@ -238,8 +238,8 @@ fn load_profile_asks_off_the_ui_thread() {
|
||||
}
|
||||
|
||||
/// The Variables panel menu's Import Data Sets… and Export Data Sets… ask off the UI thread: the
|
||||
/// data sets in the file picked are loaded, and the header and data set `B` are written to the
|
||||
/// path the dialog answers.
|
||||
/// data sets in the file picked are loaded, and the header and the data set's row are written to
|
||||
/// the path the dialog answers.
|
||||
#[test]
|
||||
fn variables_panel_data_sets_ask_off_the_ui_thread() {
|
||||
let (mut app, shown, _) = app();
|
||||
@@ -250,20 +250,20 @@ fn variables_panel_data_sets_ask_off_the_ui_thread() {
|
||||
}
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let (csv, out) = (dir.join("sets.csv"), dir.join("out.csv"));
|
||||
std::fs::write(&csv, "name,Name\nB,y\n").unwrap();
|
||||
std::fs::write(&csv, "Name\ny\n").unwrap();
|
||||
let sets = |app: &VectorcraftApp| app.session.doc().unwrap().doc.variables.datasets.iter().map(|d| d.name.clone()).collect::<Vec<_>>();
|
||||
|
||||
click(&mut app, "Import Data Sets…", crate::panels::variables::menu);
|
||||
assert!(matches!(shown.requests().as_slice(), [PickRequest::Open(_)]));
|
||||
shown.answer(&[&csv.to_string_lossy()]);
|
||||
poll(&mut app);
|
||||
assert_eq!(sets(&app), ["B"], "{}", app.ui.status);
|
||||
assert_eq!(sets(&app), ["Data Set 1"], "{}", app.ui.status);
|
||||
|
||||
click(&mut app, "Export Data Sets…", crate::panels::variables::menu);
|
||||
assert!(matches!(shown.requests().as_slice(), [PickRequest::Save(_)]));
|
||||
shown.answer(&[&out.to_string_lossy()]);
|
||||
poll(&mut app);
|
||||
assert_eq!(std::fs::read_to_string(&out).ok().as_deref(), Some("name,Name\nB,y\n"), "{}", app.ui.status);
|
||||
assert_eq!(std::fs::read_to_string(&out).ok().as_deref(), Some("Name\ny\n"), "{}", app.ui.status);
|
||||
let _ = std::fs::remove_dir_all(dir);
|
||||
}
|
||||
|
||||
|
||||
+7
-4
@@ -1684,10 +1684,13 @@ already walks Image Trace. Short agent entry points:
|
||||
renames the row, in the same undo step; Edit Data Set's Name field sends it), `dataset.rename`,
|
||||
`dataset.delete`, `dataset.capture`, `dataset.update`, `dataset.select {name}`, `dataset.next` /
|
||||
`dataset.prev` (each `dataset.select` / next / prev is one undo step over every bound object), `dataset.import {path|data|dataBase64|csv}` /
|
||||
`dataset.export {path?}` (CSV: `name`, then one column per variable). `dataset.import` also reads tab-separated
|
||||
files, and files whose first header cell isn't `name`: every header cell is then a variable, and the rows take the
|
||||
first free names `Data Set 1`, `Data Set 2` and so on. An import error about a row's values or name gives the file
|
||||
line the row starts on. A header cell that names no variable defines one, unbound, in the same undo step as the
|
||||
`dataset.export {path?}` (CSV: one column per variable, and no column of data set names). `dataset.import` also
|
||||
reads tab-separated files. Every header cell is a variable, whatever its case (`name` and `Name` included), as the
|
||||
[set-up page](https://helpx.adobe.com/illustrator/desktop/automate-visualize-data/automate-actions/set-up-data-source-files.html)
|
||||
describes the first row: "Specify your data field name in the first row of your spreadsheet." The rows replace every
|
||||
data set of the document and are named `Data Set 1`, `Data Set 2` and so on, in file order; the document's variables
|
||||
and their bindings stay, and the reply's `deletedDataSets` lists the data sets that were replaced. An import error
|
||||
about a row's values gives the file line the row starts on. A header cell that names no variable defines one, unbound, in the same undo step as the
|
||||
rows. `#Shown` defines a visibility variable `Shown`, `@Photo` defines an image variable `Photo`, and a cell without
|
||||
a prefix defines a text variable. `dataset.export` writes a visibility variable's header as `#Shown` and an image
|
||||
variable's as `@Photo`, and a text or graph variable's as its name alone. Its reply's `warning` names the variables
|
||||
|
||||
@@ -572,7 +572,7 @@ Moved from the ROADMAP's "Shipped so far" on 2026-10-10. Update it when a featur
|
||||
- Export for Screens, Save for Web (slice images, and files named after the document's title) and the Libraries panel name files with the characters Windows does not allow written as `-` and with a `-` after a Windows device name such as `CON`, on every platform (`document.exportForScreens`, `document.exportForWeb`, `library.create`).
|
||||
- Export for Screens numbers distinct row files whose names collide, compared without case, as `-2`, `-3`… instead of leaving them out; identical rows still write one file. It puts row folders that differ only in case into one folder (`document.exportForScreens`).
|
||||
- Variables (data merge): text, visibility and image variables bound to objects (an object may carry several), dataset rows applied in one undo step (`variable.*`, `dataset.*`) from a Variables panel and Window › Variables, with dialogs to define, bind and edit, Make Text/Visibility/Image Dynamic, Capture/Update Data Set, CSV import/export (`dataset.import` / `dataset.export`), and a row in italics when an edit leaves the art not matching it; an image dataset value is a file path and apply keeps the object's bounds; bindings let go of deleted art and locked art is left alone; every command, like the rest, is drivable over MCP and the CLI.
|
||||
- Variables: Import Data Sets… (`dataset.import`) reads tab-separated files as well as comma-separated ones, and files without a `name` column, whose rows take the first free names Data Set 1, Data Set 2 and so on. An import error about a row's values or name gives the file line the row starts on.
|
||||
- Variables: Import Data Sets… (`dataset.import`) reads tab-separated files as well as comma-separated ones. Every header cell is a variable, and the rows replace the document's data sets as Data Set 1, Data Set 2 and so on. An import error about a row's values gives the file line the row starts on.
|
||||
- Variables: Import Data Sets… (`dataset.import`) defines the variables a file's header names that the document lacks, unbound. `#Shown` defines a visibility variable, `@Photo` defines an image variable, and a header without a prefix defines a text variable. A header that is a variable's exact name keeps naming it, and any other `%` header (graph data files) is refused.
|
||||
- Variables: Load Variable Library… in the Variables panel menu (`variable.loadLibrary`) replaces the document's variables and data sets with a variable library's, read from its XML file, in one undo step: every variable is deleted with its bindings and values, and every data set is deleted; then the library's text variables are defined, unbound, and its data sets take the text of each value's `p` elements, one paragraph each. Variables of other kinds are left out with a warning that names their kinds.
|
||||
- Variables: Save Variable Library… in the Variables panel menu (`variable.saveLibrary`) writes the document's text variables and data sets as a variable library's XML file, which Load Variable Library reads back with the same text variables, data set names and text values. Variables of other kinds are left out with a warning that names their kinds.
|
||||
|
||||
Reference in New Issue
Block a user