sal/src/text/README.md

SAL Text Module (sal::text)

This module provides a collection of utilities for common text processing and manipulation tasks in Rust, with bindings for Rhai scripting.

Overview

The sal::text module offers functionalities for:

• Indentation: Removing common leading whitespace (dedent) and adding prefixes to lines (prefix).
• Normalization: Sanitizing strings for use as filenames (name_fix) or fixing filename components within paths (path_fix).
• Replacement: A powerful TextReplacer for performing single or multiple regex or literal text replacements in strings or files.
• Templating: A TemplateBuilder using the Tera engine to render text templates with dynamic data.

Rust API

1. Text Indentation

Located in src/text/dedent.rs (for dedent) and src/text/fix.rs (likely contains prefix, though not explicitly confirmed by file view, its Rhai registration implies existence).

• dedent(text: &str) -> String: Removes common leading whitespace from a multiline string. Tabs are treated as 4 spaces. Ideal for cleaning up heredocs or indented code snippets.

  use sal::text::dedent;
  let indented_text = "    Hello\n      World";
  assert_eq!(dedent(indented_text), "Hello\n  World");
  

• prefix(text: &str, prefix_str: &str) -> String: Adds prefix_str to the beginning of each line in text.

  use sal::text::prefix;
  let text = "line1\nline2";
  assert_eq!(prefix(text, "> "), "> line1\n> line2");
  

2. Filename and Path Normalization

Located in src/text/fix.rs.

• name_fix(text: &str) -> String: Sanitizes a string to be suitable as a name or filename component. It converts to lowercase, replaces whitespace and various special characters with underscores, and removes non-ASCII characters.

  use sal::text::name_fix;
  assert_eq!(name_fix("My File (New).txt"), "my_file_new_.txt");
  assert_eq!(name_fix("Café crème.jpg"), "caf_crm.jpg");
  

• path_fix(text: &str) -> String: Applies name_fix to the filename component of a given path string, leaving the directory structure intact.

  use sal::text::path_fix;
  assert_eq!(path_fix("/some/path/My Document.docx"), "/some/path/my_document.docx");
  

3. Text Replacement (TextReplacer)

Located in src/text/replace.rs. Provides TextReplacer and TextReplacerBuilder.

The TextReplacer allows for complex, chained replacement operations on strings or file contents.

Builder Pattern:

use sal::text::TextReplacer;

// Example: Multiple replacements, regex and literal
let replacer = TextReplacer::builder()
    .pattern(r"\d+") // Regex: match one or more digits
    .replacement("NUMBER")
    .regex(true)
    .and() // Chain another replacement
    .pattern("World") // Literal string
    .replacement("Universe")
    .regex(false) // Explicitly literal, though default
    .build()
    .expect("Failed to build replacer");

let original_text = "Hello World, item 123 and item 456.";
let modified_text = replacer.replace(original_text);
assert_eq!(modified_text, "Hello Universe, item NUMBER and item NUMBER.");

// Case-insensitive regex example
let case_replacer = TextReplacer::builder()
    .pattern("apple")
    .replacement("FRUIT")
    .regex(true)
    .case_insensitive(true)
    .build()
    .unwrap();
assert_eq!(case_replacer.replace("Apple and apple"), "FRUIT and FRUIT");

Key TextReplacerBuilder methods:

• pattern(pat: &str): Sets the search pattern (string or regex).
• replacement(rep: &str): Sets the replacement string.
• regex(yes: bool): If true, treats pattern as a regex. Default is false (literal).
• case_insensitive(yes: bool): If true (and regex is true), performs case-insensitive matching.
• and(): Finalizes the current replacement operation and prepares for a new one.
• build(): Consumes the builder and returns a Result<TextReplacer, String>.

TextReplacer methods:

• replace(input: &str) -> String: Applies all configured replacements to the input string.
• replace_file(path: P) -> io::Result<String>: Reads a file, applies replacements, returns the result.
• replace_file_in_place(path: P) -> io::Result<()>: Replaces content in the specified file directly.
• replace_file_to(input_path: P1, output_path: P2) -> io::Result<()>: Reads from input_path, applies replacements, writes to output_path.

4. Text Templating (TemplateBuilder)

Located in src/text/template.rs. Uses the Tera templating engine.

Builder Pattern:

use sal::text::TemplateBuilder;
use std::collections::HashMap;

// Assume "./my_template.txt" contains: "Hello, {{ name }}! You are {{ age }}."

// Create a temporary template file for the example
std::fs::write("./my_template.txt", "Hello, {{ name }}! You are {{ age }}.").unwrap();

let mut builder = TemplateBuilder::open("./my_template.txt").expect("Template not found");

// Add variables individually
builder = builder.add_var("name", "Alice").add_var("age", 30);

let rendered_string = builder.render().expect("Rendering failed");
assert_eq!(rendered_string, "Hello, Alice! You are 30.");

// Or add multiple variables from a HashMap
let mut vars = HashMap::new();
vars.insert("name", "Bob");
vars.insert("age", "25"); // Values in HashMap are typically strings or serializable types

let mut builder2 = TemplateBuilder::open("./my_template.txt").unwrap();
builder2 = builder2.add_vars(vars);
let rendered_string2 = builder2.render().unwrap();
assert_eq!(rendered_string2, "Hello, Bob! You are 25.");

// Render directly to a file
// builder.render_to_file("output.txt").expect("Failed to write to file");

// Clean up temporary file
std::fs::remove_file("./my_template.txt").unwrap();

Key TemplateBuilder methods:

• open(template_path: P) -> io::Result<Self>: Loads the template file.
• add_var(name: S, value: V) -> Self: Adds a single variable to the context.
• add_vars(vars: HashMap<S, V>) -> Self: Adds multiple variables from a HashMap.
• render() -> Result<String, tera::Error>: Renders the template to a string.
• render_to_file(output_path: P) -> io::Result<()>: Renders the template and writes it to the specified file.

Rhai Scripting with herodo

The sal::text module's functionalities are exposed to Rhai scripts when using herodo.

Direct Functions

• dedent(text_string): Removes common leading whitespace.
  • Example: let clean_script = dedent(" if true {\n print(\"indented\");\n }");
• prefix(text_string, prefix_string): Adds prefix_string to each line of text_string.
  • Example: let prefixed_text = prefix("hello\nworld", "# ");
• name_fix(text_string): Normalizes a string for use as a filename.
  • Example: let filename = name_fix("My Document (V2).docx"); // "my_document_v2_.docx"
• path_fix(path_string): Normalizes the filename part of a path.
  • Example: let fixed_path = path_fix("/uploads/User Files/Report [Final].pdf");

TextReplacer

Provides text replacement capabilities through a builder pattern.

1. Create a builder: let builder = text_replacer_new();
2. Configure replacements (methods return the builder for chaining):
   • builder = builder.pattern(search_pattern_string);
   • builder = builder.replacement(replacement_string);
   • builder = builder.regex(is_regex_bool); (default false)
   • builder = builder.case_insensitive(is_case_insensitive_bool); (default false, only applies if regex is true)
   • builder = builder.and(); (to add the current replacement and start a new one)
3. Build the replacer: let replacer = builder.build();
4. Use the replacer:
   • let modified_text = replacer.replace(original_text_string);
   • let modified_text_from_file = replacer.replace_file(input_filepath_string);
   • replacer.replace_file_in_place(filepath_string);
   • replacer.replace_file_to(input_filepath_string, output_filepath_string);

TemplateBuilder

Provides text templating capabilities.

1. Open a template file: let tpl_builder = template_builder_open(template_filepath_string);
2. Add variables (methods return the builder for chaining):
   • tpl_builder = tpl_builder.add_var(name_string, value); (value can be string, int, float, bool, or array)
   • tpl_builder = tpl_builder.add_vars(map_object); (map keys are variable names, values are their corresponding values)
3. Render the template:
   • let rendered_string = tpl_builder.render();
   • tpl_builder.render_to_file(output_filepath_string);

Rhai Example

// Create a temporary file for template demonstration
let template_content = "Report for {{user}}:\nItems processed: {{count}}.\nStatus: {{status}}.";
let template_path = "./temp_report_template.txt";

// Using file.write (assuming sal::file module is available and registered)
// For this example, we'll assume a way to write this file or that it exists.
// For a real script, ensure the file module is used or the file is pre-existing.
print(`Intending to write template to: ${template_path}`);
// In a real scenario: file.write(template_path, template_content);

// For demonstration, let's simulate it exists for the template_builder_open call.
// If file module is not used, this script part needs adjustment or pre-existing file.

// --- Text Normalization ---
let raw_filename = "User's Report [Draft 1].md";
let safe_filename = name_fix(raw_filename);
print(`Safe filename: ${safe_filename}`); // E.g., "users_report_draft_1_.md"

let raw_path = "/data/project files/Final Report (2023).pdf";
let safe_path = path_fix(raw_path);
print(`Safe path: ${safe_path}`); // E.g., "/data/project files/final_report_2023_.pdf"

// --- Dedent and Prefix ---
let script_block = "\n    for item in items {\n        print(item);\n    }\n";
let dedented_script = dedent(script_block);
print("Dedented script:\n" + dedented_script);

let prefixed_log = prefix("Operation successful.\nDetails logged.", "LOG: ");
print(prefixed_log);

// --- TextReplacer Example ---
let text_to_modify = "The quick brown fox jumps over the lazy dog. The dog was very lazy.";

let replacer_builder = text_replacer_new()
    .pattern("dog")
    .replacement("cat")
    .case_insensitive(true) // Replace 'dog', 'Dog', 'DOG', etc.
    .and()
    .pattern("lazy")
    .replacement("energetic")
    .regex(false); // This is the default, explicit for clarity

let replacer = replacer_builder.build();
let replaced_text = replacer.replace(text_to_modify);
print(`Replaced text: ${replaced_text}`);
// Expected: The quick brown fox jumps over the energetic cat. The cat was very energetic.

// --- TemplateBuilder Example ---
// This part assumes 'temp_report_template.txt' was successfully created with content:
// "Report for {{user}}:\nItems processed: {{count}}.\nStatus: {{status}}."
// If not, template_builder_open will fail. For a robust script, check file existence or create it.

// Create a dummy template file if it doesn't exist for the example to run
// This would typically be done using the file module, e.g. file.write()
// For simplicity here, we'll just print a message if it's missing.
// In a real script: if !file.exists(template_path) { file.write(template_path, template_content); }

// Let's try to proceed assuming the template might exist or skip if not.
// A more robust script would handle the file creation explicitly.

// For the sake of this example, let's create it directly if possible (conceptual)
// This is a placeholder for actual file writing logic.
// if (true) { // Simulate file creation for example purpose
//    std.os.remove_file(template_path); // Clean up if exists
//    let f = std.io.open(template_path, "w"); f.write(template_content); f.close();
// }

// Due to the sandbox, direct file system manipulation like above isn't typically done in Rhai examples
// without relying on registered SAL functions. We'll assume the file exists.

print("Attempting to use template: " + template_path);
// It's better to ensure the file exists before calling template_builder_open
// For this example, we'll proceed, but in a real script, handle file creation.

// Create a dummy file for the template example to work in isolation
// This is not ideal but helps for a self-contained example if file module isn't used prior.
// In a real SAL script, you'd use `file.write`.
let _dummy_template_file_path = "./example_template.rhai.tmp";
// file.write(_dummy_template_file_path, "Name: {{name}}, Age: {{age}}");

// Using a known, simple template string for robustness if file ops are tricky in example context
let tpl_builder = template_builder_open(_dummy_template_file_path); // Use the dummy/known file

if tpl_builder.is_ok() {
    let mut template_engine = tpl_builder.unwrap();
    template_engine = template_engine.add_var("user", "Jane Doe");
    template_engine = template_engine.add_var("count", 150);
    template_engine = template_engine.add_var("status", "Completed");

    let report_output = template_engine.render();
    if report_output.is_ok() {
        print("Generated Report:\n" + report_output.unwrap());
    } else {
        print("Error rendering template: " + report_output.unwrap_err());
    }

    // Example: Render to file
    // template_engine.render_to_file("./generated_report.txt");
    // print("Report also written to ./generated_report.txt");
} else {
    print("Skipping TemplateBuilder example as template file '" + _dummy_template_file_path + "' likely missing or unreadable.");
    print("Error: " + tpl_builder.unwrap_err());
    print("To run this part, ensure '" + _dummy_template_file_path + "' exists with content like: 'Name: {{name}}, Age: {{age}}'");
}

// Clean up dummy file
// file.remove(_dummy_template_file_path);

Note on Rhai Example File Operations: The Rhai example above includes comments about file creation for the TemplateBuilder part. In a real herodo script, you would use sal::file module functions (e.g., file.write, file.exists, file.remove) to manage the template file. For simplicity and to avoid making the example dependent on another module's full setup path, it highlights where such operations would occur. The example tries to use a dummy path and gracefully skips if the template isn't found, which is a common issue when running examples in restricted environments or without proper setup. The core logic of using TemplateBuilder once the template is loaded remains the same.