Add escape in strings control to ruby parser.

- Moving lines after themselves didnt work.
- moving empty lines deleted them.
- "w" didnt set state to unmodified in generic buffers.
- in "hl" aligned the numbers (by digits)

ARCHITECTURE.md

@@ -0,0 +1,149 @@
+# Base.
+
+### `PCH`
+
+Brings in all the external and standard headers. <br/>
+Apart from c++std stuff we have:
+- `mruby` and almost all it's subheaders.
+- `pcre2` (For regex.)
+- `grapheme.h` - The suckless libgrapheme (to find utf8 boundaries)
+- `unicode_width.h` - To find terminal width of unicode char's.
+
+*Maybe try removing some of them ive probably stopped using.*
+
+### `definitions.h`
+
+- Defines the `bed` namespace
+- Forward declaration of `BEd` type needed by all.
+- `fatal_error` definition, a msg, code (quits application on being thrown)
+- `ed_error`. just a message (stops the current cycle of the ed repl.)
+
+### `bed.h`
+
+**As this is based heavily on the internals, and so ill look into it at the end.**
+
+# `Internal`
+
+## `Vase`
+
+- This module doesn't depend on any other.
+- It's namespace is `bed::internal::vase`
+- Vase is the module to do with the actual storage and handling of a disk backed immutable and reference counted avl piece tree.
+- It is a O(log n) per edit text datatype.
+
+### `constants.h`
+
+Defines `PETAL_SIZE_MAX` as `32 * 1024`.
+- `PETAL_SIZE_MAX` limits teh size of a petal which makes multiple O(n) in petal searches O(1) as it is now bounded.
+
+### `storage/`
+
+A storage is something that stores the actual text, the text is actually stored in storages which are referenced by the trees.
+
+#### `storage.h`
+
+Defines the virtual `Storage` class.
+It has the methods:
+- `const char *read(uint64_t pos)`: reads text from position `pos` and returns a pointer to it.
+- `uint64_t length()`: returns the length of the storage.
+- `void retain() / release()`: Methods to help with refcounted storages.
+
+#### `original.h`
+
+Defines the `Original` type of storage.
+This is a refcounted storage.
+
+It has the fields:
+- `atomic_uint64_t refs{0}`: refcount
+- `const char *buf`: pointer to the actual text.
+- `uint64_t len`
+- `int fd = -1`: the file descriptor of teh undelying disk storage.
+
+It adds the method `initialize` over the base class.
+
+`src/vase/storage/original.cc`.
+
+The constructor uses `mkstemp` to make a temp file, then unlinks it to hide it from the fs but keeps the fd alive.
+
+`initialize` then `mmap`s the fd into a non modifyable pointer managed by the kernel then closes the fd itself.
+- This is what gives us a pointer to the file contents that the kernel can load and evict from memory as needed, and therefore if we have say a 500MB file loaded, but the cursor is arount 200MB and the user is only editing a couple lines around that, the rest of the file will not be using your memory, it might be loaded but if needed the kernel can evict it.
+
+The call site fills the fd before calling initialze, it cannot be modified after that.
+
+#### `append.h`
+
+Defines the `Append` type of storage.
+This is not actually refcounted, it is to be owned by the application just once and reused all throughout.
+It contains the actively typed stuff.
+
+It has the fields:
+- `const char *buf`: pointer to the actual text.
+- `uint64_t allocated_capacity`: the amount of bytees that can be written to it without needing to increase the size.
+- `uint64_t current_size`: Stores the size of actual text stored in it. (the cursor)
+- `int fd = -1`: the file descriptor of the underlying disk storage.
+
+It has the methods `append` (with 2 overloads) and private `grow` over the base class.
+`retain` and `release` do nothing.
+
+`src/vase/storage/append.cc`.
+
+The constructor uses `mkstemp` an `unlink` similar to the original storage.
+It then `ftruncate`s the file to `2^30` bytes or 1GiB this makes writing 1gb to the file possible but does'nt neccasarily use up 1gb on the disk.
+But when `mmap`ing we set `PROT_READ | PROT_WRITE` and `MAP_SHARED` to be able to write edits to disk.
+
+`grow` works by doubling the capacity `ftruncat`ing the fd to the new capacity and then on `linux` it uses `mremap` to remap to the new file size, but otherwise we `munmap` and `mmap` into the new size.
+
+The append methods append text into the buffer (either a single char or char * + len), they may grow the storage size. (the char*+len version uses memcpy).
+
+### `shard.h`
+
+Defines the actual peice tree.
+
+The tree is a polymorhic set of classes.
+
+The base class `Shard` has:
+- `enum Kind : uint8_t kind`: Branch or Petal.
+- `uint16_t height`: the tree height (for avl balancing.)
+- `atomic_uint32_t refs`: the refcount.
+- `uint64_t length`: the length in bytes of this (sub)tree.
+- `uint64_t lines`: the number of `\n` in this subtree.
+- It starts refcount at 1.
+
+The `Branch` class adds:
+- `Shard *left/right` the subtrees.
+- its constructor takes l, r and sets length/lines `(l->length + r->length)`
+- and height `1 + std::max(l->height, r->height)`
+
+The `Petal` (leaf) class adds:
+- `Storage *` a pointer to the storage used.
+- `uint64_t pos` the position in the storage its text starts at.
+- it takes `(uint64_t length, uint64_t lines, Storage *source, uint64_t pos)`
+- it retains `source`. (usefull for refcounted sotrages.)
+
+A `nullptr` `Shard*` is an empty peice of text (i.e. valid).
+
+`src/internal/vase/shard.cc`: defines a set of static function on the Shard class. (All the applications of it.)
+
+the operations methods are:
+- `retain/release` retain/releases them, if freeing a leaf it also releases the storage and for branches it releases both subtrees. nullptr's are ignored.
+- `balance/rotate_(right/left) and height/balance factor` are for doing simple avl balancing operations.
+- `concat` borrows 2 pointers to `Shard` and returns an owned Shard*.
+- `split` borrows a `Shard` and returns a `pair<Shard*, Shard*>` of 2 owned shards.
+- `merge_leaves` works similar to concat but it tries to merge the middle bit if they have their physical representation in order. for example if i have a shard `hell` and i type the letter `o` and because i'd been typing in order the Append storage will have text `hello` we can merge them into a single petal.
+- `append` similar to concat, but uses merge_leaves, useful for small appends like when typing.
+- `build` takes a pointer to an array of `Shard*` and start and end then makes a single avl balanced `Shard*` out of it it is generally faster than calling concat on each peice when we have many.
+
+Then for building a Shard* in the first place we have:
+- `from_command(const char *cmd)` runs the \0 terminated command string and reads its STDOUT and returns an owned Shard*.
+- `from_file(const std::filesystem::path &path)` reads the file at path.
+- `from_string(const char *data, uint64_t len)` loads the string given into a new storage, (normally inserting text can work by inserting into the append storage but this function loads it into a `OriginalStorage` object.).
+These methods all clip the final newline if it exists.
+And they also create a new `OriginalStorage` object.
+
+### `vase.h`
+
+### `iterators/`
+
+#### `line.h`
+
+#### `petal.h`

Makefile

@@ -36,7 +36,7 @@ CFLAGS_RELEASE :=\
 	-fomit-frame-pointer -DNDEBUG -s \
 	-I./include -I./libs/unicode_width
 
-CFLAGS_DEBUG += $(PCRE_CFLAGS) $(LIBGRAPHEME_CFLAGS) $(MRUBY_CFLAGS) $(C_SANITIZER)
+CFLAGS_DEBUG += $(PCRE_CFLAGS) $(LIBGRAPHEME_CFLAGS) $(MRUBY_CFLAGS)
 CFLAGS_RELEASE += $(PCRE_CFLAGS) $(LIBGRAPHEME_CFLAGS) $(MRUBY_CFLAGS)
 
 UNICODE_SRC := $(wildcard libs/unicode_width/*.c)

README.md

@@ -56,5 +56,7 @@ It should support:
 - Error handling.
 - And more.
 
-Not done yet.
+### TODO:
 
+- Make "g" command work.
+- properly handle escapes for %q ' etc in ruby parser (rn everything is escapable.)

flake.nix

@@ -70,9 +70,8 @@
 
         buildPhase = ''
           make \
-            MRBC=${mruby}/bin/mrbc \
+            MRUBY_CFLAGS="-I${mruby}/include" \
-            MRUBY_CFLAGS=-I${mruby}/include \
+            MRUBY_LIBS="-L${mruby}/lib -lmruby"
-            MRUBY_LIBS=-L${mruby}/lib -lmruby
         '';
 
         installPhase = ''
@@ -110,7 +109,6 @@
           export CC="ccache $CC"
           export CXX="ccache $CXX"
 
-          export MRBC=${mruby}/bin/mrbc
           export MRUBY_CFLAGS=-I${mruby}/include
           export MRUBY_LIBS="-L${mruby}/lib -lmruby"
         '';

include/bed.h

@@ -6,6 +6,7 @@
 #include "internal/functions/suffixes.h"
 #include "internal/io/io.h"
 #include "internal/marks/marks.h"
+#include "internal/scripting/ruby.h"
 #include "internal/theme/theme.h"
 #include "internal/ui/command.h"
 #include "internal/ui/text_mode.h"
@@ -13,19 +14,22 @@
 
 namespace bed {
 struct BEd {
+  internal::scripting::RubyState mrb;
   internal::trie::Trie<internal::functions::Function> functions;
   internal::functions::Function no_op;
   internal::functions::Function eof_op;
   std::array<std::optional<internal::functions::Suffix>, 26> suffixes;
   internal::theme::Theme theme;
   std::unordered_map<std::string, internal::syntax::Language *> languages;
-  internal::io::IO &io;
+  internal::io::IO io;
   internal::vase::AppendStorage append{"/tmp"};
+  std::unordered_map<std::string, internal::buffer::Buffer *> buffers;
 
   bool help_mode = false;
   bool prompt_mode = true;
   std::function<std::string(BEd &)> prompt = nullptr;
   bool suppress_mode = false;
+  bool color_mode = false;
   bool temporary_current = false;
   std::string last_help = "";
   std::string last_regex = "";
@@ -33,13 +37,11 @@ struct BEd {
   std::string last_replacement = "";
   std::string last_shell = "";
 
-  std::unordered_map<std::string, internal::buffer::Buffer *> buffers;
-
   internal::buffer::Range prev_1;
   internal::buffer::Range prev_2;
   internal::marks::MarksEngine marks;
 
-  BEd(std::vector<std::string> args, internal::io::IO &io);
+  BEd(std::vector<std::string> args);
   ~BEd();
 
   internal::buffer::Buffer &buffer(const std::string &);
@@ -47,6 +49,7 @@ struct BEd {
   internal::buffer::Range &prev();
   void mark(uint8_t, internal::buffer::Line);
 
+  void print_help();
   void handle(std::string_view cmd, bool eof);
   void run();
   void suffix_handle(char s);

include/definitions.h

@@ -15,17 +15,4 @@ struct fatal_error : std::runtime_error {
 struct ed_error : std::runtime_error {
   ed_error(std::string msg) : std::runtime_error(msg) {}
 };
-
-struct Highlight {
-  enum : uint8_t {
-    None = 0,
-    Bold = 1 << 0,
-    Italic = 1 << 1,
-    Strikethrough = 1 << 2,
-    Underline = 1 << 3,
-  };
-  uint32_t fg;
-  uint32_t bg;
-  uint8_t flags;
-};
 } // namespace bed

include/internal/buffer/decl.h

@@ -34,6 +34,7 @@ struct Buffer {
   virtual uint64_t bytes() = 0;
   virtual void set_filename(std::filesystem::path path) = 0;
   virtual std::filesystem::path filename() = 0;
+  virtual void saved_hook() = 0;
   virtual void load(BEd &ctx, vase::Shard *text) = 0;
   virtual vase::Shard *copy(uint64_t start_line, uint64_t end_line) = 0;
   virtual void substitute(

include/internal/buffer/types/clip.h

@@ -10,6 +10,7 @@ struct ClipBuffer : Buffer {
 
   void clip_write(vase::Shard *text);
   bool waste() override;
+  void saved_hook() override;
   uint64_t lines() override;
   uint64_t bytes() override;
   void load(BEd &ctx, vase::Shard *text) override;

include/internal/buffer/types/generic.h

@@ -1,7 +1,7 @@
 #pragma once
 
 #include "../decl.h"
-#include "history.h"
+#include "readonly.h"
 #include "shard.h"
 
 namespace bed::internal::buffer {
@@ -14,10 +14,8 @@ struct HistoryItem {
 
 struct GenericBuffer : ShardBuffer {
   uint64_t base_version{0};
-  std::chrono::system_clock::time_point timestamp{
+  std::chrono::system_clock::time_point timestamp{std::chrono::system_clock::now()};
-    std::chrono::system_clock::now()
+  std::string action{"Created buffer."};
-  };
-  std::string action{"created"};
   std::filesystem::path save_path{};
   std::vector<HistoryItem> undo_stack;
   std::vector<HistoryItem> redo_stack;
@@ -26,12 +24,14 @@ struct GenericBuffer : ShardBuffer {
       : ShardBuffer(name, nullptr, nullptr, Kind::Generic) {}
   ~GenericBuffer();
 
+  void language(BEd &ctx, std::string name);
   void list_history(BEd &ctx);
-  HistoryBuffer *get_history(uint64_t version);
+  ReadonlyBuffer *get_history(uint64_t version);
   void snapshot(std::string action);
   bool undo(BEd &ctx);
   bool redo(BEd &ctx);
-  void prune(int = 0);
+  uint64_t prune(int);
+  void saved_hook() override;
   bool waste() override;
   void load(BEd &ctx, vase::Shard *text) override;
   void set_filename(std::filesystem::path path) override;

include/internal/buffer/types/readonly.h

@@ -4,36 +4,39 @@
 #include "shard.h"
 
 namespace bed::internal::buffer {
-struct HistoryBuffer : ShardBuffer {
+struct ReadonlyBuffer : ShardBuffer {
-  HistoryBuffer(
+  bool useless = true;
+
+  ReadonlyBuffer(
     std::string name, vase::Shard *root,
     const syntax::ParserSnapshot &snapshot
   ) : ShardBuffer(std::move(name), root, snapshot, Kind::History) {}
 
+  void saved_hook() override {}
   bool waste() override {
-    return true;
+    return useless;
   }
   void load(BEd &, vase::Shard *) override {
-    throw ed_error("History buffers are read-only.");
+    throw ed_error("This buffer is read-only.");
   }
   void set_filename(std::filesystem::path) override {}
   std::filesystem::path filename() override {
     return {};
   }
   void substitute(BEd &, uint64_t, uint64_t, std::string &, std::string &, std::string &) override {
-    throw ed_error("History buffers are read-only.");
+    throw ed_error("This buffer is read-only.");
   }
   void join(BEd &, uint64_t, uint64_t) override {
-    throw ed_error("History buffers are read-only.");
+    throw ed_error("This buffer is read-only.");
   }
   void remove(BEd &, uint64_t, uint64_t) override {
-    throw ed_error("History buffers are read-only.");
+    throw ed_error("This buffer is read-only.");
   }
   void append(BEd &, vase::Shard *, uint64_t) override {
-    throw ed_error("History buffers are read-only.");
+    throw ed_error("This buffer is read-only.");
   }
   void replace(BEd &, vase::Shard *, uint64_t, uint64_t) override {
-    throw ed_error("History buffers are read-only.");
+    throw ed_error("This buffer is read-only.");
   }
 };
 } // namespace bed::internal::buffer

include/internal/io/io.h

@@ -2,6 +2,7 @@
 
 #include "definitions.h"
 #include "pch.h"
+#include "tokens.h"
 
 namespace bed::internal::io {
 struct KeyEvent {
@@ -57,13 +58,27 @@ struct IO {
   static void enable_raw();
   static volatile std::atomic_bool resized;
   static void handle_sigwinch(int);
+  BEd &bed;
 
-  IO();
+  static enum struct Mode {
+    PIPE,
+    TERMINAL
+  } mode;
+
+  std::string pipe_input;
+  std::deque<char> input_queue;
+
+  IO(BEd &);
   ~IO();
 
   IO(const IO &) = delete;
   IO &operator=(const IO &) = delete;
 
+  bool interactive() const {
+    return mode == Mode::TERMINAL;
+  }
+  void apply(const io::Token::Kind &t);
+  void reset();
   void enable_mouse();
   void disable_mouse();
 
@@ -71,14 +86,14 @@ struct IO {
   std::pair<uint16_t, uint16_t> cursor_position();
   void move_cursor(uint16_t row, uint16_t col);
 
+  std::pair<std::string, bool> read_pipe();
+
   KeyEvent read_key();
   void write(const char *, uint64_t);
   void write(std::string_view);
   void write_line(std::string_view);
   void run_pty(const std::string &);
 
-  std::deque<char> input_queue;
-
   KeyEvent::ReadResult get_next_byte(char &out);
   void enqueue_bytes(const std::string &bytes);
   static int utf8_seq_len(uint8_t byte);

include/internal/io/tokens.h

@@ -0,0 +1,99 @@
+#pragma once
+
+#include "definitions.h"
+#include "pch.h"
+
+namespace bed::internal::io {
+struct Token {
+  uint64_t start;
+  uint64_t end;
+  enum Kind : uint8_t {
+    TempCurrent,
+    BufferName,
+    AddressSeperator,
+    AddressSymbol,
+    Number,
+    Mark,
+    RubyFunction,
+    RubyArg,
+    Function,
+    Shell,
+    File,
+    Suffix,
+    Color1,
+    Color2,
+    Color3,
+    Color4,
+    Color5,
+    Warning,
+    Data,
+    Shebang,
+    Comment,
+    Error,
+    String,
+    Escape,
+    Interpolation,
+    Regexp,
+    True,
+    False,
+    Char,
+    Keyword,
+    KeywordOperator,
+    Operator,
+    Namespace,
+    Class,
+    Module,
+    Type,
+    Constant,
+    VariableInstance,
+    VariableGlobal,
+    Annotation,
+    Directive,
+    Label,
+    Brace1,
+    Brace2,
+    Brace3,
+    Brace4,
+    Brace5,
+    Heading1,
+    Heading2,
+    Heading3,
+    Heading4,
+    Heading5,
+    Heading6,
+    Blockquote,
+    List,
+    ListItem,
+    Code,
+    LanguageName,
+    LinkLabel,
+    ImageLabel,
+    Link,
+    Table,
+    TableHeader,
+    Italic,
+    Bold,
+    Underline,
+    Strikethrough,
+    HorizontalRule,
+    Tag,
+    Attribute,
+    CheckDone,
+    CheckNotDone,
+    Count
+  } type;
+};
+
+struct Highlight {
+  enum : uint8_t {
+    None = 0,
+    Bold = 1 << 0,
+    Italic = 1 << 1,
+    Strikethrough = 1 << 2,
+    Underline = 1 << 3,
+  };
+  uint32_t fg;
+  uint32_t bg;
+  uint8_t flags;
+};
+} // namespace bed::internal::io

include/internal/parser/parser.h

@@ -85,14 +85,17 @@ struct Parser {
   std::string_view cmd;
   uint16_t i;
   Command *command;
-  std::vector<ui::Token> *tokens;
+  std::vector<io::Token> *tokens;
   CompletionContext *completion;
 
   explicit Parser(
     std::string_view cmd, BEd &bed, Command *command,
-    std::vector<ui::Token> *tokens, CompletionContext *completion
+    std::vector<io::Token> *tokens, CompletionContext *completion
   );
 
+  void token(io::Token::Kind);
+  void end_token();
+
   char peek(uint16_t = 0); // == \0 if at eof.
   std::string_view peek_str(uint16_t = UINT16_MAX);
   void advance(uint16_t = 1);
@@ -107,7 +110,7 @@ struct Parser {
   void operation();
 
   void parse();
-
+  static std::vector<io::Token> get_highlight(std::string_view cmd, BEd &bed);
   static Command get_command(std::string_view, BEd &);
   static std::vector<AddressPromise> get_addresses(std::string_view cmd, BEd &bed);
 };

include/internal/scripting/ruby.h

@@ -0,0 +1,124 @@
+#pragma once
+
+#include "definitions.h"
+#include "pch.h"
+
+namespace bed::internal::scripting {
+struct RubyState {
+  mrb_state *state = nullptr;
+  int arena = 0;
+
+  explicit RubyState(BEd *ctx) : state(mrb_open()) {
+    if (!state)
+      throw fatal_error("Failed to initialize mruby.", 1);
+    state->ud = ctx;
+    arena = mrb_gc_arena_save(state);
+  }
+
+  ~RubyState() {
+    state->ud = nullptr;
+    if (state) {
+      mrb_gc_arena_restore(state, arena);
+      mrb_full_gc(state);
+      mrb_close(state);
+    }
+  }
+
+  RubyState(const RubyState &) = delete;
+  RubyState &operator=(const RubyState &) = delete;
+};
+
+struct Block {
+  mrb_state *mrb = nullptr;
+  mrb_value proc = mrb_nil_value();
+
+  Block(mrb_state *mrb, mrb_value proc) noexcept
+      : mrb(mrb), proc(proc) {
+    mrb_gc_register(mrb, proc);
+  }
+
+  Block() noexcept = default;
+
+  ~Block() noexcept {
+    if (!mrb_nil_p(proc) && mrb)
+      mrb_gc_unregister(mrb, proc);
+  }
+
+  Block(const Block &other) noexcept
+      : mrb(other.mrb), proc(other.proc) {
+    if (mrb && !mrb_nil_p(proc))
+      mrb_gc_register(mrb, proc);
+  }
+
+  Block &operator=(const Block &other) noexcept {
+    if (this != &other) {
+      if (mrb && !mrb_nil_p(proc))
+        mrb_gc_unregister(mrb, proc);
+      mrb = other.mrb;
+      proc = other.proc;
+      if (mrb && !mrb_nil_p(proc))
+        mrb_gc_register(mrb, proc);
+    }
+    return *this;
+  }
+
+  Block(Block &&other) noexcept
+      : mrb(other.mrb), proc(other.proc) {
+    other.mrb = nullptr;
+    other.proc = mrb_nil_value();
+  }
+
+  Block &operator=(Block &&other) noexcept {
+    if (this != &other) {
+      if (mrb && !mrb_nil_p(proc))
+        mrb_gc_unregister(mrb, proc);
+      mrb = other.mrb;
+      proc = other.proc;
+      other.mrb = nullptr;
+      other.proc = mrb_nil_value();
+    }
+    return *this;
+  }
+
+  void set_proc(mrb_state *mrb_, mrb_value new_proc) {
+    if (!mrb_nil_p(proc) && mrb)
+      mrb_gc_unregister(mrb, proc);
+    mrb = mrb_;
+    proc = new_proc;
+    mrb_gc_register(mrb, proc);
+  }
+
+  mrb_value call(int argc = 0, mrb_value *argv = nullptr) const {
+    if (mrb_nil_p(proc))
+      return mrb_nil_value();
+    mrb_value result = mrb_funcall_argv(mrb, proc, mrb_intern_cstr(mrb, "call"), argc, argv);
+    if (!mrb->exc)
+      return result;
+    mrb_value exc = mrb_obj_value(mrb->exc);
+    mrb_value msg = mrb_funcall(mrb, exc, "message", 0);
+    std::string error;
+    if (mrb_string_p(msg))
+      error.assign(RSTRING_PTR(msg), RSTRING_LEN(msg));
+    auto *fatal_class = mrb_class_get(mrb, "FatalError");
+    if (mrb_obj_is_kind_of(mrb, exc, fatal_class)) {
+      mrb_value code =
+        mrb_iv_get(mrb, exc, mrb_intern_lit(mrb, "@code"));
+      mrb->exc = nullptr;
+      int c = 1;
+      if (mrb_fixnum_p(code))
+        c = mrb_fixnum(code);
+      throw fatal_error(error, c);
+    }
+    auto *ed_class = mrb_class_get(mrb, "EdError");
+    if (mrb_obj_is_kind_of(mrb, exc, ed_class)) {
+      mrb->exc = nullptr;
+      throw ed_error(error);
+    }
+    mrb->exc = nullptr;
+    throw ed_error("Ruby Exception: " + error);
+  }
+};
+
+void register_basic(BEd &ctx);
+std::string run(BEd &ctx, const std::string &str);
+}; // namespace bed::internal::scripting

include/internal/syntax/decl.h

@@ -1,74 +1,11 @@
 #pragma once
 
+#include "internal/io/tokens.h"
 #include "internal/trie/trie.h"
 #include "internal/vase/vase.h"
 #include "pch.h"
 
 namespace bed::internal::syntax {
-struct Token {
-  uint32_t start;
-  uint32_t end;
-  enum Kind : uint8_t {
-    Data,
-    Shebang,
-    Comment,
-    Error,
-    String,
-    Escape,
-    Interpolation,
-    Regexp,
-    Number,
-    True,
-    False,
-    Char,
-    Keyword,
-    KeywordOperator,
-    Operator,
-    Function,
-    Namespace,
-    Class,
-    Module,
-    Type,
-    Constant,
-    VariableInstance,
-    VariableGlobal,
-    Annotation,
-    Directive,
-    Label,
-    Brace1,
-    Brace2,
-    Brace3,
-    Brace4,
-    Brace5,
-    Heading1,
-    Heading2,
-    Heading3,
-    Heading4,
-    Heading5,
-    Heading6,
-    Blockquote,
-    List,
-    ListItem,
-    Code,
-    LanguageName,
-    LinkLabel,
-    ImageLabel,
-    Link,
-    Table,
-    TableHeader,
-    Italic,
-    Bold,
-    Underline,
-    Strikethrough,
-    HorizontalRule,
-    Tag,
-    Attribute,
-    CheckDone,
-    CheckNotDone,
-    Count
-  } type;
-};
-
 struct ParseEvent {
   uint8_t closing;
   ParseEvent(uint8_t t) : closing(t) {}
@@ -78,7 +15,7 @@ struct Language {
   std::function<void *()> none_state;
   std::function<void(
     void **, std::string_view,
-    bool, std::vector<Token> *, std::vector<ParseEvent> *
+    bool, std::vector<io::Token> *, std::vector<ParseEvent> *
   )>
     parse;
   std::function<void *(void *)> copy;
@@ -111,6 +48,7 @@ struct ParseState {
     uint64_t line, uint64_t original, uint64_t final
   );
   static ParseState *concat(Language &lang, ParseState *a, ParseState *b);
+  static void *state_before(Language &lang, ParseState *root, vase::Shard *vase, uint64_t line);
 };
 
 struct ParseStateBranch : ParseState {
@@ -146,18 +84,25 @@ struct ParsePieceBuilder {
   std::vector<ParseState *> pieces;
   std::vector<uint16_t> blocks;
   void *piece_state{nullptr};
+  void *prev_state{nullptr};
   uint64_t chunk_start{0};
   uint64_t chunk_lines{0};
-  ParsePieceBuilder(Language &lang, uint64_t first_line)
+  ParsePieceBuilder(Language &lang, uint64_t first_line, void *entry_state)
-      : lang(lang), chunk_start(first_line) {}
+      : lang(lang), chunk_start(first_line) {
-  void add(
+    prev_state = lang.copy(entry_state);
-    void *state,
+  }
-    uint64_t line,
+  ~ParsePieceBuilder() {
-    const std::vector<ParseEvent> &events
+    if (prev_state)
-  ) {
+      lang.destroy(prev_state);
+    if (piece_state)
+      lang.destroy(piece_state);
+  }
+  ParsePieceBuilder(const ParsePieceBuilder &) = delete;
+  ParsePieceBuilder &operator=(const ParsePieceBuilder &) = delete;
+  void add(void *state, uint64_t line, const std::vector<ParseEvent> &events) {
     if (chunk_lines == 0) {
       chunk_start = line;
-      piece_state = lang.copy(state);
+      piece_state = lang.copy(prev_state);
     }
     for (const auto &ev : events) {
       blocks.push_back(
@@ -166,6 +111,9 @@ struct ParsePieceBuilder {
       );
     }
     ++chunk_lines;
+    if (prev_state)
+      lang.destroy(prev_state);
+    prev_state = lang.copy(state);
     if (chunk_lines == ParseStateLeaf::MAX_CHUNK)
       flush();
   }

include/internal/syntax/miniparser.h

@@ -0,0 +1,21 @@
+#pragma once
+
+#include "decl.h"
+
+namespace bed::internal::syntax {
+struct MiniParser {
+  Language ⟨
+  void *start_state;
+  std::vector<std::pair<void *, std::vector<io::Token>>> lines;
+
+  explicit MiniParser(Language &lang, vase::Shard *vase, void *initial_state);
+  ~MiniParser();
+
+  MiniParser(const MiniParser &) = delete;
+  MiniParser &operator=(const MiniParser &) = delete;
+
+  void dirty(vase::Shard *vase, uint64_t start, uint64_t count);
+  void insert(vase::Shard *vase, uint64_t start, uint64_t count);
+  void erase(vase::Shard *vase, uint64_t start, uint64_t count);
+};
+} // namespace bed::internal::syntax

include/internal/syntax/parser.h

@@ -13,6 +13,7 @@ ParserSnapshot make_parser(vase::Shard *vase, uint64_t lines, Language *lang);
 ParserSnapshot retain(const ParserSnapshot &snap);
 void release(ParserSnapshot &snap);
 
+void *state_before(const ParserSnapshot &snap, vase::Shard *vase, uint64_t line);
 uint64_t next_closing(const ParserSnapshot &snap, uint64_t line);
 uint64_t prev_opening(const ParserSnapshot &snap, uint64_t line);
 
@@ -21,7 +22,7 @@ struct Iterator {
   std::optional<vase::Iterator> it;
   void *state;
   uint64_t at;
-  std::vector<Token> tokens;
+  std::vector<io::Token> tokens;
   std::vector<ParseEvent> events;
   Iterator(uint64_t, ParserSnapshot, vase::Shard *);
   ~Iterator();

include/internal/syntax/ruby/decl.h

@@ -24,7 +24,8 @@ struct alignas(2) RubyState {
       DEF_NAME = 0b10,
       MODULE_NAME = 0b11
     };
-    static constexpr uint8_t NEWLINE = 1 << 5;
+    static constexpr uint8_t NEWLINE = 1 << 4;
+    static constexpr uint8_t ALLOW_ESCAPE = 1 << 5;
     static constexpr uint8_t ALLOW_INTERPOLATION = 1 << 6;
     static constexpr uint8_t EXPECTING_EXPRESSION = 1 << 7;
     uint8_t flags = 0;

include/internal/syntax/ruby/parser.h

@@ -104,6 +104,6 @@ struct RubyParser {
 
 void ruby_parse(
   void **v_state, std::string_view line, bool first_line,
-  std::vector<Token> *tokens, std::vector<ParseEvent> *events
+  std::vector<io::Token> *tokens, std::vector<ParseEvent> *events
 );
 } // namespace bed::internal::syntax::ruby

include/internal/syntax/ruby/tries.h

@@ -68,6 +68,11 @@ const static std::vector<std::string> builtins = {
 };
 
 const static std::vector<std::string> methods = {
+  // BEd methods.
+  "handle",
+  "register",
+  "unregister",
+  // Normal:
   "abort",
   "at_exit",
   "binding",

include/internal/theme/theme.h

@@ -6,11 +6,11 @@
 
 namespace bed::internal::theme {
 struct Theme {
-  std::array<Highlight, internal::syntax::Token::Count> hl;
+  std::array<io::Highlight, io::Token::Count> hl;
 
   Theme();
 
-  Highlight get(internal::syntax::Token token) const;
+  io::Highlight get(const io::Token::Kind &token) const;
 
   static Theme default_theme();
   static Theme from_name(std::string_view name);

include/internal/ui/command.h

@@ -4,31 +4,6 @@
 #include "pch.h"
 
 namespace bed::internal::ui {
-struct Token {
-  enum struct Type : uint8_t {
-    TempCurrent,      // @
-    AddressSeperator, // ; ,
-    Address,          // . $ % [ ] ^ ~
-    Offset,           // +N -N + -
-    AddressRegex,     // /re/ ?re?
-    AddressSymbol,    // >s> <s< <s>
-    Number,           // 10
-    Mark,             // 'm
-    RubyFunction,     // (func:arg)
-    RubyArg,
-    Function,
-    Any,
-    Shell,
-    Ruby,
-    File,
-    Regex,
-    Replacement,
-    Suffix
-  } type;
-  uint16_t start;
-  uint16_t end;
-};
-
 struct CommandIO {
   std::string cmd;
   uint16_t cursor;
@@ -41,6 +16,8 @@ struct CommandIO {
 
   CommandIO(BEd &);
   std::pair<std::string, bool> run();
+  std::pair<std::string, bool> run_pipe();
+  std::pair<std::string, bool> run_terminal();
   void redraw();
 };
 } // namespace bed::internal::ui

include/internal/ui/text_mode.h

@@ -1,21 +1,31 @@
 #pragma once
 
-#include "definitions.h"
+#include "internal/syntax/miniparser.h"
 #include "pch.h"
 
-namespace bed::internal::ui {
+namespace bed::internal::ui::text_mode {
 struct TextMode {
-  std::string cmd;
+  struct Row {
-  uint16_t cursor;
+    uint64_t start;
+    uint64_t length;
+  };
+  using Lines = std::vector<Row>;
+  vase::Shard *vase;
+  vase::Point cursor;
+  std::optional<syntax::MiniParser> parser;
+  std::vector<Lines> lines;
   uint16_t start;
   uint16_t height;
   uint16_t term_height;
   uint16_t term_width;
   BEd &bed;
 
-  TextMode(BEd &);
+  TextMode(BEd &, vase::Shard *, syntax::Language *, void *);
   std::pair<vase::Shard *, bool> run();
+  std::pair<vase::Shard *, bool> run_pipe();
+  std::pair<vase::Shard *, bool> run_terminal();
   void grow();
   void redraw();
+  void layout_lines();
 };
-} // namespace bed::internal::ui
+} // namespace bed::internal::ui::text_mode

include/internal/vase/shard.h

@@ -24,9 +24,9 @@ struct Shard {
   static void retain(Shard *n);
   static void release(Shard *n);
 
-  static Shard *from_file(const std::filesystem::path &path, bool posix_ending);
+  static Shard *from_file(const std::filesystem::path &path);
-  static Shard *from_string(const char *data, uint64_t len, bool posix_ending);
+  static Shard *from_string(const char *data, uint64_t len);
-  static Shard *from_command(const char *cmd, bool posix_ending);
+  static Shard *from_command(const char *cmd);
 
   static std::pair<Shard *, Shard *> split(Shard *n, uint64_t offset);
   static Shard *concat(Shard *a, Shard *b);

include/internal/vase/vase.h

@@ -41,7 +41,7 @@ struct ReplacePart {
 
 uint64_t offset_of(Shard *root, uint64_t line);
 uint64_t offset_of(Shard *root, Point point);
-Point point_of(Shard *root, uint64_t offset);
+Point eof_point(Shard *root);
 
 std::string to_string(Shard *root);
 std::string to_string(Shard *root, Range range);

include/pch.h

@@ -4,10 +4,29 @@
 
 #include <mruby.h>
 #include <mruby/array.h>
+#include <mruby/boxing_word.h>
+#include <mruby/class.h>
+#include <mruby/common.h>
 #include <mruby/compile.h>
+#include <mruby/data.h>
+#include <mruby/dump.h>
+#include <mruby/error.h>
+#include <mruby/gc.h>
 #include <mruby/hash.h>
+#include <mruby/internal.h>
 #include <mruby/irep.h>
+#include <mruby/numeric.h>
+#include <mruby/object.h>
+#include <mruby/opcode.h>
+#include <mruby/presym.h>
+#include <mruby/proc.h>
+#include <mruby/range.h>
+#include <mruby/re.h>
 #include <mruby/string.h>
+#include <mruby/throw.h>
+#include <mruby/value.h>
+#include <mruby/variable.h>
+#include <mruby/version.h>
 #include <pcre2.h>
 extern "C" {
 #include <grapheme.h>

samples/bed.rb

@@ -0,0 +1,3 @@
+register :happy, address: :none do
+  puts "be nice"
+end

src/bed/bed.cc

@@ -0,0 +1,73 @@
+#include "bed.h"
+
+namespace bed {
+BEd::BEd(std::vector<std::string> args)
+    : mrb(this), theme(internal::theme::Theme::default_theme()), io(*this) {
+  std::string prompt_ = "";
+  std::string file = "";
+  bool suppress = false;
+  bool color = true;
+  for (size_t i = 1; i < args.size(); i++) {
+    if (args[i] == "-p") {
+      i++;
+      if (i >= args.size())
+        throw fatal_error("Prompt not specified!", 1);
+      prompt_ = args[i];
+    } else if (args[i] == "-s") {
+      suppress = true;
+    } else if (args[i] == "--no-color") {
+      color = false;
+    } else if (args[i] == "-v" || args[i] == "--verbose") {
+      help_mode = true;
+    } else if (args[i] == "-h" || args[i] == "--help") {
+      print_help();
+      throw fatal_error("", 0);
+    } else {
+      if (file.size())
+        throw fatal_error("Invalid arguments given.", 1);
+      file = args[i];
+    }
+  }
+  if (prompt_ != "")
+    prompt = [p = std::move(prompt_)](BEd &) { return p; };
+  else
+    prompt_mode = false;
+  suppress_mode = suppress;
+  const char *no_color = getenv("NO_COLOR");
+  if (no_color && *no_color != '\0')
+    color = false;
+  const char *colorterm = getenv("COLORTERM");
+  if (colorterm
+      && strcmp(colorterm, "truecolor") != 0
+      && strcmp(colorterm, "24bit") != 0)
+    color = false;
+  color_mode = color;
+  internal::functions::Function::register_posix(*this);
+  internal::functions::Function::register_extented(*this);
+  internal::functions::Suffix::register_suffixes(*this);
+  internal::scripting::register_basic(*this);
+  languages["ruby"] = new internal::syntax::Language(internal::syntax::ruby::lang_ruby());
+  buffers["clip"] = new internal::buffer::ClipBuffer("clip");
+  current() = {"default", 0};
+  try {
+    if (file != "")
+      handle(":default:E " + file, false);
+  } catch (ed_error &e) {
+    io.write_line("?");
+    if (help_mode)
+      io.write_line(e.what());
+    last_help = e.what();
+  }
+}
+
+BEd::~BEd() {
+  for (auto &[_, buffer] : buffers)
+    delete buffer;
+  for (auto &[_, lang] : languages)
+    delete lang;
+}
+
+void BEd::print_help() {
+  io.write("BEd - A line editor.\n");
+}
+} // namespace bed

src/bed/handle.cc

@@ -0,0 +1,115 @@
+#include "bed.h"
+#include "internal/parser/parser.h"
+
+namespace bed {
+void BEd::handle(std::string_view cmd, bool eof) {
+  if (eof && cmd.empty()) {
+    eof_op.handle(*this, "", nullptr, std::monostate(), nullptr);
+    return;
+  }
+  internal::parser::Command c = internal::parser::Parser::get_command(cmd, *this);
+  if (c.temp_address) {
+    marks.get(251) = marks.get(250);
+    prev_2 = prev_1;
+    temporary_current = true;
+  }
+  internal::buffer::Address address;
+  switch (c.function->address_kind) {
+  case internal::functions::Function::AddressKind::None: {
+    auto a = internal::parser::AddressPromise::get_line(*this, c.addresses);
+    if (!a.has_value()) {
+      auto vec = internal::parser::Parser::get_addresses(c.function->default_address, *this);
+      a = internal::parser::AddressPromise::get_line(*this, vec);
+      if (!a.has_value())
+        a = current();
+    }
+    address = a->buffername;
+  } break;
+  case internal::functions::Function::AddressKind::Line: {
+    auto a = internal::parser::AddressPromise::get_line(*this, c.addresses);
+    if (!a.has_value()) {
+      auto vec = internal::parser::Parser::get_addresses(c.function->default_address, *this);
+      a = internal::parser::AddressPromise::get_line(*this, vec);
+      if (!a.has_value())
+        a = current();
+    }
+    address = *a;
+  } break;
+  case internal::functions::Function::AddressKind::Range: {
+    auto a = internal::parser::AddressPromise::get_range(*this, c.addresses);
+    if (!a.has_value()) {
+      auto vec = internal::parser::Parser::get_addresses(c.function->default_address, *this);
+      a = internal::parser::AddressPromise::get_range(*this, vec);
+      if (!a.has_value())
+        a = internal::buffer::Range(current(), current());
+    }
+    address = *a;
+  } break;
+  }
+  if (std::holds_alternative<internal::buffer::Line>(c.argument)) {
+    if (c.argument_addresses.empty())
+      throw ed_error("Function needs address argument.");
+    auto a = internal::parser::AddressPromise::get_line(*this, c.argument_addresses);
+    if (a.has_value())
+      c.argument = *a;
+    else
+      c.argument = current();
+  } else if (std::holds_alternative<internal::buffer::Range>(c.argument)) {
+    if (c.argument_addresses.empty())
+      throw ed_error("Function needs address argument.");
+    auto a = internal::parser::AddressPromise::get_range(*this, c.argument_addresses);
+    if (a.has_value())
+      c.argument = *a;
+    else
+      c.argument = internal::buffer::Range(current(), current());
+  }
+  if (!c.function->accept_zero) {
+    if (std::holds_alternative<internal::buffer::Line>(address)) {
+      if (std::get<internal::buffer::Line>(address).number == 0)
+        throw ed_error("Line number can't be zero.");
+    } else if (std::holds_alternative<internal::buffer::Range>(address)) {
+      auto r = std::get<internal::buffer::Range>(address);
+      if (r.start == 0 || r.end == 0)
+        throw ed_error("Line number can't be zero.");
+    }
+  }
+  internal::vase::Shard *text = nullptr;
+  if (c.function->input_mode == internal::functions::Function::InputMode::Text) {
+    internal::vase::Shard *vase = nullptr;
+    internal::syntax::Language *lang = nullptr;
+    void *state = nullptr;
+    if (c.function->pre_text_mode)
+      std::tie(vase, lang, state) = c.function->pre_text_mode(*this, address, c.argument);
+    internal::ui::text_mode::TextMode tm(*this, vase, lang, state);
+    auto [a, b] = tm.run();
+    if (!b) {
+      text = a;
+    } else {
+      if (a) {
+        auto p = internal::syntax::make_parser(a, a->lines + 1, lang);
+        auto cancel_buf = new internal::buffer::ReadonlyBuffer("cancel", a, p);
+        internal::syntax::release(p);
+        buffers["cancel"] = cancel_buf;
+        cancel_buf->useless = false;
+        internal::vase::Shard::release(a);
+      }
+      throw ed_error("Operation cancelled.");
+    }
+  }
+  if (c.function->handle)
+    c.function->handle(*this, address, text, c.argument, nullptr);
+  if (c.suffix)
+    c.suffix->handle(*this);
+  if (c.temp_address)
+    temporary_current = false;
+  for (auto it = buffers.begin(); it != buffers.end();) {
+    internal::buffer::Buffer *buf = it->second;
+    if (buf->waste()) {
+      delete buf;
+      it = buffers.erase(it);
+    } else {
+      ++it;
+    }
+  }
+}
+} // namespace bed

src/bed/run.cc

@@ -2,179 +2,38 @@
 #include "internal/parser/parser.h"
 
 namespace bed {
-BEd::BEd(std::vector<std::string> args, internal::io::IO &io)
-    : theme(internal::theme::Theme::default_theme()), io(io) {
-  internal::functions::Function::register_posix(*this);
-  internal::functions::Function::register_extented(*this);
-  internal::functions::Suffix::register_suffixes(*this);
-  languages["ruby"] = new internal::syntax::Language(internal::syntax::ruby::lang_ruby());
-  std::string prompt_ = "";
-  std::string file = "";
-  bool suppress = false;
-  for (size_t i = 1; i < args.size(); i++) {
-    if (args[i] == "-p") {
-      i++;
-      if (i >= args.size())
-        throw fatal_error("Prompt not specified!", 1);
-      prompt_ = args[i];
-    } else if (args[i] == "-s") {
-      suppress = true;
-    } else if (args[i] == "-v" || args[i] == "--verbose") {
-      help_mode = true;
-    } else {
-      if (file.size())
-        throw fatal_error("Invalid arguments given.", 1);
-      file = args[i];
-    }
-  }
-  if (prompt_ != "")
-    prompt = [p = std::move(prompt_)](BEd &) { return p; };
-  else
-    prompt_mode = false;
-  suppress_mode = suppress;
-  buffers["clip"] = new internal::buffer::ClipBuffer("clip");
-  current() = {"default", 0};
-  try {
-    if (file != "")
-      handle(":default:E " + file, false);
-  } catch (ed_error &e) {
-    io.write_line("?");
-    if (help_mode)
-      io.write_line(e.what());
-    last_help = e.what();
-  }
-}
-
-BEd::~BEd() {
-  for (auto &[_, buffer] : buffers)
-    delete buffer;
-  for (auto &[_, lang] : languages)
-    delete lang;
-}
-
 void BEd::run() {
   while (true) {
     internal::ui::CommandIO command(*this);
     auto [cmd, eof] = command.run();
     try {
       handle(cmd, eof);
-    } catch (ed_error &e) {
+    } catch (const ed_error &e) {
+      io.apply(internal::io::Token::Warning);
       io.write_line("?");
       if (help_mode)
         io.write_line(e.what());
+      io.reset();
       last_help = e.what();
     }
   }
 }
 
-void BEd::handle(std::string_view cmd, bool eof) {
-  if (eof) {
-    eof_op.handle(*this, "", nullptr, std::monostate(), nullptr);
-    return;
-  }
-  internal::parser::Command c = internal::parser::Parser::get_command(cmd, *this);
-  if (c.temp_address) {
-    marks.get(251) = marks.get(250);
-    prev_2 = prev_1;
-    temporary_current = true;
-  }
-  internal::buffer::Address address;
-  switch (c.function->address_kind) {
-  case internal::functions::Function::AddressKind::None: {
-    auto a = internal::parser::AddressPromise::get_line(*this, c.addresses);
-    if (!a.has_value()) {
-      auto vec = internal::parser::Parser::get_addresses(c.function->default_address, *this);
-      a = internal::parser::AddressPromise::get_line(*this, vec);
-      if (!a.has_value())
-        a = current();
-    }
-    address = a->buffername;
-  } break;
-  case internal::functions::Function::AddressKind::Line: {
-    auto a = internal::parser::AddressPromise::get_line(*this, c.addresses);
-    if (!a.has_value()) {
-      auto vec = internal::parser::Parser::get_addresses(c.function->default_address, *this);
-      a = internal::parser::AddressPromise::get_line(*this, vec);
-      if (!a.has_value())
-        a = current();
-    }
-    address = *a;
-  } break;
-  case internal::functions::Function::AddressKind::Range: {
-    auto a = internal::parser::AddressPromise::get_range(*this, c.addresses);
-    if (!a.has_value()) {
-      auto vec = internal::parser::Parser::get_addresses(c.function->default_address, *this);
-      a = internal::parser::AddressPromise::get_range(*this, vec);
-      if (!a.has_value())
-        a = internal::buffer::Range(current(), current());
-    }
-    address = *a;
-  } break;
-  }
-  if (std::holds_alternative<internal::buffer::Line>(c.argument)) {
-    if (c.argument_addresses.empty())
-      throw ed_error("Function needs address argument.");
-    auto a = internal::parser::AddressPromise::get_line(*this, c.argument_addresses);
-    if (a.has_value())
-      c.argument = *a;
-    else
-      c.argument = current();
-  } else if (std::holds_alternative<internal::buffer::Range>(c.argument)) {
-    if (c.argument_addresses.empty())
-      throw ed_error("Function needs address argument.");
-    auto a = internal::parser::AddressPromise::get_range(*this, c.argument_addresses);
-    if (a.has_value())
-      c.argument = *a;
-    else
-      c.argument = internal::buffer::Range(current(), current());
-  }
-  if (!c.function->accept_zero) {
-    if (std::holds_alternative<internal::buffer::Line>(address)) {
-      if (std::get<internal::buffer::Line>(address).number == 0)
-        throw ed_error("Line number can't be zero.");
-    } else if (std::holds_alternative<internal::buffer::Range>(address)) {
-      auto r = std::get<internal::buffer::Range>(address);
-      if (r.start == 0 || r.end == 0)
-        throw ed_error("Line number can't be zero.");
-    }
-  }
-  internal::vase::Shard *text = nullptr;
-  if (c.function->input_mode == internal::functions::Function::InputMode::Text) {
-    internal::ui::TextMode tm(*this);
-    auto [a, b] = tm.run();
-    if (!b)
-      text = a;
-  }
-  if (c.function->handle)
-    c.function->handle(*this, address, text, c.argument, nullptr);
-  if (c.suffix)
-    c.suffix->handle(*this);
-  if (c.temp_address)
-    temporary_current = false;
-  for (auto it = buffers.begin(); it != buffers.end();) {
-    internal::buffer::Buffer *buf = it->second;
-    if (buf->waste()) {
-      delete buf;
-      it = buffers.erase(it);
-    } else {
-      ++it;
-    }
-  }
-}
-
 internal::buffer::Buffer &BEd::buffer(const std::string &name) {
   if (name.empty())
-    throw ed_error("can't have empty buffer name");
+    throw ed_error("Can't have empty buffer name");
   {
     auto it = buffers.find(name);
     if (it != buffers.end())
       return *it->second;
   }
   constexpr std::string_view prefix = "history/";
+  if (name == "cancel")
+    throw ed_error("Cancel buffer empty.");
   if (name.starts_with(prefix)) {
     std::string_view path{name};
     path.remove_prefix(prefix.size());
-    auto slash = path.rfind('/');
+    auto slash = path.rfind('_');
     if (slash == std::string_view::npos || slash == 0 || slash == path.size() - 1)
       throw ed_error("invalid history");
     std::string bufname(path.substr(0, slash));
@@ -197,6 +56,15 @@ internal::buffer::Buffer &BEd::buffer(const std::string &name) {
     }
     throw ed_error("Buffer has no history.");
   }
+  for (auto &c : name)
+    if (!(('0' <= c && c <= '9')
+          || ('a' <= c && c <= 'z')
+          || ('A' <= c && c <= 'Z')
+          || c == '-' || c == '_'
+          || c == '+' || c == '.'
+          || c == ',' || c == '$'
+          || c == '/' || c == '~'))
+      throw ed_error("Invalid buffer name.");
   auto *buf = new internal::buffer::GenericBuffer(name);
   buffers.emplace(name, buf);
   return *buf;

src/internal/buffer/generic.cc

@@ -15,6 +15,8 @@ GenericBuffer::~GenericBuffer() {
 
 void GenericBuffer::list_history(BEd &ctx) {
   uint64_t current = base_version + undo_stack.size();
+  uint64_t max_version = current + redo_stack.size();
+  size_t width = std::to_string(max_version).size();
   for (size_t i = 0; i < undo_stack.size(); ++i) {
     auto &item = undo_stack[i];
     uint64_t version = base_version + i;
@@ -22,8 +24,9 @@ void GenericBuffer::list_history(BEd &ctx) {
     std::tm tm = *std::localtime(&time);
     ctx.io.write_line(
       std::format(
-        "  {}  {:04}-{:02}-{:02} {:02}:{:02}:{:02} {}",
+        "  {:>{}}  {:04}-{:02}-{:02} {:02}:{:02}:{:02} {}",
         version,
+        width,
         tm.tm_year + 1900,
         tm.tm_mon + 1,
         tm.tm_mday,
@@ -39,8 +42,9 @@ void GenericBuffer::list_history(BEd &ctx) {
     std::tm tm = *std::localtime(&time);
     ctx.io.write_line(
       std::format(
-        "X {}  {:04}-{:02}-{:02} {:02}:{:02}:{:02} {}",
+        "* {:>{}}  {:04}-{:02}-{:02} {:02}:{:02}:{:02} {}",
         current,
+        width,
         tm.tm_year + 1900,
         tm.tm_mon + 1,
         tm.tm_mday,
@@ -58,8 +62,9 @@ void GenericBuffer::list_history(BEd &ctx) {
     std::tm tm = *std::localtime(&time);
     ctx.io.write_line(
       std::format(
-        "  {}  {:04}-{:02}-{:02} {:02}:{:02}:{:02}",
+        "  {:>{}}  {:04}-{:02}-{:02} {:02}:{:02}:{:02} {}",
         version,
+        width,
         tm.tm_year + 1900,
         tm.tm_mon + 1,
         tm.tm_mday,
@@ -72,33 +77,37 @@ void GenericBuffer::list_history(BEd &ctx) {
   }
 }
 
-HistoryBuffer *GenericBuffer::get_history(uint64_t version) {
+ReadonlyBuffer *GenericBuffer::get_history(uint64_t version) {
   uint64_t current = base_version + undo_stack.size();
   if (version < base_version)
     throw ed_error("History version has been pruned.");
   if (version < current) {
     auto &item = undo_stack[version - base_version];
-    return new HistoryBuffer(name, item.text, item.parse_state);
+    return new ReadonlyBuffer(name, item.text, item.parse_state);
   }
   if (version == current)
-    return new HistoryBuffer(name, root, parse);
+    return new ReadonlyBuffer(name, root, parse);
   uint64_t redo_offset = version - current - 1;
   if (redo_offset >= redo_stack.size())
     throw ed_error("No such history version.");
   auto &item = redo_stack[redo_stack.size() - 1 - redo_offset];
-  return new HistoryBuffer(name, item.text, item.parse_state);
+  return new ReadonlyBuffer(name, item.text, item.parse_state);
 }
 
 void GenericBuffer::snapshot(std::string action_) {
-  vase::Shard::retain(root);
+  if (base_version == 0) {
-  undo_stack.push_back(
+    base_version++;
-    HistoryItem{
+  } else {
-      syntax::retain(parse),
+    vase::Shard::retain(root);
-      root,
+    undo_stack.push_back(
-      timestamp,
+      HistoryItem{
-      action
+        syntax::retain(parse),
-    }
+        root,
-  );
+        timestamp,
+        action
+      }
+    );
+  }
   for (auto &item : redo_stack) {
     syntax::release(item.parse_state);
     vase::Shard::release(item.text);
@@ -126,6 +135,8 @@ bool GenericBuffer::undo(BEd &ctx) {
   root = prev.text;
   syntax::release(parse);
   parse = prev.parse_state;
+  timestamp = prev.timestamp;
+  action = prev.summary;
   state = Modified;
   if (!root) {
     ctx.prev().buffername = name;
@@ -157,6 +168,8 @@ bool GenericBuffer::redo(BEd &ctx) {
   root = next.text;
   syntax::release(parse);
   parse = next.parse_state;
+  timestamp = next.timestamp;
+  action = next.summary;
   state = Modified;
   if (!root) {
     ctx.prev().buffername = name;
@@ -170,7 +183,7 @@ bool GenericBuffer::redo(BEd &ctx) {
   return true;
 }
 
-void GenericBuffer::prune(int keep) {
+uint64_t GenericBuffer::prune(int keep) {
   size_t drop =
     undo_stack.size() > (size_t)keep
       ? undo_stack.size() - keep
@@ -189,11 +202,33 @@ void GenericBuffer::prune(int keep) {
     vase::Shard::release(item.text);
   }
   redo_stack.clear();
+  return undo_stack.size();
 }
 
 bool GenericBuffer::waste() {
   return save_path.empty()
-         && root == nullptr;
+         && root == nullptr
+         && undo_stack.empty()
+         && parse.lang == nullptr;
+}
+
+void GenericBuffer::saved_hook() {
+  state = buffer::GenericBuffer::Unmodified;
+}
+
+void GenericBuffer::language(BEd &ctx, std::string name) {
+  syntax::Language *lang = nullptr;
+  if (name.size()) {
+    auto it = ctx.languages.find(name);
+    if (it == ctx.languages.end())
+      throw ed_error("Language not found.");
+    lang = it->second;
+  }
+  if (lang == parse.lang)
+    return;
+  snapshot("Change buffer language.");
+  syntax::release(parse);
+  parse = syntax::make_parser(root, lines(), lang);
 }
 
 void GenericBuffer::load(BEd &ctx, vase::Shard *text) {
@@ -213,8 +248,9 @@ void GenericBuffer::load(BEd &ctx, vase::Shard *text) {
     ctx.prev().start = 1;
     ctx.prev().end = text->lines + 1;
   }
+  ctx.current() = {name, lines()};
   syntax::release(parse);
-  parse = syntax::make_parser(root, lines(), ctx.languages["ruby"]);
+  parse = syntax::make_parser(root, lines(), nullptr);
 }
 
 void GenericBuffer::set_filename(std::filesystem::path path) {
@@ -226,16 +262,15 @@ std::filesystem::path GenericBuffer::filename() {
 };
 
 void GenericBuffer::append(BEd &ctx, vase::Shard *text, uint64_t line) {
-  if (!text)
+  snapshot(std::format("Insert {} lines after line {}", (text ? text->lines + 1 : 1), line));
-    return;
-  snapshot(std::format("Insert {} lines after line {}", text->lines + 1, line));
   ctx.prev().buffername = name;
   ctx.prev().start = line + 1;
-  ctx.prev().end = line + text->lines + 1;
+  ctx.prev().end = line + (text ? text->lines + 1 : 1);
+  ctx.current() = {name, line + (text ? text->lines + 1 : 1)};
   root = vase::insert(&ctx.append, root, text, line);
-  ctx.marks.insert(name, line, text->lines + 1);
+  ctx.marks.insert(name, line, (text ? text->lines + 1 : 1));
   if (parse.lang)
-    syntax::insert(parse, root, line, text->lines + 1);
+    syntax::insert(parse, root, line, (text ? text->lines + 1 : 1));
   state = Modified;
 }
 
@@ -243,8 +278,10 @@ void GenericBuffer::remove(BEd &ctx, uint64_t start_line, uint64_t end_line) {
   snapshot(std::format("Remove lines {} to {}", start_line, end_line));
   root = vase::erase(root, start_line, end_line);
   ctx.prev().buffername = name;
-  ctx.prev().start = std::min(start_line, lines());
+  uint64_t l = std::min(start_line, lines());
-  ctx.prev().end = std::min(start_line, lines());
+  ctx.prev().start = l;
+  ctx.prev().end = l;
+  ctx.current() = {name, l};
   ctx.marks.erase(name, start_line, end_line - start_line + 1);
   if (parse.lang)
     syntax::erase(parse, root, start_line, end_line - start_line + 1);
@@ -260,6 +297,7 @@ void GenericBuffer::replace(BEd &ctx, vase::Shard *text, uint64_t start_line, ui
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
   ctx.prev().end = start_line + text->lines;
+  ctx.current() = {name, start_line + text->lines};
   uint64_t new_count = text->lines + 1;
   uint64_t old_count = end_line - start_line + 1;
   root = vase::replace(root, text, start_line, end_line);
@@ -285,6 +323,7 @@ void GenericBuffer::join(BEd &ctx, uint64_t start_line, uint64_t end_line) {
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
   ctx.prev().end = start_line;
+  ctx.current() = {name, start_line};
   ctx.marks.collapse(name, start_line, end_line - start_line);
   if (parse.lang)
     syntax::erase(parse, root, start_line, end_line - start_line);
@@ -321,11 +360,11 @@ void GenericBuffer::substitute(
         if (edit)
           edit->insert(line, new_lines);
       }
+      ctx.current() = {name, line + new_lines};
     }
   );
   if (edit)
     edit->commit(root);
-  ctx.prev().buffername = name;
   state = Modified;
 }
 } // namespace bed::internal::buffer

src/internal/buffer/shard.cc

@@ -29,8 +29,6 @@ uint64_t ShardBuffer::find_prev(std::string_view pattern, uint64_t start) {
 uint64_t ShardBuffer::next_closing(uint64_t start) {
   if (parse.lang) {
     uint64_t closing = syntax::next_closing(parse, start - 1);
-    if (closing == UINT64_MAX)
-      return lines();
     return closing + 1;
   } else {
     start += 10;
@@ -50,32 +48,6 @@ uint64_t ShardBuffer::prev_closing(uint64_t start) {
   }
 }
 
-inline void apply(io::IO &io, const Highlight &hl) {
-  io.write("\x1b[0m");
-  const uint8_t r = (hl.fg >> 16) & 0xff;
-  const uint8_t g = (hl.fg >> 8) & 0xff;
-  const uint8_t b = hl.fg & 0xff;
-  io.write(std::format("\x1b[38;2;{};{};{}m", r, g, b));
-  if (hl.bg != 0) {
-    const uint8_t br = (hl.bg >> 16) & 0xff;
-    const uint8_t bg = (hl.bg >> 8) & 0xff;
-    const uint8_t bb = hl.bg & 0xff;
-    io.write(std::format("\x1b[48;2;{};{};{}m", br, bg, bb));
-  }
-  if (hl.flags & Highlight::Bold)
-    io.write("\x1b[1m");
-  if (hl.flags & Highlight::Italic)
-    io.write("\x1b[3m");
-  if (hl.flags & Highlight::Underline)
-    io.write("\x1b[4m");
-  if (hl.flags & Highlight::Strikethrough)
-    io.write("\x1b[9m");
-}
-
-inline void reset(io::IO &io) {
-  io.write("\x1b[0m");
-}
-
 void ShardBuffer::print(BEd &ctx, uint64_t start_line, uint64_t end_line) {
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
@@ -97,10 +69,9 @@ void ShardBuffer::print(BEd &ctx, uint64_t start_line, uint64_t end_line) {
           break;
         if (cursor < start)
           ctx.io.write(line.data() + cursor, start - cursor);
-        const auto highlight = ctx.theme.get(token);
+        ctx.io.apply(token.type);
-        apply(ctx.io, highlight);
         ctx.io.write(line.data() + start, end - start);
-        reset(ctx.io);
+        ctx.io.reset();
         cursor = end;
       }
       if (cursor < line.size())
@@ -141,10 +112,9 @@ void ShardBuffer::number_print(BEd &ctx, uint64_t start_line, uint64_t end_line)
           break;
         if (cursor < start)
           ctx.io.write(line.data() + cursor, start - cursor);
-        const auto highlight = ctx.theme.get(token);
+        ctx.io.apply(token.type);
-        apply(ctx.io, highlight);
         ctx.io.write(line.data() + start, end - start);
-        reset(ctx.io);
+        ctx.io.reset();
         cursor = end;
       }
       if (cursor < line.size())

src/internal/buffer/special/clip.cc

@@ -8,15 +8,17 @@ bool ClipBuffer::waste() {
   return false;
 }
 
+void ClipBuffer::saved_hook() {}
+
 uint64_t ClipBuffer::lines() {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   uint64_t lines = s ? s->lines + 1 : 0;
   vase::Shard::release(s);
   return lines;
 }
 
 uint64_t ClipBuffer::bytes() {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   uint64_t length = s ? s->length + 1 : 0;
   vase::Shard::release(s);
   return length;
@@ -55,7 +57,7 @@ void ClipBuffer::append(BEd &ctx, vase::Shard *text, uint64_t line) {
   ctx.prev().buffername = name;
   ctx.prev().start = line + 1;
   ctx.prev().end = line + (text ? text->lines + 1 : 0);
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::insert(&ctx.append, s, text, line);
   clip_write(s);
   vase::Shard::release(s);
@@ -63,7 +65,7 @@ void ClipBuffer::append(BEd &ctx, vase::Shard *text, uint64_t line) {
 }
 
 void ClipBuffer::remove(BEd &ctx, uint64_t start_line, uint64_t end_line) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::erase(s, start_line, end_line);
   clip_write(s);
   ctx.prev().buffername = name;
@@ -83,7 +85,7 @@ void ClipBuffer::replace(BEd &ctx, vase::Shard *text, uint64_t start_line, uint6
   ctx.prev().end = start_line + text->lines;
   uint64_t new_count = text->lines + 1;
   uint64_t old_count = end_line - start_line + 1;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::replace(s, text, start_line, end_line);
   clip_write(s);
   vase::Shard::release(s);
@@ -94,11 +96,10 @@ void ClipBuffer::replace(BEd &ctx, vase::Shard *text, uint64_t start_line, uint6
     uint64_t diff = old_count - new_count;
     ctx.marks.collapse(name, start_line + new_count - 1, diff);
   }
-  state = Modified;
 }
 
 void ClipBuffer::join(BEd &ctx, uint64_t start_line, uint64_t end_line) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::join(s, start_line, end_line);
   clip_write(s);
   vase::Shard::release(s);
@@ -115,7 +116,7 @@ void ClipBuffer::substitute(
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
   ctx.prev().end = end_line;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::substitute(
     &ctx.append,
     s,
@@ -128,7 +129,7 @@ void ClipBuffer::substitute(
       if (old_lines)
         ctx.marks.erase(name, line, old_lines);
       if (new_lines)
-        ctx.marks.insert(name, line, new_lines - 1);
+        ctx.marks.insert(name, line, new_lines);
     }
   );
   clip_write(s);
@@ -137,21 +138,21 @@ void ClipBuffer::substitute(
 }
 
 vase::Shard *ClipBuffer::copy(uint64_t start_line, uint64_t end_line) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   vase::Shard *o = vase::copy(s, start_line, end_line);
   vase::Shard::release(s);
   return o;
 }
 
 uint64_t ClipBuffer::find_next(std::string_view pattern, uint64_t start) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   uint64_t line = vase::find_next(s, pattern, start);
   vase::Shard::release(s);
   return line;
 }
 
 uint64_t ClipBuffer::find_prev(std::string_view pattern, uint64_t start) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   uint64_t line = vase::find_prev(s, pattern, start);
   vase::Shard::release(s);
   return line;
@@ -175,7 +176,7 @@ void ClipBuffer::print(BEd &ctx, uint64_t start_line, uint64_t end_line) {
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
   ctx.prev().end = end_line;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   vase::Iterator it(s, start_line - 1, Direction::Forward);
   while (it.next() && start_line++ <= end_line)
     ctx.io.write_line(it.line);
@@ -189,7 +190,7 @@ void ClipBuffer::number_print(BEd &ctx, uint64_t start_line, uint64_t end_line)
   uint8_t width = 1;
   for (uint64_t n = end_line; n >= 10; n /= 10)
     ++width;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   vase::Iterator it(s, start_line - 1, Direction::Forward);
   while (it.next() && start_line <= end_line)
     ctx.io.write_line(std::format("{:>{}}\t{}", start_line++, width, it.line));
@@ -200,7 +201,7 @@ void ClipBuffer::list_print(BEd &ctx, uint64_t start_line, uint64_t end_line) {
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
   ctx.prev().end = end_line;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   vase::Iterator it(s, start_line - 1, Direction::Forward);
   while (it.next() && start_line++ <= end_line)
     ctx.io.write_line(list_string(it.line));

src/internal/functions/extended.cc

@@ -3,14 +3,330 @@
 #include "internal/functions/suffixes.h"
 
 namespace bed::internal::functions {
+static std::string_view address_kind_str(Function::AddressKind k) {
+  switch (k) {
+  case Function::AddressKind::None:
+    return "none";
+  case Function::AddressKind::Line:
+    return "1 line";
+  case Function::AddressKind::Range:
+    return "range";
+  }
+  return "";
+}
+
+static std::string_view input_mode_str(Function::InputMode m) {
+  switch (m) {
+  case Function::InputMode::None:
+    return "";
+  case Function::InputMode::Text:
+    return "text";
+  case Function::InputMode::Interactive:
+    return "interactive";
+  case Function::InputMode::CommandList:
+    return "command list";
+  }
+  return "";
+}
+
+static std::string_view argument_kind_str(Function::ArgumentKind a) {
+  switch (a) {
+  case Function::ArgumentKind::None:
+    return "";
+  case Function::ArgumentKind::Regex:
+    return "regex";
+  case Function::ArgumentKind::Shell:
+    return "shell command";
+  case Function::ArgumentKind::Any:
+    return "any";
+  case Function::ArgumentKind::File:
+    return "file";
+  case Function::ArgumentKind::Global:
+    return "ed global-style";
+  case Function::ArgumentKind::Mark:
+    return "mark name";
+  case Function::ArgumentKind::Number:
+    return "number";
+  case Function::ArgumentKind::Line:
+    return "address";
+  case Function::ArgumentKind::Range:
+    return "range";
+  case Function::ArgumentKind::Ruby:
+    return "ruby code";
+  }
+  return "";
+}
+
+static void append_field(BEd &ctx, std::string_view label, io::Token::Kind color, std::string_view value) {
+  if (value.empty())
+    return;
+  ctx.io.write("\n");
+  ctx.io.apply(color);
+  ctx.io.write(label);
+  ctx.io.reset();
+  ctx.io.write(": ");
+  ctx.io.write(value);
+}
+
+static void describe_function(BEd &ctx, const Function &func) {
+  ctx.io.write(func.desc);
+  ctx.io.reset();
+  append_field(ctx, "Default address", io::Token::Color1, func.default_address);
+  append_field(ctx, "Address", io::Token::Color2, address_kind_str(func.address_kind));
+  if (func.accept_zero)
+    append_field(ctx, "Zero address", io::Token::Color3, "allowed");
+  append_field(ctx, "Input", io::Token::Color4, input_mode_str(func.input_mode));
+  append_field(ctx, "Argument", io::Token::Color5, argument_kind_str(func.argument_kind));
+  ctx.io.write("\n");
+}
+
 void Function::register_extented(BEd &ctx) {
+  ctx.functions.insert(
+    "*",
+    Function{
+      .address_kind = Function::AddressKind::None,
+      .argument_kind = Function::ArgumentKind::Any,
+      .input_mode = Function::InputMode::None,
+      .desc = "Explain a command",
+      .default_address = "",
+      .accept_zero = false,
+      .pre_text_mode = nullptr,
+      .handle = [](
+                  BEd &ctx,
+                  const buffer::Address &,
+                  vase::Shard *,
+                  const Argument &arg_,
+                  std::vector<buffer::Line> *
+                ) {
+        auto str = std::get<std::string>(arg_);
+        const auto first = str.find_first_not_of(" \t");
+        const auto last = str.find_last_not_of(" \t");
+        if (first == std::string::npos)
+          str.clear();
+        else
+          str = str.substr(first, last - first + 1);
+        if (str.empty()) {
+          describe_function(ctx, ctx.no_op);
+          return;
+        }
+        auto len = ctx.functions.longest_match(str);
+        if (len == str.size()) {
+          describe_function(ctx, *ctx.functions.get_ptr(str));
+          return;
+        }
+        throw ed_error("Not a valid function name.");
+      },
+    }
+  );
+  ctx.functions.insert(
+    "b",
+    Function{
+      .address_kind = Function::AddressKind::None,
+      .argument_kind = Function::ArgumentKind::None,
+      .input_mode = Function::InputMode::None,
+      .desc = "List active buffers",
+      .default_address = "",
+      .accept_zero = false,
+      .pre_text_mode = nullptr,
+      .handle = [](
+                  BEd &ctx,
+                  const buffer::Address &addr_,
+                  vase::Shard *,
+                  const Argument &,
+                  std::vector<buffer::Line> *
+                ) {
+        auto &current = std::get<std::string>(addr_);
+        bool current_real = false;
+        for (auto &[name, _] : ctx.buffers) {
+          if (current == name) {
+            ctx.io.write("* ");
+            current_real = true;
+          } else {
+            ctx.io.write("  ");
+          }
+          ctx.io.write_line(name);
+        }
+        if (!current_real)
+          ctx.io.write_line("* " + current);
+      },
+    }
+  );
+  ctx.functions.insert(
+    "#",
+    Function{
+      .address_kind = Function::AddressKind::None,
+      .argument_kind = Function::ArgumentKind::Any,
+      .input_mode = Function::InputMode::None,
+      .desc = "Write a comment",
+      .default_address = "",
+      .accept_zero = false,
+      .pre_text_mode = nullptr,
+      .handle = [](
+                  BEd &,
+                  const buffer::Address &,
+                  vase::Shard *,
+                  const Argument &,
+                  std::vector<buffer::Line> *
+                ) {},
+    }
+  );
+  ctx.functions.insert(
+    "`",
+    Function{
+      .address_kind = Function::AddressKind::Range,
+      .argument_kind = Function::ArgumentKind::Ruby,
+      .input_mode = Function::InputMode::None,
+      .desc = "Execute given ruby code.",
+      .default_address = "0,0",
+      .accept_zero = true,
+      .pre_text_mode = nullptr,
+      .handle = [](
+                  BEd &ctx,
+                  const buffer::Address &addr_,
+                  vase::Shard *,
+                  const Argument &arg_,
+                  std::vector<buffer::Line> *
+                ) {
+        auto &addr = std::get<buffer::Range>(addr_);
+        auto &arg = std::get<RubyArg>(arg_);
+        auto pre_code = "$START=" + std::to_string(addr.start)
+                        + ";$END=" + std::to_string(addr.end)
+                        + ";$BUFNAME=\"" + addr.buffername + "\"";
+        scripting::run(ctx, pre_code);
+        auto line = scripting::run(ctx, arg.cmd);
+        ctx.io.write("=> ", 3);
+        auto parser = syntax::MiniParser(
+          *ctx.languages["ruby"],
+          vase::Shard::from_string(line.data(), line.size()),
+          nullptr
+        );
+        const auto &tokens = parser.lines[0].second;
+        uint32_t cursor = 0;
+        for (const auto &token : tokens) {
+          const uint32_t start = token.start;
+          const uint32_t end = token.end;
+          if (start > line.size() || end > line.size())
+            break;
+          if (cursor < start)
+            ctx.io.write(line.data() + cursor, start - cursor);
+          ctx.io.apply(token.type);
+          ctx.io.write(line.data() + start, end - start);
+          ctx.io.reset();
+          cursor = end;
+        }
+        if (cursor < line.size())
+          ctx.io.write(line.data() + cursor, line.size() - cursor);
+        ctx.io.write_line("");
+      },
+    }
+  );
+  ctx.functions.insert(
+    "``",
+    Function{
+      .address_kind = Function::AddressKind::Range,
+      .argument_kind = Function::ArgumentKind::None,
+      .input_mode = Function::InputMode::None,
+      .desc = "Execute addressed lines as ruby code.",
+      .default_address = "1,$",
+      .accept_zero = false,
+      .pre_text_mode = nullptr,
+      .handle = [](
+                  BEd &ctx,
+                  const buffer::Address &addr_,
+                  vase::Shard *,
+                  const Argument &,
+                  std::vector<buffer::Line> *
+                ) {
+        auto &addr = std::get<buffer::Range>(addr_);
+        auto &buf = ctx.buffer(addr.buffername);
+        auto code = buf.copy(addr.start, addr.end);
+        ctx.prev().buffername = addr.buffername;
+        ctx.prev().start = addr.start;
+        ctx.prev().end = addr.end;
+        ctx.current() = {addr.buffername, addr.end};
+        auto pre_code = "$START=" + std::to_string(addr.start)
+                        + ";$END=" + std::to_string(addr.end)
+                        + ";$BUFNAME=\"" + addr.buffername + "\"";
+        scripting::run(ctx, pre_code);
+        scripting::run(ctx, vase::to_string(code));
+        vase::Shard::release(code);
+      },
+    }
+  );
+  ctx.functions.insert(
+    "echo",
+    Function{
+      .address_kind = Function::AddressKind::Range,
+      .argument_kind = Function::ArgumentKind::Any,
+      .input_mode = Function::InputMode::None,
+      .desc = "Echo the given message (replacing $1-$4 with address information)",
+      .default_address = "",
+      .accept_zero = true,
+      .pre_text_mode = nullptr,
+      .handle = [](
+                  BEd &ctx,
+                  const buffer::Address &addr_,
+                  vase::Shard *,
+                  const Argument &arg_,
+                  std::vector<buffer::Line> *
+                ) {
+        auto &addr = std::get<buffer::Range>(addr_);
+        auto str = std::get<std::string>(arg_);
+        const auto first = str.find_first_not_of(" \t");
+        const auto last = str.find_last_not_of(" \t");
+        if (first == std::string::npos)
+          str.clear();
+        else
+          str = str.substr(first, last - first + 1);
+        for (size_t i = 0; i < str.size();) {
+          if (str[i] == '\\') {
+            if (i + 1 >= str.size())
+              break;
+            str.erase(i++, 1);
+            if (str[i - 1] == 'n')
+              str[i - 1] = '\n';
+            continue;
+          }
+          if (str[i] == '$' && i + 1 < str.size() && '1' <= str[i + 1] && str[i + 1] <= '4') {
+            char c = str[i + 1];
+            str.erase(i, 2);
+            switch (c) {
+            case '1': {
+              auto start = std::to_string(addr.start);
+              str.insert(i, start);
+              i += start.size();
+            } break;
+            case '2': {
+              auto end = std::to_string(addr.end);
+              str.insert(i, end);
+              i += end.size();
+            } break;
+            case '3': {
+              str.insert(i, addr.buffername);
+              i += addr.buffername.size();
+            } break;
+            case '4': {
+              auto &buf = ctx.buffer(addr.buffername);
+              auto filename = buf.filename().string();
+              str.insert(i, filename);
+              i += filename.size();
+            } break;
+            }
+            continue;
+          }
+          i++;
+        }
+        ctx.io.write_line(str);
+      },
+    }
+  );
   ctx.functions.insert(
     "cd",
     Function{
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::Any,
       .input_mode = Function::InputMode::None,
-      .desc = "Change directory.",
+      .desc = "Change directory",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -48,7 +364,7 @@ void Function::register_extented(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Print directory.",
+      .desc = "Print the current working directory",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -72,7 +388,7 @@ void Function::register_extented(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::Range,
       .input_mode = Function::InputMode::None,
-      .desc = "Exchange a range of lines for another.",
+      .desc = "Exchange a range of lines for another",
       .default_address = ".,.",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -93,7 +409,6 @@ void Function::register_extented(BEd &ctx) {
           vase::Shard::release(text);
           throw;
         }
-        ctx.current() = {arg.buffername, arg.start + addr.end - addr.start + 1};
       },
     }
   );
@@ -103,7 +418,7 @@ void Function::register_extented(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Redo last undo.",
+      .desc = "Redo the last undo modification",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -125,12 +440,12 @@ void Function::register_extented(BEd &ctx) {
     }
   );
   ctx.functions.insert(
-    "history-list",
+    "hl",
     Function{
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "List history versions.",
+      .desc = "List available history versions",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -150,5 +465,67 @@ void Function::register_extented(BEd &ctx) {
       }
     }
   );
+  ctx.functions.insert(
+    "hp",
+    Function{
+      .address_kind = Function::AddressKind::None,
+      .argument_kind = Function::ArgumentKind::Number,
+      .input_mode = Function::InputMode::None,
+      .desc = "Prune old history versions",
+      .default_address = "",
+      .accept_zero = false,
+      .pre_text_mode = nullptr,
+      .handle = [](
+                  BEd &ctx,
+                  const buffer::Address &addr_,
+                  vase::Shard *,
+                  const Argument &arg_,
+                  std::vector<buffer::Line> *
+                ) {
+        auto &addr = std::get<std::string>(addr_);
+        auto &arg = std::get<int64_t>(arg_);
+        auto &buf_ = ctx.buffer(addr);
+        if (buf_.kind != buffer::Buffer::Kind::Generic)
+          throw ed_error("Can't prune buffer.");
+        auto &buf = *(buffer::GenericBuffer *)&buf_;
+        auto versions = buf.prune(arg);
+        if (!ctx.suppress_mode)
+          ctx.io.write_line(std::format("{} undo versions left.", versions));
+      }
+    }
+  );
+  ctx.functions.insert(
+    "lang",
+    Function{
+      .address_kind = Function::AddressKind::None,
+      .argument_kind = Function::ArgumentKind::Any,
+      .input_mode = Function::InputMode::None,
+      .desc = "Set buffer language",
+      .default_address = "",
+      .accept_zero = false,
+      .pre_text_mode = nullptr,
+      .handle = [](
+                  BEd &ctx,
+                  const buffer::Address &addr_,
+                  vase::Shard *,
+                  const Argument &arg_,
+                  std::vector<buffer::Line> *
+                ) {
+        auto &addr = std::get<std::string>(addr_);
+        auto name = std::get<std::string>(arg_);
+        const auto first = name.find_first_not_of(" \t");
+        const auto last = name.find_last_not_of(" \t");
+        if (first == std::string::npos)
+          name.clear();
+        else
+          name = name.substr(first, last - first + 1);
+        auto &buf_ = ctx.buffer(addr);
+        if (buf_.kind != buffer::Buffer::Kind::Generic)
+          throw ed_error("Can't set language to buffer.");
+        auto &buf = *(buffer::GenericBuffer *)&buf_;
+        buf.language(ctx, name);
+      }
+    }
+  );
 }
 } // namespace bed::internal::functions

src/internal/functions/posix.cc

@@ -34,10 +34,22 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Line,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::Text,
-      .desc = "Append text to a line.",
+      .desc = "Append text to a line",
       .default_address = ".",
       .accept_zero = true,
-      .pre_text_mode = nullptr,
+      .pre_text_mode = [](
+                         BEd &ctx,
+                         const buffer::Address &addr_,
+                         const Argument &
+                       ) -> std::tuple<vase::Shard *, syntax::Language *, void *> {
+        const auto &addr = std::get<buffer::Line>(addr_);
+        auto &buf_ = ctx.buffer(addr.buffername);
+        if (buf_.kind != buffer::Buffer::Kind::Generic)
+          return {nullptr, nullptr, nullptr};
+        auto &buf = *(buffer::GenericBuffer *)&buf_;
+        void *state = syntax::state_before(buf.parse, buf.root, addr.number);
+        return {nullptr, buf.parse.lang, state};
+      },
       .handle = [](
                   BEd &ctx,
                   const buffer::Address &addr_,
@@ -47,9 +59,7 @@ void Function::register_posix(BEd &ctx) {
                 ) {
         auto addr = std::get<buffer::Line>(addr_);
         ctx.buffer(addr.buffername).append(ctx, text, addr.number);
-        ctx.current() = {ctx.prev().buffername, ctx.prev().end};
+        vase::Shard::release(text); }
-        vase::Shard::release(text);
-      }
     }
   );
   ctx.functions.insert(
@@ -58,10 +68,22 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::Text,
-      .desc = "Change set of lines.",
+      .desc = "Change a range of lines",
       .default_address = ".,.",
       .accept_zero = false,
-      .pre_text_mode = nullptr,
+      .pre_text_mode = [](
+                         BEd &ctx,
+                         const buffer::Address &addr_,
+                         const Argument &
+                       ) -> std::tuple<vase::Shard *, syntax::Language *, void *> {
+        const auto &addr = std::get<buffer::Range>(addr_);
+        auto &buf_ = ctx.buffer(addr.buffername);
+        if (buf_.kind != buffer::Buffer::Kind::Generic)
+          return {buf_.copy(addr.start, addr.end), nullptr, nullptr};
+        auto &buf = *(buffer::GenericBuffer *)&buf_;
+        void *state = syntax::state_before(buf.parse, buf.root, addr.start - 1);
+        return {buf.copy(addr.start, addr.end), buf.parse.lang, state};
+      },
       .handle = [](
                   BEd &ctx,
                   const buffer::Address &addr_,
@@ -69,11 +91,9 @@ void Function::register_posix(BEd &ctx) {
                   const Argument &,
                   std::vector<buffer::Line> *
                 ) {
-        auto addr = std::get<buffer::Range>(addr_);
+        const auto &addr = std::get<buffer::Range>(addr_);
         ctx.buffer(addr.buffername).replace(ctx, text, addr.start, addr.end);
-        ctx.current() = {ctx.prev().buffername, ctx.prev().end};
+        vase::Shard::release(text); }
-        vase::Shard::release(text);
-      }
     }
   );
   ctx.functions.insert(
@@ -82,7 +102,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Delete set of lines.",
+      .desc = "Delete a range of lines",
       .default_address = ".,.",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -95,7 +115,6 @@ void Function::register_posix(BEd &ctx) {
                 ) {
         auto addr = std::get<buffer::Range>(addr_);
         ctx.buffer(addr.buffername).remove(ctx, addr.start, addr.end);
-        ctx.current() = {addr.buffername, addr.start};
       }
     }
   );
@@ -105,7 +124,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::File,
       .input_mode = Function::InputMode::None,
-      .desc = "Try load a file into the current buffer.",
+      .desc = "Load a file into the current buffer, warn once if unsaved changes exist",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -125,17 +144,17 @@ void Function::register_posix(BEd &ctx) {
         vase::Shard *s = nullptr;
         if (std::holds_alternative<std::filesystem::path>(arg)) {
           auto path = std::get<std::filesystem::path>(arg);
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
           buf.set_filename(path);
         } else if (std::holds_alternative<ShellArg>(arg)) {
           auto cmd = std::get<ShellArg>(arg).cmd;
           ctx.escape_command(cmd, buf.filename().string());
-          s = vase::Shard::from_command(cmd.c_str(), true);
+          s = vase::Shard::from_command(cmd.c_str());
         } else {
           auto path = buf.filename();
           if (path.empty())
             throw ed_error("Need filename.");
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
         };
         try {
           buf.load(ctx, s);
@@ -144,8 +163,8 @@ void Function::register_posix(BEd &ctx) {
           vase::Shard::release(s);
           throw;
         }
-        ctx.io.write_line(std::format("{}", buf.bytes()));
+        if (!ctx.suppress_mode)
-        ctx.current() = {addr, buf.lines()};
+          ctx.io.write_line(std::format("{}", buf.bytes()));
       }
     }
   );
@@ -155,7 +174,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::File,
       .input_mode = Function::InputMode::None,
-      .desc = "Load a file into the current buffer.",
+      .desc = "Load a file into the current buffer, discarding unsaved changes",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -171,17 +190,17 @@ void Function::register_posix(BEd &ctx) {
         vase::Shard *s = nullptr;
         if (std::holds_alternative<std::filesystem::path>(arg)) {
           auto path = std::get<std::filesystem::path>(arg);
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
           buf.set_filename(path);
         } else if (std::holds_alternative<ShellArg>(arg)) {
           auto cmd = std::get<ShellArg>(arg).cmd;
           ctx.escape_command(cmd, buf.filename().string());
-          s = vase::Shard::from_command(cmd.c_str(), true);
+          s = vase::Shard::from_command(cmd.c_str());
         } else {
           auto path = buf.filename();
           if (path.empty())
             throw ed_error("Need filename.");
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
         };
         try {
           buf.load(ctx, s);
@@ -190,8 +209,8 @@ void Function::register_posix(BEd &ctx) {
           vase::Shard::release(s);
           throw;
         }
-        ctx.io.write_line(std::format("{}", buf.bytes()));
+        if (!ctx.suppress_mode)
-        ctx.current() = {addr, buf.lines()};
+          ctx.io.write_line(std::format("{}", buf.bytes()));
       }
     }
   );
@@ -201,7 +220,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::File,
       .input_mode = Function::InputMode::None,
-      .desc = "Set a save path.",
+      .desc = "Set/print a save path",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -241,7 +260,9 @@ void Function::register_posix(BEd &ctx) {
                   const Argument &,
                   std::vector<buffer::Line> *
                 ) {
+        ctx.io.apply(internal::io::Token::Warning);
         ctx.io.write_line(ctx.last_help);
+        ctx.io.reset();
       }
     }
   );
@@ -251,7 +272,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Toggle help mode.",
+      .desc = "Toggle help mode",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -263,8 +284,10 @@ void Function::register_posix(BEd &ctx) {
                   std::vector<buffer::Line> *
                 ) {
         ctx.help_mode = !ctx.help_mode;
+        ctx.io.apply(internal::io::Token::Warning);
         if (ctx.help_mode)
           ctx.io.write_line(ctx.last_help);
+        ctx.io.reset();
       }
     }
   );
@@ -274,10 +297,24 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Line,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::Text,
-      .desc = "Insert text before a line.",
+      .desc = "Insert text before a line",
       .default_address = ".",
       .accept_zero = true,
-      .pre_text_mode = nullptr,
+      .pre_text_mode = [](
+                         BEd &ctx,
+                         const buffer::Address &addr_,
+                         const Argument &
+                       ) -> std::tuple<vase::Shard *, syntax::Language *, void *> {
+        auto addr = std::get<buffer::Line>(addr_);
+        if (addr.number)
+          addr.number--;
+        auto &buf_ = ctx.buffer(addr.buffername);
+        if (buf_.kind != buffer::Buffer::Kind::Generic)
+          return {nullptr, nullptr, nullptr};
+        auto &buf = *(buffer::GenericBuffer *)&buf_;
+        void *state = syntax::state_before(buf.parse, buf.root, addr.number);
+        return {nullptr, buf.parse.lang, state};
+      },
       .handle = [](
                   BEd &ctx,
                   const buffer::Address &addr_,
@@ -289,9 +326,7 @@ void Function::register_posix(BEd &ctx) {
         if (addr.number)
           addr.number--;
         ctx.buffer(addr.buffername).append(ctx, text, addr.number);
-        ctx.current() = {ctx.prev().buffername, ctx.prev().end};
+        vase::Shard::release(text); }
-        vase::Shard::release(text);
-      }
     }
   );
   ctx.functions.insert(
@@ -300,7 +335,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Join a set of lines.",
+      .desc = "Join a range of lines",
       .default_address = ".,.+1",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -313,7 +348,6 @@ void Function::register_posix(BEd &ctx) {
                 ) {
         auto addr = std::get<buffer::Range>(addr_);
         ctx.buffer(addr.buffername).join(ctx, addr.start, addr.end);
-        ctx.current() = {addr.buffername, addr.start};
       }
     }
   );
@@ -323,7 +357,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Line,
       .argument_kind = Function::ArgumentKind::Mark,
       .input_mode = Function::InputMode::None,
-      .desc = "Mark a line.",
+      .desc = "Mark a line",
       .default_address = ".",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -345,7 +379,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "List (print unambiguous) range",
+      .desc = "Print a range, showing nonprinting characters unambiguously",
       .default_address = ".,.",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -368,7 +402,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::Line,
       .input_mode = Function::InputMode::None,
-      .desc = "Move a range of lines.",
+      .desc = "Move a range of lines",
       .default_address = ".,.",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -382,8 +416,7 @@ void Function::register_posix(BEd &ctx) {
         auto addr = std::get<buffer::Range>(addr_);
         auto arg = std::get<buffer::Line>(arg_);
         if (arg.buffername == addr.buffername
-            && addr.start <= arg.number
+            && arg.number >= addr.start && arg.number < addr.end)
-            && addr.end < arg.number)
           throw ed_error("Can't move lines within themselves.");
         auto text = ctx.buffer(addr.buffername).copy(addr.start, addr.end);
         ctx.mark(252, arg);
@@ -403,7 +436,6 @@ void Function::register_posix(BEd &ctx) {
           vase::Shard::release(text);
           throw;
         }
-        ctx.current() = {arg.buffername, arg.number + addr.end - addr.start + 1};
       },
     }
   );
@@ -413,7 +445,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Print range with line numbers",
+      .desc = "Print a range with line numbers",
       .default_address = ".,.",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -436,7 +468,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Print range",
+      .desc = "Print a range",
       .default_address = ".,.",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -457,9 +489,9 @@ void Function::register_posix(BEd &ctx) {
     "P",
     Function{
       .address_kind = Function::AddressKind::None,
-      .argument_kind = Function::ArgumentKind::None,
+      .argument_kind = Function::ArgumentKind::Any,
       .input_mode = Function::InputMode::None,
-      .desc = "Toggle prompt.",
+      .desc = "Toggle/set prompt string",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -467,13 +499,18 @@ void Function::register_posix(BEd &ctx) {
                   BEd &ctx,
                   const buffer::Address &,
                   vase::Shard *,
-                  const Argument &,
+                  const Argument &arg_,
                   std::vector<buffer::Line> *
                 ) {
-        ctx.prompt_mode = !ctx.prompt_mode;
+        auto &arg = std::get<std::string>(arg_);
-        if (ctx.prompt_mode && !ctx.prompt) {
+        if (arg.size()) {
-          ctx.prompt = [](BEd &) { return "*"; };
+          ctx.prompt_mode = true;
+          ctx.prompt = [arg](BEd &) { return arg; };
+          return;
         }
+        ctx.prompt_mode = !ctx.prompt_mode;
+        if (ctx.prompt_mode && !ctx.prompt)
+          ctx.prompt = [](BEd &) { return "*"; };
       }
     }
   );
@@ -483,7 +520,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Try quitting.",
+      .desc = "Quit, warn once if unsaved changes exist",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -514,7 +551,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Force quit.",
+      .desc = "Quit without saving, discarding unsaved changes",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -535,7 +572,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Line,
       .argument_kind = Function::ArgumentKind::File,
       .input_mode = Function::InputMode::None,
-      .desc = "Read from file into buffer.",
+      .desc = "Read a file's contents into the buffer after the given address",
       .default_address = "$",
       .accept_zero = true,
       .pre_text_mode = nullptr,
@@ -551,28 +588,28 @@ void Function::register_posix(BEd &ctx) {
         vase::Shard *s = nullptr;
         if (std::holds_alternative<std::filesystem::path>(arg)) {
           auto path = std::get<std::filesystem::path>(arg);
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
           if (buf.filename().empty())
             buf.set_filename(path);
         } else if (std::holds_alternative<ShellArg>(arg)) {
           auto cmd = std::get<ShellArg>(arg).cmd;
           ctx.escape_command(cmd, buf.filename().string());
-          s = vase::Shard::from_command(cmd.c_str(), true);
+          s = vase::Shard::from_command(cmd.c_str());
         } else {
           auto path = buf.filename();
           if (path.empty())
             throw ed_error("Need filename.");
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
         };
         try {
           buf.append(ctx, s, addr.number);
-          ctx.io.write_line(std::format("{}", s ? s->length + 1 : 0));
-          ctx.current() = {addr.buffername, addr.number + (s ? s->lines + 1 : 0)};
           vase::Shard::release(s);
         } catch (...) {
           vase::Shard::release(s);
           throw;
         }
+        if (!ctx.suppress_mode)
+          ctx.io.write_line(std::format("{}", s ? s->length + 1 : 0));
       }
     }
   );
@@ -582,7 +619,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::Regex,
       .input_mode = Function::InputMode::None,
-      .desc = "Substitute regex.",
+      .desc = "Substitute pattern in range for replacement",
       .default_address = ".,.",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -621,7 +658,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::Line,
       .input_mode = Function::InputMode::None,
-      .desc = "Copy a range of lines.",
+      .desc = "Copy a range of lines",
       .default_address = ".,.",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -642,7 +679,6 @@ void Function::register_posix(BEd &ctx) {
           vase::Shard::release(text);
           throw;
         }
-        ctx.current() = {arg.buffername, arg.number + addr.end - addr.start + 1};
       },
     }
   );
@@ -652,7 +688,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Undo last modification to buffer.",
+      .desc = "Undo the last modification to the buffer",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -679,7 +715,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::File,
       .input_mode = Function::InputMode::None,
-      .desc = "Write buffer to disk.",
+      .desc = "Write a range of lines to disk",
       .default_address = "0,$",
       .accept_zero = true,
       .pre_text_mode = nullptr,
@@ -715,12 +751,14 @@ void Function::register_posix(BEd &ctx) {
               throw ed_error("Need filename.");
             vase::write_file(path, text);
           };
-          ctx.io.write(std::format("{}\n", text ? text->length + 1 : 0));
           vase::Shard::release(text);
         } catch (...) {
           vase::Shard::release(text);
           throw;
         }
+        buf.saved_hook();
+        if (!ctx.suppress_mode)
+          ctx.io.write(std::format("{}\n", text ? text->length + 1 : 0));
       }
     }
   );
@@ -730,7 +768,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::Range,
       .argument_kind = Function::ArgumentKind::None,
       .input_mode = Function::InputMode::None,
-      .desc = "Print line numbers",
+      .desc = "Print the line number of the given address",
       .default_address = "$",
       .accept_zero = true,
       .pre_text_mode = nullptr,
@@ -742,10 +780,22 @@ void Function::register_posix(BEd &ctx) {
                   std::vector<buffer::Line> *
                 ) {
         auto &addr = std::get<buffer::Range>(addr_);
-        if (addr.start == addr.end)
+        ctx.io.apply(io::Token::BufferName);
-          ctx.io.write_line(std::format(":{}:{}", addr.buffername, addr.start));
+        ctx.io.write(":" + addr.buffername + ":");
-        else
+        ctx.io.apply(io::Token::Number);
-          ctx.io.write_line(std::format(":{}:{},{}", addr.buffername, addr.start, addr.end));
+        if (addr.start == addr.end) {
+          ctx.io.write_line(std::format(" {}", addr.start));
+        } else {
+          ctx.io.write(std::format(" {}", addr.start));
+          ctx.io.apply(io::Token::AddressSeperator);
+          ctx.io.write(",");
+          ctx.io.apply(io::Token::Number);
+          ctx.io.write_line(std::format("{}", addr.end));
+        }
+        ctx.io.reset();
+        ctx.prev().buffername = addr.buffername;
+        ctx.prev().start = addr.start;
+        ctx.prev().end = addr.end;
         ctx.current() = {addr.buffername, addr.end};
       },
     }
@@ -756,7 +806,7 @@ void Function::register_posix(BEd &ctx) {
       .address_kind = Function::AddressKind::None,
       .argument_kind = Function::ArgumentKind::Shell,
       .input_mode = Function::InputMode::None,
-      .desc = "Run a shell command.",
+      .desc = "Run a shell command",
       .default_address = "",
       .accept_zero = false,
       .pre_text_mode = nullptr,
@@ -774,7 +824,8 @@ void Function::register_posix(BEd &ctx) {
         if (ctx.escape_command(cmd, filename.string()))
           ctx.io.write(cmd + "\n");
         ctx.io.run_pty(cmd);
-        ctx.io.write("!\n");
+        if (!ctx.suppress_mode)
+          ctx.io.write("!\n");
       },
     }
   );
@@ -782,7 +833,7 @@ void Function::register_posix(BEd &ctx) {
     .address_kind = Function::AddressKind::Line,
     .argument_kind = Function::ArgumentKind::None,
     .input_mode = Function::InputMode::None,
-    .desc = "Prints a line and jumps to it.",
+    .desc = "Print given line and move the cursor to it",
     .default_address = ".+1",
     .accept_zero = true,
     .pre_text_mode = nullptr,
@@ -797,13 +848,16 @@ void Function::register_posix(BEd &ctx) {
       if (addr.number != 0)
         ctx.buffer(addr.buffername).print(ctx, addr.number, addr.number);
       ctx.current() = addr;
+      ctx.prev().buffername = addr.buffername;
+      ctx.prev().start = addr.number;
+      ctx.prev().end = addr.number;
     }
   };
   ctx.eof_op = Function{
     .address_kind = Function::AddressKind::None,
     .argument_kind = Function::ArgumentKind::None,
     .input_mode = Function::InputMode::None,
-    .desc = "Try quitting.",
+    .desc = "Quit, warn once if unsaved changes exist",
     .default_address = "",
     .accept_zero = false,
     .pre_text_mode = nullptr,

src/internal/io/input.cc

@@ -1,6 +1,33 @@
 #include "internal/io/io.h"
 
 namespace bed::internal::io {
+std::pair<std::string, bool> IO::read_pipe() {
+  char buf[4096];
+  while (true) {
+    auto pos = pipe_input.find('\n');
+    if (pos != std::string::npos) {
+      std::string line = pipe_input.substr(0, pos);
+      pipe_input.erase(0, pos + 1);
+      return {line, false};
+    }
+    ssize_t n = read(STDIN_FILENO, buf, sizeof(buf));
+    if (n > 0) {
+      pipe_input.append(buf, n);
+      continue;
+    }
+    if (n == 0) {
+      if (pipe_input.empty())
+        return {"", true};
+      std::string line = std::move(pipe_input);
+      pipe_input.clear();
+      return {line, false};
+    }
+    if (errno == EINTR)
+      continue;
+    throw fatal_error("Can't read stdin.", 1);
+  }
+}
+
 KeyEvent::ReadResult IO::get_next_byte(char &out) {
   if (!input_queue.empty()) {
     out = input_queue.front();

src/internal/io/io.cc

@@ -1,12 +1,17 @@
 #include "internal/io/io.h"
+#include "bed.h"
 
 namespace bed::internal::io {
 termios IO::orig_termios{};
 termios IO::raw_termios{};
 bool IO::cleaned = true;
 volatile std::atomic_bool IO::resized(false);
+IO::Mode IO::mode = IO::Mode::PIPE;
 
-IO::IO() {
+IO::IO(BEd &bed) : bed(bed) {
+  if (!isatty(STDIN_FILENO))
+    return;
+  mode = Mode::TERMINAL;
   if (tcgetattr(STDIN_FILENO, &orig_termios) == -1)
     throw fatal_error("Can't get terminal state.", 1);
   struct sigaction sa{};
@@ -29,17 +34,54 @@ IO::~IO() {
   cleanup();
 }
 
+void IO::apply(const io::Token::Kind &t) {
+  if (!bed.color_mode)
+    return;
+  write("\x1b[0m");
+  const auto &hl = bed.theme.get(t);
+  const uint8_t r = (hl.fg >> 16) & 0xff;
+  const uint8_t g = (hl.fg >> 8) & 0xff;
+  const uint8_t b = hl.fg & 0xff;
+  write(std::format("\x1b[38;2;{};{};{}m", r, g, b));
+  if (hl.bg != 0) {
+    const uint8_t br = (hl.bg >> 16) & 0xff;
+    const uint8_t bg = (hl.bg >> 8) & 0xff;
+    const uint8_t bb = hl.bg & 0xff;
+    write(std::format("\x1b[48;2;{};{};{}m", br, bg, bb));
+  }
+  if (hl.flags & Highlight::Bold)
+    write("\x1b[1m");
+  if (hl.flags & Highlight::Italic)
+    write("\x1b[3m");
+  if (hl.flags & Highlight::Underline)
+    write("\x1b[4m");
+  if (hl.flags & Highlight::Strikethrough)
+    write("\x1b[9m");
+}
+
+void IO::reset() {
+  if (!bed.color_mode)
+    return;
+  write("\x1b[0m");
+}
+
 void IO::enable_mouse() {
+  if (mode == Mode::PIPE)
+    throw fatal_error("no mouse in pipe mode.", 1);
   const char *seq = "\x1b[?1000h";
   write_all(STDOUT_FILENO, seq, 8);
 }
 
 void IO::disable_mouse() {
+  if (mode == Mode::PIPE)
+    throw fatal_error("no mouse in pipe mode.", 1);
   const char *seq = "\x1b[?1000l";
   write_all(STDOUT_FILENO, seq, 8);
 }
 
 std::pair<uint16_t, uint16_t> IO::terminal_size() {
+  if (mode == Mode::PIPE)
+    throw fatal_error("no terminal size in pipe mode.", 1);
   struct winsize ws{};
   if (ioctl(STDOUT_FILENO, TIOCGWINSZ, &ws) == -1)
     throw fatal_error("Can't get terminal size.", 1);
@@ -47,6 +89,8 @@ std::pair<uint16_t, uint16_t> IO::terminal_size() {
 }
 
 std::pair<uint16_t, uint16_t> IO::cursor_position() {
+  if (mode == Mode::PIPE)
+    throw fatal_error("no cursor in pipe mode.", 1);
   write_all(STDOUT_FILENO, "\x1b[6n", 4);
   std::string response;
   char c;
@@ -69,7 +113,7 @@ std::pair<uint16_t, uint16_t> IO::cursor_position() {
 }
 
 void IO::enable_raw() {
-  if (!cleaned)
+  if (!cleaned || mode == Mode::PIPE)
     return;
   std::string os = "\x1b[?2004h";
   write_all(STDOUT_FILENO, os.c_str(), os.size());
@@ -79,7 +123,7 @@ void IO::enable_raw() {
 }
 
 void IO::cleanup() {
-  if (cleaned)
+  if (cleaned || mode == Mode::PIPE)
     return;
   std::string os = "\x1b[?1000l\x1b[?2004l";
   write_all(STDOUT_FILENO, os.c_str(), os.size());
@@ -93,6 +137,8 @@ void IO::handle_sigwinch(int) {
 }
 
 void IO::move_cursor(uint16_t row, uint16_t col) {
+  if (mode == Mode::PIPE)
+    return;
   char buf[32];
   int n = snprintf(buf, sizeof(buf), "\x1b[%u;%uH", row, col);
   write_all(STDOUT_FILENO, buf, n);
@@ -112,6 +158,8 @@ void IO::write_line(std::string_view s) {
 }
 
 void IO::run_pty(const std::string &cmd) {
+  if (mode == Mode::PIPE)
+    throw ed_error("Shell running not allowed in pipe mode.");
   int master_fd = -1;
   struct winsize ws{};
   if (ioctl(STDOUT_FILENO, TIOCGWINSZ, &ws) == -1)

src/internal/parser/address.cc

@@ -6,34 +6,49 @@ void Parser::locator(AddressPromise &addr) {
   addr.base = AddressPromise::None{};
   switch (peek()) {
   case '.':
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
     addr.base = AddressPromise::Current{};
     break;
   case '$':
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
     addr.base = AddressPromise::Last{};
     break;
   case '%':
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
     addr.base = AddressPromise::LastRange{};
     break;
   case '[':
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
     addr.base = AddressPromise::Block{Direction::Backward};
     break;
   case ']':
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
     addr.base = AddressPromise::Block{Direction::Forward};
     break;
   case '^':
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
     addr.base = AddressPromise::Diagnostic{Direction::Backward};
     break;
   case '~':
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
     addr.base = AddressPromise::Diagnostic{Direction::Forward};
     break;
   case '\'':
+    token(io::Token::Mark);
     advance();
     if (('a' <= peek() && peek() <= 'z')
         || ('A' <= peek() && peek() <= 'Z'))
@@ -41,12 +56,16 @@ void Parser::locator(AddressPromise &addr) {
     else
       throw ed_error("Valid mark needed after \'");
     advance();
+    end_token();
     break;
   case '{': {
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
     uint16_t j = 0;
     std::string func;
     std::string arg;
+    token(io::Token::RubyFunction);
     while (peek(j) != '}') {
       if (peek(j) == '\0')
         throw ed_error("Scripted address not terminated");
@@ -55,6 +74,8 @@ void Parser::locator(AddressPromise &addr) {
       if (peek(j) == ':') {
         func = peek_str(j);
         advance(j + 1);
+        end_token();
+        token(io::Token::RubyArg);
         j = 0;
         continue;
       }
@@ -65,9 +86,14 @@ void Parser::locator(AddressPromise &addr) {
     else
       func = peek_str(j);
     addr.base = AddressPromise::Scripted{std::move(func), std::move(arg)};
-    advance(j + 1);
+    advance(j);
+    end_token();
+    token(io::Token::AddressSymbol);
+    advance();
+    end_token();
   } break;
   case '/': {
+    token(io::Token::Regexp);
     advance();
     uint64_t j = 0;
     while (true) {
@@ -88,8 +114,10 @@ void Parser::locator(AddressPromise &addr) {
       std::string(peek_str(j))
     );
     advance(j + 1);
+    end_token();
   } break;
   case '?': {
+    token(io::Token::Regexp);
     advance();
     uint16_t j = 0;
     while (true) {
@@ -110,8 +138,10 @@ void Parser::locator(AddressPromise &addr) {
       std::string(peek_str(j))
     );
     advance(j + 1);
+    end_token();
   } break;
   case '<': {
+    token(io::Token::Label);
     advance();
     uint16_t j = 0;
     while (peek(j) != '>'
@@ -133,8 +163,10 @@ void Parser::locator(AddressPromise &addr) {
       break;
     }
     advance(j + 1);
+    end_token();
   } break;
   case '>': {
+    token(io::Token::Label);
     advance();
     uint16_t j = 0;
     while (peek(j) != '>' && peek(j) != '\0')
@@ -150,9 +182,13 @@ void Parser::locator(AddressPromise &addr) {
       break;
     }
     advance(j + 1);
+    end_token();
   } break;
   case '+': {
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
+    token(io::Token::Number);
     addr.base = AddressPromise::Current{};
     uint16_t j = 0;
     uint64_t num = 0;
@@ -162,11 +198,15 @@ void Parser::locator(AddressPromise &addr) {
     }
     if (j == 0)
       num = 1;
-    advance(j);
     addr.offset += num;
+    advance(j);
+    end_token();
   } break;
   case '-': {
+    token(io::Token::AddressSymbol);
     advance();
+    end_token();
+    token(io::Token::Number);
     addr.base = AddressPromise::Current{};
     uint16_t j = 0;
     uint64_t num = 0;
@@ -176,17 +216,20 @@ void Parser::locator(AddressPromise &addr) {
     }
     if (j == 0)
       num = 1;
-    advance(j);
     addr.offset -= num;
+    advance(j);
+    end_token();
   } break;
   default:
     if ('0' <= peek() && peek() <= '9') {
+      token(io::Token::Number);
       uint64_t num = 0;
       while ('0' <= peek() && peek() <= '9') {
         num = num * 10 + (peek() - '0');
         advance();
       }
       addr.base = AddressPromise::Number{num};
+      end_token();
     }
   }
 }
@@ -196,8 +239,11 @@ int64_t Parser::offset() {
   while (peek() == '+' || peek() == '-'
          || ('0' <= peek() && peek() <= '9')) {
     bool positive = peek() != '-';
+    token(io::Token::AddressSymbol);
     if (peek() == '+' || peek() == '-')
       advance();
+    end_token();
+    token(io::Token::Number);
     uint16_t j = 0;
     int64_t num = 0;
     while ('0' <= peek(j) && peek(j) <= '9')
@@ -205,6 +251,7 @@ int64_t Parser::offset() {
     if (j == 0)
       num = 1;
     advance(j);
+    end_token();
     offset += positive ? num : -num;
     skip_ws();
   }
@@ -214,6 +261,7 @@ int64_t Parser::offset() {
 
 void Parser::address(AddressPromise &addr) {
   if (peek() == ':') {
+    token(io::Token::BufferName);
     advance();
     uint16_t j = 0;
     while (peek(j) != ':' && peek(j) != '\0')
@@ -222,6 +270,7 @@ void Parser::address(AddressPromise &addr) {
     advance(j);
     if (peek() == ':')
       advance();
+    end_token();
   }
   skip_ws();
   if (peek() == '\0')
@@ -238,6 +287,7 @@ void Parser::addresses(std::vector<AddressPromise> &addresses) {
   address(*addr);
   while (peek() == ',' || peek() == ';') {
     addr->jumping = peek() == ';';
+    token(io::Token::AddressSeperator);
     advance();
     skip_ws();
     addresses.push_back({});

src/internal/parser/address_promise.cc

@@ -81,16 +81,10 @@ std::optional<buffer::Line> AddressPromise::get_line(BEd &ctx, std::vector<Addre
   AddressPromise prev;
   for (std::size_t idx = 0; idx < list.size(); idx++) {
     AddressPromise &curr = list[idx];
-    if (curr.bufname.has_value()) {
+    if (!curr.bufname.has_value())
-      if (curr.bufname->empty()) {
-        bufname = ctx.current().buffername;
-        curr.bufname = bufname;
-      } else {
-        bufname = *curr.bufname;
-      }
-    } else {
       curr.bufname = bufname;
-    }
+    else if (curr.bufname->empty())
+      curr.bufname = bufname = ctx.current().buffername;
     bool is_final = idx + 1 == list.size();
     if (!is_final) {
       if (std::holds_alternative<None>(curr.base)) {
@@ -107,15 +101,15 @@ std::optional<buffer::Line> AddressPromise::get_line(BEd &ctx, std::vector<Addre
       } else {
         prev_given = true;
       }
-      if (curr.jumping) {
+      buffer::Line resolved = curr.resolve(ctx);
-        buffer::Line resolved = curr.resolve(ctx);
+      if (curr.jumping)
         ctx.current() = resolved;
-        curr.bufname = resolved.buffername;
+      curr.bufname = resolved.buffername;
-        curr.base = Number(resolved.number);
+      curr.base = Number(resolved.number);
-        curr.offset = 0;
+      curr.offset = 0;
-      }
       prev = curr;
       prev_set = true;
+      bufname = *curr.bufname;
     } else {
       if (std::holds_alternative<None>(curr.base)) {
         if (prev_set) {
@@ -144,16 +138,10 @@ std::optional<buffer::Range> AddressPromise::get_range(BEd &ctx, std::vector<Add
   AddressPromise prev;
   for (std::size_t idx = 0; idx < list.size(); idx++) {
     AddressPromise &curr = list[idx];
-    if (curr.bufname.has_value()) {
+    if (!curr.bufname.has_value())
-      if (curr.bufname->empty()) {
-        bufname = ctx.current().buffername;
-        curr.bufname = bufname;
-      } else {
-        bufname = *curr.bufname;
-      }
-    } else {
       curr.bufname = bufname;
-    }
+    else if (curr.bufname->empty())
+      curr.bufname = bufname = ctx.current().buffername;
     bool is_final = idx + 1 == list.size();
     if (!is_final) {
       if (std::holds_alternative<None>(curr.base)) {
@@ -170,15 +158,15 @@ std::optional<buffer::Range> AddressPromise::get_range(BEd &ctx, std::vector<Add
       } else {
         prev_given = true;
       }
-      if (curr.jumping) {
+      buffer::Line resolved = curr.resolve(ctx);
-        buffer::Line resolved = curr.resolve(ctx);
+      curr.bufname = resolved.buffername;
+      curr.base = Number(resolved.number);
+      curr.offset = 0;
+      if (curr.jumping)
         ctx.current() = resolved;
-        curr.bufname = resolved.buffername;
-        curr.base = Number(resolved.number);
-        curr.offset = 0;
-      }
       prev = curr;
       prev_set = true;
+      bufname = *curr.bufname;
     } else {
       if (std::holds_alternative<None>(curr.base)) {
         if (prev_set) {

src/internal/parser/operation.cc

@@ -11,7 +11,9 @@ void Parser::operation() {
   if (len == 0)
     throw ed_error("Function not found.");
   functions::Function *function = bed.functions.get_ptr(peek_str(len));
+  token(io::Token::Function);
   advance(len);
+  end_token();
   command->function = function;
   char suffix = '\0';
   switch (command->function->argument_kind) {
@@ -27,11 +29,15 @@ void Parser::operation() {
       command->argument = peek();
     else
       throw ed_error("Valid mark needed.");
+    token(io::Token::Mark);
     advance();
+    end_token();
     break;
   case functions::Function::ArgumentKind::Any:
     command->argument = std::string(peek_str());
+    token(io::Token::Data);
     advance(peek_str().size());
+    end_token();
     break;
   case functions::Function::ArgumentKind::Global: {
     char delim;
@@ -108,16 +114,22 @@ void Parser::operation() {
     skip_ws();
     switch (peek()) {
     case '!':
+      token(io::Token::Error);
       advance();
+      end_token();
+      token(io::Token::Shell);
       command->argument = functions::Function::ShellArg(std::string(peek_str()));
       advance(peek_str().size());
+      end_token();
       break;
     case '\0':
       command->argument = std::monostate();
       break;
     default:
+      token(io::Token::File);
       command->argument = std::filesystem::path(peek_str());
       advance(peek_str().size());
+      end_token();
       break;
     }
     break;
@@ -133,6 +145,7 @@ void Parser::operation() {
     char delim = peek();
     if (delim == '\0')
       throw ed_error("regex expected");
+    token(io::Token::Regexp);
     advance();
     uint16_t j = 0;
     while (true) {
@@ -165,16 +178,20 @@ void Parser::operation() {
         j++;
     }
     std::string replacement(peek_str(j));
+    std::string options;
     if (peek(j) != '\0') {
       advance(j + 1);
     } else {
       advance(j);
+      options = "p";
     }
-    std::string options;
+    end_token();
+    token(io::Token::Suffix);
     if (peek() != '\0') {
       options = std::string(peek_str());
       advance(peek_str().size());
     }
+    end_token();
     std::erase_if(options, [&](char c) {
       if (bed.suffixes[c - 'a'].has_value()) {
         suffix = c;
@@ -184,18 +201,33 @@ void Parser::operation() {
     });
     command->argument = functions::Function::RegexArg(expression, replacement, options);
   } break;
-  case functions::Function::ArgumentKind::Ruby:
+  case functions::Function::ArgumentKind::Ruby: {
     command->argument = functions::Function::RubyArg(std::string(peek_str()));
+    auto *ruby_parser = bed.languages["ruby"];
+    void *state = ruby_parser->none_state();
+    std::vector<syntax::ParseEvent> events;
+    std::vector<io::Token> tokens_;
+    ruby_parser->parse(&state, peek_str(), false, &tokens_, &events);
+    for (auto &token : tokens_) {
+      token.start += i;
+      token.end += i;
+      tokens->push_back(token);
+    }
+    ruby_parser->destroy(state);
     advance(peek_str().size());
-    break;
+  } break;
   case functions::Function::ArgumentKind::Shell:
     command->argument = functions::Function::ShellArg(std::string(peek_str()));
+    token(io::Token::Shell);
     advance(peek_str().size());
+    end_token();
     break;
   }
   if (!suffix) {
     suffix = peek();
+    token(io::Token::Suffix);
     advance();
+    end_token();
   }
   if (suffix) {
     auto &s = bed.suffixes[suffix - 'a'];

src/internal/parser/parser.cc

@@ -2,6 +2,19 @@
 #include "bed.h"
 
 namespace bed::internal::parser {
+void Parser::token(io::Token::Kind k) {
+  end_token();
+  tokens->push_back({.start = i, .end = 0, .type = k});
+}
+
+void Parser::end_token() {
+  if (tokens->empty())
+    return;
+  auto &b = tokens->back();
+  if (!b.end)
+    b.end = i;
+}
+
 char Parser::peek(uint16_t o) {
   return i + o < cmd.size() ? cmd[i + o] : '\0';
 }
@@ -15,36 +28,60 @@ void Parser::advance(uint16_t c) {
 }
 
 void Parser::skip_ws() {
+  end_token();
   while (peek() == ' ' || peek() == '\t')
     advance();
 }
 
 void Parser::parse() {
-  skip_ws();
+  try {
-  if (peek() == '@') {
+    skip_ws();
-    advance();
+    if (peek() == '@') {
-    command->temp_address = true;
+      token(io::Token::TempCurrent);
-  } else {
+      advance();
-    command->temp_address = false;
+      command->temp_address = true;
+    } else {
+      command->temp_address = false;
+    }
+    skip_ws();
+    addresses(command->addresses);
+    operation();
+    skip_ws();
+    if (peek() != '\0') {
+      token(io::Token::Error);
+      throw ed_error("Malformed command");
+    }
+    advance(cmd.size() - i);
+    end_token();
+  } catch (const ed_error &e) {
+    advance(cmd.size() - i);
+    end_token();
+    throw;
   }
-  skip_ws();
-  addresses(command->addresses);
-  operation();
-  skip_ws();
-  if (peek() != '\0')
-    throw ed_error("Malformed command");
 }
 
 Parser::Parser(
   std::string_view cmd, BEd &bed, Command *command,
-  std::vector<ui::Token> *tokens, CompletionContext *completion
+  std::vector<io::Token> *tokens, CompletionContext *completion
 ) : bed(bed), cmd(cmd), command(command), tokens(tokens), completion(completion) {
   i = 0;
 }
 
+std::vector<io::Token> Parser::get_highlight(std::string_view cmd, BEd &bed) {
+  Command c;
+  std::vector<io::Token> tokens;
+  CompletionContext completion;
+  try {
+    Parser p(cmd, bed, &c, &tokens, &completion);
+    p.parse();
+  } catch (const ed_error &e) {
+  }
+  return tokens;
+}
+
 Command Parser::get_command(std::string_view cmd, BEd &bed) {
   Command c;
-  std::vector<ui::Token> tokens;
+  std::vector<io::Token> tokens;
   CompletionContext completion;
   Parser p(cmd, bed, &c, &tokens, &completion);
   p.parse();
@@ -53,7 +90,7 @@ Command Parser::get_command(std::string_view cmd, BEd &bed) {
 
 std::vector<AddressPromise> Parser::get_addresses(std::string_view cmd, BEd &bed) {
   std::vector<AddressPromise> result;
-  std::vector<ui::Token> tokens;
+  std::vector<io::Token> tokens;
   CompletionContext completion;
   Parser p(cmd, bed, nullptr, &tokens, &completion);
   p.addresses(result);

src/internal/scripting/ruby.cc

@@ -0,0 +1,149 @@
+#include "internal/scripting/ruby.h"
+#include "bed.h"
+
+namespace bed::internal::scripting {
+static mrb_value mrb_bed_exit(mrb_state *mrb, mrb_value) {
+  mrb_raise(mrb, E_RUNTIME_ERROR, "Use `handle(\"q\")` to quit.");
+  return mrb_nil_value();
+}
+
+static void raise_fatal(mrb_state *mrb, const fatal_error &e) {
+  struct RClass *klass = mrb_class_get(mrb, "FatalError");
+  mrb_value exc = mrb_exc_new_str(mrb, klass, mrb_str_new_cstr(mrb, e.what()));
+  mrb_iv_set(mrb, exc, mrb_intern_lit(mrb, "@code"), mrb_fixnum_value(e.code));
+  mrb_exc_raise(mrb, exc);
+}
+
+static mrb_value mrb_bed_handle(mrb_state *mrb, mrb_value) {
+  auto &ctx = *(BEd *)mrb->ud;
+  const char *command;
+  mrb_int len;
+  mrb_get_args(mrb, "s", &command, &len);
+  std::string_view cmd(command, len);
+  try {
+    ctx.handle(cmd, false);
+  } catch (const ed_error &e) {
+    mrb_raise(mrb, mrb_class_get(mrb, "EdError"), e.what());
+  } catch (const fatal_error &f) {
+    raise_fatal(mrb, f);
+  }
+  return mrb_nil_value();
+}
+
+static mrb_value hash_get(mrb_state *mrb, mrb_value hash, const char *name) {
+  if (mrb_nil_p(hash))
+    return mrb_nil_value();
+  return mrb_hash_get(mrb, hash, mrb_symbol_value(mrb_intern_cstr(mrb, name)));
+}
+
+static functions::Function::AddressKind parse_address_kind(mrb_state *mrb, mrb_value value) {
+  if (mrb_nil_p(value))
+    return functions::Function::AddressKind::None;
+  if (!mrb_symbol_p(value))
+    mrb_raise(mrb, E_TYPE_ERROR, "address must be a Symbol");
+  auto name = mrb_sym_name(mrb, mrb_symbol(value));
+  if (strcmp(name, "none") == 0)
+    return functions::Function::AddressKind::None;
+  if (strcmp(name, "line") == 0)
+    return functions::Function::AddressKind::Line;
+  if (strcmp(name, "range") == 0)
+    return functions::Function::AddressKind::Range;
+  mrb_raisef(mrb, E_ARGUMENT_ERROR, "invalid address type: :%s", name);
+  return functions::Function::AddressKind::None;
+}
+
+static mrb_value mrb_bed_register(mrb_state *mrb, mrb_value) {
+  auto &ctx = *(BEd *)mrb->ud;
+  mrb_sym r_cmd_name;
+  mrb_value proc;
+  mrb_value opts = mrb_nil_value();
+  mrb_get_args(mrb, "n&|H", &r_cmd_name, &proc, &opts);
+  mrb_int cmd_name_len;
+  const char *name = mrb_sym_name_len(mrb, r_cmd_name, &cmd_name_len);
+  std::string_view cmd_name(name, cmd_name_len);
+  std::string desc;
+  mrb_value r_desc = hash_get(mrb, opts, "desc");
+  if (mrb_string_p(r_desc))
+    desc.assign(RSTRING_PTR(r_desc), RSTRING_LEN(r_desc));
+  std::string default_address;
+  mrb_value r_default_address = hash_get(mrb, opts, "default");
+  if (mrb_string_p(r_default_address))
+    default_address.assign(RSTRING_PTR(r_default_address), RSTRING_LEN(r_default_address));
+  ctx.functions.insert(
+    cmd_name,
+    functions::Function{
+      .address_kind = parse_address_kind(mrb, hash_get(mrb, opts, "address")),
+      .argument_kind = functions::Function::ArgumentKind::None,
+      .input_mode = functions::Function::InputMode::None,
+      .desc = desc,
+      .default_address = default_address,
+      .accept_zero = true,
+      .pre_text_mode = nullptr,
+      .handle = [b = Block(mrb, proc)](
+                  BEd &,
+                  const buffer::Address &,
+                  vase::Shard *,
+                  const functions::Function::Argument &,
+                  std::vector<buffer::Line> *
+                ) {
+        b.call();
+      }
+    }
+  );
+  return mrb_nil_value();
+}
+
+static mrb_value mrb_bed_unregister(mrb_state *mrb, mrb_value) {
+  auto &ctx = *(BEd *)mrb->ud;
+  mrb_sym r_cmd_name;
+  mrb_get_args(mrb, "n", &r_cmd_name);
+  mrb_int cmd_name_len;
+  const char *name = mrb_sym_name_len(mrb, r_cmd_name, &cmd_name_len);
+  std::string_view cmd_name(name, cmd_name_len);
+  ctx.functions.remove(cmd_name);
+  return mrb_nil_value();
+}
+
+void register_basic(BEd &ctx) {
+  auto mrb = ctx.mrb.state;
+  auto *bed_error =
+    mrb_define_class(mrb, "EdError", mrb_exc_get_id(mrb, MRB_ERROR_SYM(RuntimeError)));
+  mrb_define_class(mrb, "FatalError", bed_error);
+  mrb_define_method(mrb, mrb->kernel_module, "exit", mrb_bed_exit, MRB_ARGS_NONE());
+  mrb_define_method(mrb, mrb->kernel_module, "handle", mrb_bed_handle, MRB_ARGS_REQ(1));
+  mrb_define_method(mrb, mrb->kernel_module, "register", mrb_bed_register, MRB_ARGS_REQ(1) | MRB_ARGS_OPT(1) | MRB_ARGS_BLOCK());
+  mrb_define_method(mrb, mrb->kernel_module, "unregister", mrb_bed_unregister, MRB_ARGS_REQ(1));
+}
+
+std::string run(BEd &ctx, const std::string &str) {
+  mrb_state *mrb = ctx.mrb.state;
+  mrb_value result = mrb_load_nstring(mrb, str.data(), str.size());
+  if (!mrb->exc) {
+    mrb_value inspected = mrb_funcall(mrb, result, "inspect", 0);
+    std::string output(RSTRING_PTR(inspected), RSTRING_LEN(inspected));
+    return output;
+  }
+  mrb_value exc = mrb_obj_value(mrb->exc);
+  mrb_value msg = mrb_funcall(mrb, exc, "message", 0);
+  std::string error;
+  if (mrb_string_p(msg))
+    error.assign(RSTRING_PTR(msg), RSTRING_LEN(msg));
+  auto *fatal_class = mrb_class_get(mrb, "FatalError");
+  if (mrb_obj_is_kind_of(mrb, exc, fatal_class)) {
+    mrb_value code =
+      mrb_iv_get(mrb, exc, mrb_intern_lit(mrb, "@code"));
+    mrb->exc = nullptr;
+    int c = 1;
+    if (mrb_fixnum_p(code))
+      c = mrb_fixnum(code);
+    throw fatal_error(error, c);
+  }
+  auto *ed_class = mrb_class_get(mrb, "EdError");
+  if (mrb_obj_is_kind_of(mrb, exc, ed_class)) {
+    mrb->exc = nullptr;
+    throw ed_error(error);
+  }
+  mrb->exc = nullptr;
+  throw ed_error("Ruby Exception: " + error);
+}
+} // namespace bed::internal::scripting

src/internal/syntax/iterator.cc

@@ -2,19 +2,11 @@
 
 namespace bed::internal::syntax {
 Iterator::Iterator(uint64_t target, ParserSnapshot p, vase::Shard *vase)
-    : snap(p) {
+    : snap(p), at(target) {
-  uint64_t offset;
+  at = target;
-  TreeCursor c = TreeCursor(*p.lang, p.root, target, &offset);
+  if (snap.lang)
-  at = target - offset;
+    state = ParseState::state_before(*snap.lang, snap.root, vase, at);
-  state = p.lang->copy(c.leaf->state);
   it = vase::Iterator(vase, at, Direction::Forward);
-  while (at < target) {
-    it->next();
-    tokens.clear();
-    events.clear();
-    p.lang->parse(&state, it->line, at == 0, &tokens, &events);
-    at++;
-  }
 }
 
 Iterator::~Iterator() {
@@ -46,6 +38,8 @@ Iterator &Iterator::operator=(Iterator &&other) {
 }
 
 void Iterator::next() {
+  if (!state)
+    return;
   it->next();
   tokens.clear();
   events.clear();

src/internal/syntax/miniparser.cc

@@ -0,0 +1,120 @@
+#include "internal/syntax/miniparser.h"
+
+namespace bed::internal::syntax {
+MiniParser::MiniParser(Language &lang, vase::Shard *vase, void *initial_state)
+    : lang(lang),
+      start_state(initial_state ? lang.copy(initial_state) : lang.none_state()) {
+  vase::Iterator it(vase, 0, Direction::Forward);
+  void *state = lang.copy(start_state);
+  std::vector<io::Token> tokens;
+  std::vector<ParseEvent> events;
+  const uint64_t count = vase ? vase->lines + 1 : 1;
+  lines.reserve(count);
+  for (uint64_t i = 0; i < count; ++i) {
+    it.next();
+    tokens.clear();
+    events.clear();
+    lang.parse(&state, it.line, i == 0, &tokens, &events);
+    lines.emplace_back(lang.copy(state), std::move(tokens));
+  }
+  lang.destroy(state);
+}
+
+MiniParser::~MiniParser() {
+  if (start_state)
+    lang.destroy(start_state);
+  for (auto &[state, tokens] : lines)
+    if (state)
+      lang.destroy(state);
+}
+
+void MiniParser::dirty(vase::Shard *vase, uint64_t start, uint64_t count) {
+  vase::Iterator it(vase, start, Direction::Forward);
+  void *state = start ? lang.copy(lines[start - 1].first)
+                      : lang.copy(start_state);
+  std::vector<ParseEvent> events;
+  uint64_t i = start;
+  for (; i < start + count; i++) {
+    it.next();
+    auto &line = lines[i];
+    line.second.clear();
+    events.clear();
+    lang.parse(&state, it.line, i == 0, &line.second, &events);
+    lang.destroy(line.first);
+    line.first = lang.copy(state);
+  }
+  while (it.next()) {
+    auto &line = lines[i++];
+    line.second.clear();
+    events.clear();
+    lang.parse(&state, it.line, false, &line.second, &events);
+    if (lang.equal(line.first, state))
+      break;
+    lang.destroy(line.first);
+    line.first = lang.copy(state);
+  }
+  lang.destroy(state);
+}
+
+void MiniParser::insert(vase::Shard *vase, uint64_t start, uint64_t count) {
+  vase::Iterator it(vase, start, Direction::Forward);
+  void *state = start ? lang.copy(lines[start - 1].first)
+                      : lang.copy(start_state);
+  std::vector<ParseEvent> events;
+  std::vector<std::pair<void *, std::vector<io::Token>>> lines_new;
+  std::vector<io::Token> tokens;
+  lines_new.reserve(count);
+  for (uint64_t i = 0; i < count; ++i) {
+    it.next();
+    tokens.clear();
+    events.clear();
+    lang.parse(&state, it.line, start + i == 0, &tokens, &events);
+    lines_new.emplace_back(lang.copy(state), std::move(tokens));
+  }
+  uint64_t i = start;
+  while (it.next()) {
+    auto &line = lines[i++];
+    line.second.clear();
+    events.clear();
+    lang.parse(&state, it.line, i == 0, &line.second, &events);
+    if (lang.equal(line.first, state))
+      break;
+    lang.destroy(line.first);
+    line.first = lang.copy(state);
+  }
+  lines.insert(
+    lines.begin() + start,
+    std::make_move_iterator(lines_new.begin()),
+    std::make_move_iterator(lines_new.end())
+  );
+  lang.destroy(state);
+}
+
+void MiniParser::erase(vase::Shard *vase, uint64_t start, uint64_t count) {
+  if (count == 0)
+    return;
+  void *state = start ? lang.copy(lines[start - 1].first)
+                      : lang.copy(start_state);
+  std::vector<ParseEvent> events;
+  for (uint64_t i = start; i < start + count; ++i)
+    if (lines[i].first)
+      lang.destroy(lines[i].first);
+  lines.erase(
+    lines.begin() + start,
+    lines.begin() + start + count
+  );
+  vase::Iterator it(vase, start, Direction::Forward);
+  uint64_t i = start;
+  while (i < lines.size() && it.next()) {
+    auto &line = lines[i];
+    line.second.clear();
+    events.clear();
+    lang.parse(&state, it.line, i++ == 0, &line.second, &events);
+    if (lang.equal(line.first, state))
+      break;
+    lang.destroy(line.first);
+    line.first = lang.copy(state);
+  }
+  lang.destroy(state);
+}
+} // namespace bed::internal::syntax

src/internal/syntax/parser.cc

@@ -22,6 +22,12 @@ void release(ParserSnapshot &snap) {
   snap.root = nullptr;
 }
 
+void *state_before(const ParserSnapshot &snap, vase::Shard *vase, uint64_t line) {
+  if (!snap.lang)
+    return nullptr;
+  return ParseState::state_before(*snap.lang, snap.root, vase, line);
+}
+
 uint64_t next_closing(const ParserSnapshot &snap, uint64_t line) {
   if (!snap.root || !snap.lang)
     return line + 10;
@@ -50,7 +56,7 @@ uint64_t next_closing(const ParserSnapshot &snap, uint64_t line) {
     c.next();
     first_leaf = false;
   }
-  return (snap.root->lines() - line > 10 ? line + 10 : snap.root->lines());
+  return (snap.root->lines() - line > 10 ? line + 10 : snap.root->lines() - 1);
 }
 
 uint64_t prev_opening(const ParserSnapshot &snap, uint64_t line) {

src/internal/syntax/parser_state.cc

@@ -106,9 +106,9 @@ ParseState *ParseState::splice(
   if (!root || (line == 0 && root->lines() == original)) {
     vase::Iterator it(vase, 0, Direction::Forward);
     void *state = lang.none_state();
-    std::vector<Token> tokens;
+    std::vector<io::Token> tokens;
     std::vector<ParseEvent> events;
-    ParsePieceBuilder builder(lang, 0);
+    ParsePieceBuilder builder(lang, 0, state);
     for (uint64_t at = 0; at < final; ++at) {
       it.next();
       tokens.clear();
@@ -130,13 +130,13 @@ ParseState *ParseState::splice(
     state = lang.copy(c.leaf->state);
   }
   vase::Iterator it(vase, at, Direction::Forward);
-  std::vector<Token> tokens;
+  std::vector<io::Token> tokens;
   std::vector<ParseEvent> events;
   uint64_t end_in_tree = line + original;
   uint64_t end_extra;
   TreeCursor c = TreeCursor(lang, root, end_in_tree, &end_extra);
   uint64_t end_in_vase = line + final;
-  ParsePieceBuilder builder(lang, at);
+  ParsePieceBuilder builder(lang, at, state);
   while (at < end_in_vase + end_extra) {
     it.next();
     tokens.clear();
@@ -192,4 +192,28 @@ ParseState *ParseState::concat(Language &lang, ParseState *a, ParseState *b) {
   }
   return balance(lang, new ParseStateBranch(a, b));
 }
+
+void *ParseState::state_before(Language &lang, ParseState *root, vase::Shard *vase, uint64_t line) {
+  uint64_t offset;
+  uint64_t at;
+  void *state;
+  if (!root) {
+    at = 0;
+    state = lang.none_state();
+  } else {
+    TreeCursor c = TreeCursor(lang, root, line, &offset);
+    at = line - offset;
+    state = lang.copy(c.leaf->state);
+  }
+  vase::Iterator it(vase, at, Direction::Forward);
+  std::vector<io::Token> tokens;
+  std::vector<ParseEvent> events;
+  for (; at < line; ++at) {
+    it.next();
+    tokens.clear();
+    events.clear();
+    lang.parse(&state, it.line, at == 0, &tokens, &events);
+  }
+  return state;
+}
 } // namespace bed::internal::syntax

src/internal/syntax/ruby/parse.cc

@@ -34,75 +34,87 @@ inline uint8_t utf8_codepoint_width(unsigned char c) {
   return 1;
 }
 
-bool handle_escapes(RubyParser &p, std::vector<Token> *tokens, uint32_t &start, bool string = true) {
+bool handle_escapes(RubyParser &p, std::vector<io::Token> *tokens, uint32_t &start, bool string = true) {
-  if (p.peek() == '\\') {
+  if (p.peek() != '\\')
-    if (string)
+    return false;
-      tokens->push_back({start, p.i, Token::String});
+  if (!(p.current().flags & RubyState::RubyInternalState::ALLOW_ESCAPE)) {
-    else
-      tokens->push_back({start, p.i, Token::Regexp});
-    start = p.i;
     p.advance();
-    if (p.peek() == 'x') {
+    if (p.peek() != '\'' && p.peek() != '\\')
-      p.advance();
+      return false;
-      if (is_hex(p.peek()))
+    if (string)
-        p.advance();
+      tokens->push_back({start, p.i - 1, io::Token::String});
-      if (is_hex(p.peek()))
+    else
-        p.advance();
+      tokens->push_back({start, p.i - 1, io::Token::Regexp});
-    } else if (p.peek() == 'u') {
+    p.advance();
-      p.advance();
+    tokens->push_back({p.i - 2, p.i, io::Token::Escape});
-      if (p.peek() == '{') {
-        p.advance();
-        while (p.peek() != '}' && p.peek() != '\0')
-          p.advance();
-        if (p.peek() == '}')
-          p.advance();
-      } else {
-        if (is_hex(p.peek()))
-          p.advance();
-        if (is_hex(p.peek()))
-          p.advance();
-        if (is_hex(p.peek()))
-          p.advance();
-        if (is_hex(p.peek()))
-          p.advance();
-      }
-    } else if ('0' <= p.peek() && p.peek() <= '7') {
-      p.advance();
-      if ('0' <= p.peek() && p.peek() <= '7')
-        p.advance();
-      if ('0' <= p.peek() && p.peek() <= '7')
-        p.advance();
-    } else if (p.peek() == 'c') {
-      p.advance();
-      if (p.peek() != '\\')
-        p.advance();
-    } else if (p.peek() == 'M' || p.peek() == 'C') {
-      p.advance();
-      if (p.peek() == '-') {
-        p.advance();
-        if (p.peek() != '\\')
-          p.advance();
-      }
-    } else if (p.peek() == 'N') {
-      p.advance();
-      if (p.peek() == '{') {
-        p.advance();
-        while (p.peek() != '}' && p.peek() != '\0')
-          p.advance();
-        if (p.peek() == '}')
-          p.advance();
-      }
-    } else {
-      p.advance();
-    }
-    tokens->push_back({start, p.i, Token::Escape});
     start = p.i;
     return true;
   }
-  return false;
+  if (string)
+    tokens->push_back({start, p.i, io::Token::String});
+  else
+    tokens->push_back({start, p.i, io::Token::Regexp});
+  start = p.i;
+  p.advance();
+  if (p.peek() == 'x') {
+    p.advance();
+    if (is_hex(p.peek()))
+      p.advance();
+    if (is_hex(p.peek()))
+      p.advance();
+  } else if (p.peek() == 'u') {
+    p.advance();
+    if (p.peek() == '{') {
+      p.advance();
+      while (p.peek() != '}' && p.peek() != '\0')
+        p.advance();
+      if (p.peek() == '}')
+        p.advance();
+    } else {
+      if (is_hex(p.peek()))
+        p.advance();
+      if (is_hex(p.peek()))
+        p.advance();
+      if (is_hex(p.peek()))
+        p.advance();
+      if (is_hex(p.peek()))
+        p.advance();
+    }
+  } else if ('0' <= p.peek() && p.peek() <= '7') {
+    p.advance();
+    if ('0' <= p.peek() && p.peek() <= '7')
+      p.advance();
+    if ('0' <= p.peek() && p.peek() <= '7')
+      p.advance();
+  } else if (p.peek() == 'c') {
+    p.advance();
+    if (p.peek() != '\\')
+      p.advance();
+  } else if (p.peek() == 'M' || p.peek() == 'C') {
+    p.advance();
+    if (p.peek() == '-') {
+      p.advance();
+      if (p.peek() != '\\')
+        p.advance();
+    }
+  } else if (p.peek() == 'N') {
+    p.advance();
+    if (p.peek() == '{') {
+      p.advance();
+      while (p.peek() != '}' && p.peek() != '\0')
+        p.advance();
+      if (p.peek() == '}')
+        p.advance();
+    }
+  } else {
+    p.advance();
+  }
+  tokens->push_back({start, p.i, io::Token::Escape});
+  start = p.i;
+  return true;
 };
 
-bool handle_heredoc(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseEvent> *events) {
+bool handle_heredoc(RubyParser &p, std::vector<io::Token> *tokens, std::vector<ParseEvent> *events) {
   uint8_t *heredocs = p.state->heredocs();
   uint32_t start = p.i;
   if (start == 0) {
@@ -115,21 +127,21 @@ bool handle_heredoc(RubyParser &p, std::vector<Token> *tokens, std::vector<Parse
       if (!p.dequeue_doc(heredoc_len))
         p.current().state = RubyState::RubyInternalState::NONE;
       events->push_back(true);
-      tokens->push_back({p.i, p.len(), Token::Annotation});
+      tokens->push_back({p.i, p.len(), io::Token::Annotation});
       p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
       return true;
     }
   }
   if (!(heredocs[0] & RubyState::Heredocs::ALLOW_INTERPOLATION)) {
-    tokens->push_back({p.i, p.len(), Token::String});
+    tokens->push_back({p.i, p.len(), io::Token::String});
     return true;
   } else {
     while (p.i < p.len()) {
       if (handle_escapes(p, tokens, start))
         continue;
       if (p.peek_str(2) == "#{") {
-        tokens->push_back({start, p.i, Token::String});
+        tokens->push_back({start, p.i, io::Token::String});
-        tokens->push_back({p.i, p.i + 2, Token::Interpolation});
+        tokens->push_back({p.i, p.i + 2, io::Token::Interpolation});
         p.advance(2);
         p.push_state();
         return false;
@@ -137,20 +149,20 @@ bool handle_heredoc(RubyParser &p, std::vector<Token> *tokens, std::vector<Parse
       p.advance();
     }
     if (p.i >= p.len())
-      tokens->push_back({start, p.len(), Token::String});
+      tokens->push_back({start, p.len(), io::Token::String});
     return true;
   }
 }
 
-void handle_string(RubyParser &p, std::vector<Token> *tokens) {
+void handle_string(RubyParser &p, std::vector<io::Token> *tokens) {
   uint32_t start = p.i;
   while (p.i < p.len()) {
     if (handle_escapes(p, tokens, start))
       continue;
     if ((p.current().flags & RubyState::RubyInternalState::ALLOW_INTERPOLATION)
         && p.peek_str(2) == "#{") {
-      tokens->push_back({start, p.i, Token::String});
+      tokens->push_back({start, p.i, io::Token::String});
-      tokens->push_back({p.i, p.i + 2, Token::Interpolation});
+      tokens->push_back({p.i, p.i + 2, io::Token::Interpolation});
       p.advance(2);
       p.push_state();
       return;
@@ -161,7 +173,7 @@ void handle_string(RubyParser &p, std::vector<Token> *tokens) {
     if (p.peek() == p.current().delim_end) {
       if (p.current().delim_start == p.current().delim_end) {
         p.advance();
-        tokens->push_back({start, p.i, Token::String});
+        tokens->push_back({start, p.i, io::Token::String});
         p.current().state = RubyState::RubyInternalState::NONE;
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
         return;
@@ -169,7 +181,7 @@ void handle_string(RubyParser &p, std::vector<Token> *tokens) {
         p.current().lit_brace_level--;
         if (p.current().lit_brace_level == 0) {
           p.advance();
-          tokens->push_back({start, p.i, Token::String});
+          tokens->push_back({start, p.i, io::Token::String});
           p.current().state = RubyState::RubyInternalState::NONE;
           p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
           return;
@@ -179,18 +191,18 @@ void handle_string(RubyParser &p, std::vector<Token> *tokens) {
     p.advance();
   }
   if (p.i >= p.len())
-    tokens->push_back({start, p.len(), Token::String});
+    tokens->push_back({start, p.len(), io::Token::String});
 }
 
-void handle_regex(RubyParser &p, std::vector<Token> *tokens) {
+void handle_regex(RubyParser &p, std::vector<io::Token> *tokens) {
   uint32_t start = p.i;
   while (p.i < p.len()) {
     if (handle_escapes(p, tokens, start, false))
       continue;
     if ((p.current().flags & RubyState::RubyInternalState::ALLOW_INTERPOLATION)
         && p.peek_str(2) == "#{") {
-      tokens->push_back({start, p.i, Token::Regexp});
+      tokens->push_back({start, p.i, io::Token::Regexp});
-      tokens->push_back({p.i, p.i + 2, Token::Interpolation});
+      tokens->push_back({p.i, p.i + 2, io::Token::Interpolation});
       p.advance(2);
       p.push_state();
       return;
@@ -201,7 +213,7 @@ void handle_regex(RubyParser &p, std::vector<Token> *tokens) {
     if (p.peek() == p.current().delim_end) {
       if (p.current().delim_start == p.current().delim_end) {
         p.advance();
-        tokens->push_back({start, p.i, Token::Regexp});
+        tokens->push_back({start, p.i, io::Token::Regexp});
         p.current().state = RubyState::RubyInternalState::NONE;
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
         return;
@@ -209,7 +221,7 @@ void handle_regex(RubyParser &p, std::vector<Token> *tokens) {
         p.current().lit_brace_level--;
         if (p.current().lit_brace_level == 0) {
           p.advance();
-          tokens->push_back({start, p.i, Token::Regexp});
+          tokens->push_back({start, p.i, io::Token::Regexp});
           p.current().state = RubyState::RubyInternalState::NONE;
           p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
           return;
@@ -219,15 +231,15 @@ void handle_regex(RubyParser &p, std::vector<Token> *tokens) {
     p.advance();
   }
   if (p.i >= p.len())
-    tokens->push_back({start, p.len(), Token::Regexp});
+    tokens->push_back({start, p.len(), io::Token::Regexp});
 }
 
-bool handle_line_markers(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseEvent> *events) {
+bool handle_line_markers(RubyParser &p, std::vector<io::Token> *tokens, std::vector<ParseEvent> *events) {
   if (p.len() == 6 && p.peek_str(6) == "=begin") {
     p.current().state = RubyState::RubyInternalState::COMMENT;
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     events->push_back(false);
-    tokens->push_back({0, p.len(), Token::Comment});
+    tokens->push_back({0, p.len(), io::Token::Comment});
     return true;
   }
   if (p.len() == 7 && p.peek_str(7) == "__END__") {
@@ -238,20 +250,20 @@ bool handle_line_markers(RubyParser &p, std::vector<Token> *tokens, std::vector<
   return false;
 }
 
-bool handle_comment(RubyParser &p, std::vector<Token> *tokens, bool first_line) {
+bool handle_comment(RubyParser &p, std::vector<io::Token> *tokens, bool first_line) {
   if (p.peek() == '#') {
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     if (first_line && p.i == 0 && p.peek(1) == '!') {
-      tokens->push_back({0, p.len(), Token::Shebang});
+      tokens->push_back({0, p.len(), io::Token::Shebang});
       return true;
     }
-    tokens->push_back({p.i, p.len(), Token::Comment});
+    tokens->push_back({p.i, p.len(), io::Token::Comment});
     return true;
   }
   return false;
 }
 
-bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseEvent> *events) {
+bool handle_syntax(RubyParser &p, std::vector<io::Token> *tokens, std::vector<ParseEvent> *events) {
   static const RubyTries tries = RubyTries();
   if (p.peek() == ' ' || p.peek() == '\t') {
     while (p.peek() == ' ' || p.peek() == '\t')
@@ -277,7 +289,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
         j++;
         while (identifier_char(p.peek(j)))
           j++;
-        tokens->push_back({p.i, p.i + j, Token::Constant});
+        tokens->push_back({p.i, p.i + j, io::Token::Constant});
         p.advance(j);
         while (p.peek() == ' ' || p.peek() == '\t')
           p.advance();
@@ -299,11 +311,11 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
         if (p.peek(j) == '!' || p.peek(j) == '?')
           j++;
         if ('A' <= p.peek() && p.peek() <= 'Z')
-          tokens->push_back({p.i, p.i + j, Token::Constant});
+          tokens->push_back({p.i, p.i + j, io::Token::Constant});
         else if (j == 4 && p.peek_str(4) == "self")
-          tokens->push_back({p.i, p.i + j, Token::Keyword});
+          tokens->push_back({p.i, p.i + j, io::Token::Keyword});
         else
-          tokens->push_back({p.i, p.i + j, Token::Function});
+          tokens->push_back({p.i, p.i + j, io::Token::Function});
         p.advance(j);
         while (p.peek() == ' ' || p.peek() == '\t')
           p.advance();
@@ -322,7 +334,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
         j++;
         while (identifier_char(p.peek(j)))
           j++;
-        tokens->push_back({p.i, p.i + j, Token::Constant});
+        tokens->push_back({p.i, p.i + j, io::Token::Constant});
         p.advance(j);
         while (p.peek() == ' ' || p.peek() == '\t')
           p.advance();
@@ -344,7 +356,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
       indented = true;
     if (p.peek(j) == '~' || p.peek(j) == '-')
       j++;
-    tokens->push_back({p.i, p.i + j, Token::Operator});
+    tokens->push_back({p.i, p.i + j, io::Token::Operator});
     if (p.i + j >= p.len())
       return false;
     std::string delim;
@@ -368,7 +380,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     if (!delim.empty()) {
       events->push_back(false);
-      tokens->push_back({s, p.i + j, Token::Annotation});
+      tokens->push_back({s, p.i + j, io::Token::Annotation});
       uint8_t header = delim.size();
       if (interpolation)
         header |= RubyState::Heredocs::ALLOW_INTERPOLATION;
@@ -382,7 +394,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     return false;
   }
   if (p.peek() == '/' && p.current().flags & RubyState::RubyInternalState::EXPECTING_EXPRESSION) {
-    tokens->push_back({p.i, p.i + 1, Token::Regexp});
+    tokens->push_back({p.i, p.i + 1, io::Token::Regexp});
     p.current().state = RubyState::RubyInternalState::REGEXP;
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     p.current().flags |= RubyState::RubyInternalState::ALLOW_INTERPOLATION;
@@ -401,7 +413,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
       if (p.peek() == '.')
         p.advance();
     }
-    tokens->push_back({start, p.i, Token::Operator});
+    tokens->push_back({start, p.i, io::Token::Operator});
     return false;
   }
   case ':': {
@@ -409,17 +421,17 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     uint32_t start = p.i;
     p.advance();
     if (p.i >= p.len()) {
-      tokens->push_back({start, p.i, Token::Operator});
+      tokens->push_back({start, p.i, io::Token::Operator});
       p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
       return false;
     }
     if (p.peek() == ':') {
       p.advance();
-      tokens->push_back({start, p.i, Token::Operator});
+      tokens->push_back({start, p.i, io::Token::Operator});
       return false;
     }
     if (p.peek() == '\'' || p.peek() == '"') {
-      tokens->push_back({start, p.i, Token::Label});
+      tokens->push_back({start, p.i, io::Token::Label});
       p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
       return false;
     }
@@ -430,12 +442,12 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
         p.advance();
       while (identifier_char(p.peek()))
         p.advance();
-      tokens->push_back({start, p.i, Token::Label});
+      tokens->push_back({start, p.i, io::Token::Label});
       return false;
     }
     uint32_t op_len = tries.operator_trie.longest_match(p.peek_str(p.len() - p.i));
     if (op_len > 0) {
-      tokens->push_back({start, p.i + op_len, Token::Label});
+      tokens->push_back({start, p.i + op_len, io::Token::Label});
       p.advance(op_len);
       return false;
     }
@@ -445,10 +457,10 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
         p.advance();
       if (p.peek() == '!' || p.peek() == '?')
         p.advance();
-      tokens->push_back({start, p.i, Token::Label});
+      tokens->push_back({start, p.i, io::Token::Label});
       return false;
     }
-    tokens->push_back({start, p.i, Token::Operator});
+    tokens->push_back({start, p.i, io::Token::Operator});
     return false;
   }
   case '@': {
@@ -465,7 +477,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
       return false;
     while (identifier_char(p.peek()))
       p.advance();
-    tokens->push_back({start, p.i, Token::VariableInstance});
+    tokens->push_back({start, p.i, io::Token::VariableInstance});
     return false;
   }
   case '$': {
@@ -512,7 +524,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
         return false;
       }
     }
-    tokens->push_back({start, p.i, Token::VariableGlobal});
+    tokens->push_back({start, p.i, io::Token::VariableGlobal});
     return false;
   }
   case '?': {
@@ -528,7 +540,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           p.advance();
         if (is_hex(p.peek()))
           p.advance();
-        tokens->push_back({start, p.i, Token::Char});
+        tokens->push_back({start, p.i, io::Token::Char});
         return false;
       } else if (p.peek() == 'u') {
         p.advance();
@@ -548,7 +560,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           if (is_hex(p.peek()))
             p.advance();
         }
-        tokens->push_back({start, p.i, Token::Char});
+        tokens->push_back({start, p.i, io::Token::Char});
         return false;
       } else if ('0' <= p.peek() && p.peek() <= '7') {
         p.advance();
@@ -556,7 +568,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           p.advance();
         if ('0' <= p.peek() && p.peek() <= '7')
           p.advance();
-        tokens->push_back({start, p.i, Token::Char});
+        tokens->push_back({start, p.i, io::Token::Char});
         return false;
       } else if (p.peek() == 'c') {
         p.advance();
@@ -564,7 +576,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           p.advance();
         else
           goto combination;
-        tokens->push_back({start, p.i, Token::Char});
+        tokens->push_back({start, p.i, io::Token::Char});
         return false;
       } else if (p.peek() == 'M' || p.peek() == 'C') {
         p.advance();
@@ -575,7 +587,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           else
             goto combination;
         }
-        tokens->push_back({start, p.i, Token::Char});
+        tokens->push_back({start, p.i, io::Token::Char});
         return false;
       } else if (p.peek() == 'N') {
         p.advance();
@@ -586,28 +598,28 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           if (p.peek() == '}')
             p.advance();
         }
-        tokens->push_back({start, p.i, Token::Char});
+        tokens->push_back({start, p.i, io::Token::Char});
         return false;
       } else {
         p.advance();
-        tokens->push_back({start, p.i, Token::Char});
+        tokens->push_back({start, p.i, io::Token::Char});
         return false;
       }
     } else if (p.peek() != '\0' && p.peek() != ' ' && p.peek() != '\t') {
       p.advance();
-      tokens->push_back({start, p.i, Token::Char});
+      tokens->push_back({start, p.i, io::Token::Char});
       return false;
     } else {
       p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-      tokens->push_back({start, p.i, Token::Operator});
+      tokens->push_back({start, p.i, io::Token::Operator});
       return false;
     }
   }
   case '{': {
     p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     uint8_t brace_color =
-      (uint8_t)Token::Brace1 + (p.current().brace_level % 5);
+      (uint8_t)io::Token::Brace1 + (p.current().brace_level % 5);
-    tokens->push_back({p.i, p.i + 1, (Token::Kind)brace_color});
+    tokens->push_back({p.i, p.i + 1, (io::Token::Kind)brace_color});
     p.current().brace_level++;
     p.advance();
     return false;
@@ -616,11 +628,11 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     if (!--p.current().brace_level && p.state->top > 1) {
       p.pop_state();
-      tokens->push_back({p.i, p.i + 1, Token::Interpolation});
+      tokens->push_back({p.i, p.i + 1, io::Token::Interpolation});
     } else {
       uint8_t brace_color =
-        (uint8_t)Token::Brace1 + (p.current().brace_level % 5);
+        (uint8_t)io::Token::Brace1 + (p.current().brace_level % 5);
-      tokens->push_back({p.i, p.i + 1, (Token::Kind)brace_color});
+      tokens->push_back({p.i, p.i + 1, (io::Token::Kind)brace_color});
     }
     p.advance();
     return false;
@@ -628,8 +640,8 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
   case '(': {
     p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     uint8_t brace_color =
-      (uint8_t)Token::Brace1 + (p.current().brace_level % 5);
+      (uint8_t)io::Token::Brace1 + (p.current().brace_level % 5);
-    tokens->push_back({p.i, p.i + 1, (Token::Kind)brace_color});
+    tokens->push_back({p.i, p.i + 1, (io::Token::Kind)brace_color});
     p.current().brace_level++;
     p.advance();
     return false;
@@ -638,16 +650,16 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     p.current().brace_level--;
     uint8_t brace_color =
-      (uint8_t)Token::Brace1 + (p.current().brace_level % 5);
+      (uint8_t)io::Token::Brace1 + (p.current().brace_level % 5);
-    tokens->push_back({p.i, p.i + 1, (Token::Kind)brace_color});
+    tokens->push_back({p.i, p.i + 1, (io::Token::Kind)brace_color});
     p.advance();
     return false;
   }
   case '[': {
     p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     uint8_t brace_color =
-      (uint8_t)Token::Brace1 + (p.current().brace_level % 5);
+      (uint8_t)io::Token::Brace1 + (p.current().brace_level % 5);
-    tokens->push_back({p.i, p.i + 1, (Token::Kind)brace_color});
+    tokens->push_back({p.i, p.i + 1, (io::Token::Kind)brace_color});
     p.current().brace_level++;
     p.advance();
     return false;
@@ -656,34 +668,36 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
     p.current().brace_level--;
     uint8_t brace_color =
-      (uint8_t)Token::Brace1 + (p.current().brace_level % 5);
+      (uint8_t)io::Token::Brace1 + (p.current().brace_level % 5);
-    tokens->push_back({p.i, p.i + 1, (Token::Kind)brace_color});
+    tokens->push_back({p.i, p.i + 1, (io::Token::Kind)brace_color});
     p.advance();
     return false;
   }
   case '\'': {
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-    tokens->push_back({p.i, p.i + 1, Token::String});
+    tokens->push_back({p.i, p.i + 1, io::Token::String});
     p.current().state = RubyState::RubyInternalState::STRING;
     p.current().delim_start = '\'';
     p.current().delim_end = '\'';
     p.current().flags &= ~RubyState::RubyInternalState::ALLOW_INTERPOLATION;
+    p.current().flags &= ~RubyState::RubyInternalState::ALLOW_ESCAPE;
     p.advance();
     return false;
   }
   case '"': {
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-    tokens->push_back({p.i, p.i + 1, Token::String});
+    tokens->push_back({p.i, p.i + 1, io::Token::String});
     p.current().state = RubyState::RubyInternalState::STRING;
     p.current().delim_start = '"';
     p.current().delim_end = '"';
     p.current().flags |= RubyState::RubyInternalState::ALLOW_INTERPOLATION;
+    p.current().flags |= RubyState::RubyInternalState::ALLOW_ESCAPE;
     p.advance();
     return false;
   }
   case '`': {
     p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-    tokens->push_back({p.i, p.i + 1, Token::String});
+    tokens->push_back({p.i, p.i + 1, io::Token::String});
     p.current().state = RubyState::RubyInternalState::STRING;
     p.current().delim_start = '`';
     p.current().delim_end = '`';
@@ -693,7 +707,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
   }
   case '%': {
     if (p.i + 1 >= p.len()) {
-      tokens->push_back({p.i, p.i + 1, Token::Operator});
+      tokens->push_back({p.i, p.i + 1, io::Token::Operator});
       p.advance();
       p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
       return false;
@@ -703,6 +717,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     char delim_start = '\0';
     char delim_end = '\0';
     bool allow_interp = true;
+    bool allow_escape = true;
     int prefix_len = 1;
     bool is_regexp = false;
     switch (type) {
@@ -716,6 +731,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     case 'I':
     case 'W':
       allow_interp = true;
+      allow_escape = true;
       prefix_len = 2;
       break;
     case 'w':
@@ -723,22 +739,24 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     case 'i':
     case 's':
       allow_interp = false;
+      allow_escape = false;
       prefix_len = 2;
       break;
     default:
       allow_interp = true;
+      allow_escape = true;
       prefix_len = 1;
       break;
     }
     if (p.i + prefix_len >= p.len()) {
-      tokens->push_back({p.i, p.i + 1, Token::Operator});
+      tokens->push_back({p.i, p.i + 1, io::Token::Operator});
       p.advance(prefix_len);
       p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
       return false;
     }
     delim_start = p.peek(prefix_len);
     if (identifier_char(delim_start) || delim_start == ' ') {
-      tokens->push_back({p.i, p.i + 1, Token::Operator});
+      tokens->push_back({p.i, p.i + 1, io::Token::Operator});
       p.advance(prefix_len);
       p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
       return false;
@@ -760,7 +778,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
       delim_end = delim_start;
       break;
     }
-    tokens->push_back({p.i, p.i + prefix_len + 1, (is_regexp ? Token::Regexp : Token::String)});
+    tokens->push_back({p.i, p.i + prefix_len + 1, (is_regexp ? io::Token::Regexp : io::Token::String)});
     p.current().state = is_regexp
                           ? RubyState::RubyInternalState::REGEXP
                           : RubyState::RubyInternalState::STRING;
@@ -768,6 +786,8 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
     p.current().delim_end = delim_end;
     if (allow_interp)
       p.current().flags |= RubyState::RubyInternalState::ALLOW_INTERPOLATION;
+    if (allow_escape)
+      p.current().flags |= RubyState::RubyInternalState::ALLOW_ESCAPE;
     p.current().lit_brace_level = 1;
     p.advance(prefix_len + 1);
     return false;
@@ -782,11 +802,6 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
       p.ending = false;
     p.advance();
     return false;
-  case '\0':
-    if (p.ending)
-      p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-    p.advance();
-    return false;
   default:
     if ('0' <= p.peek() && p.peek() <= '9') {
       p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
@@ -862,7 +877,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           }
         }
       }
-      tokens->push_back({start, p.i, Token::Number});
+      tokens->push_back({start, p.i, io::Token::Number});
       return false;
     } else if (identifier_start_char(p.peek())) {
       uint32_t j = 1;
@@ -872,93 +887,93 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
         j++;
       if (j == tries.base_keywords_trie.longest_match(p.peek_str(j))) {
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::Keyword});
+        tokens->push_back({p.i, p.i + j, io::Token::Keyword});
         p.advance(j);
         return false;
       } else if (j == tries.expecting_keywords_trie.longest_match(p.peek_str(j))) {
         p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::Keyword});
+        tokens->push_back({p.i, p.i + j, io::Token::Keyword});
         p.advance(j);
         return false;
       } else if (j == tries.operator_keywords_trie.longest_match(p.peek_str(j))) {
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::KeywordOperator});
+        tokens->push_back({p.i, p.i + j, io::Token::KeywordOperator});
         p.advance(j);
         return false;
       } else if (j == tries.expecting_operators_trie.longest_match(p.peek_str(j))) {
         p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::KeywordOperator});
+        tokens->push_back({p.i, p.i + j, io::Token::KeywordOperator});
         p.advance(j);
         return false;
       } else if (j == tries.types_trie.longest_match(p.peek_str(j))) {
         p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::Type});
+        tokens->push_back({p.i, p.i + j, io::Token::Type});
         p.advance(j);
         return false;
       } else if (j == tries.methods_trie.longest_match(p.peek_str(j))) {
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::Function});
+        tokens->push_back({p.i, p.i + j, io::Token::Function});
         p.advance(j);
         return false;
       } else if (j == tries.expecting_end_keywords_trie.longest_match(p.peek_str(j))) {
         events->push_back(false);
         p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::Keyword});
+        tokens->push_back({p.i, p.i + j, io::Token::Keyword});
         p.advance(j);
         return false;
       } else if (j == tries.conditional_keywords_trie.longest_match(p.peek_str(j))) {
         if (p.current().flags & RubyState::RubyInternalState::NEWLINE || p.op_last)
           events->push_back(false);
         p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::Keyword});
+        tokens->push_back({p.i, p.i + j, io::Token::Keyword});
         p.advance(j);
         return false;
       } else if (j == tries.end_keywords_trie.longest_match(p.peek_str(j))) {
         p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
         events->push_back(false);
-        tokens->push_back({p.i, p.i + j, Token::Keyword});
+        tokens->push_back({p.i, p.i + j, io::Token::Keyword});
         p.advance(j);
         return false;
       } else if (j == tries.builtins_trie.longest_match(p.peek_str(j))) {
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::Constant});
+        tokens->push_back({p.i, p.i + j, io::Token::Constant});
         p.advance(j);
         return false;
       } else if (j == tries.errors_trie.longest_match(p.peek_str(j))) {
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
-        tokens->push_back({p.i, p.i + j, Token::Error});
+        tokens->push_back({p.i, p.i + j, io::Token::Error});
         p.advance(j);
         return false;
       } else if ('A' <= p.peek() && p.peek() <= 'Z' && !(p.peek(j) == '!' || p.peek(j) == '?')) {
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
         if (j >= 5 && p.peek_str(j).substr(j - 5) == "Error") {
-          tokens->push_back({p.i, p.i + j, Token::Error});
+          tokens->push_back({p.i, p.i + j, io::Token::Error});
           p.advance(j);
           return false;
         }
-        tokens->push_back({p.i, p.i + j, Token::Constant});
+        tokens->push_back({p.i, p.i + j, io::Token::Constant});
         p.advance(j);
         return false;
       } else {
         p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
         if (j == 4 && p.peek_str(4) == "true") {
-          tokens->push_back({p.i, p.i + j, Token::True});
+          tokens->push_back({p.i, p.i + j, io::Token::True});
           p.advance(4);
           return false;
         }
         if (j == 5 && p.peek_str(5) == "false") {
-          tokens->push_back({p.i, p.i + j, Token::False});
+          tokens->push_back({p.i, p.i + j, io::Token::False});
           p.advance(5);
           return false;
         }
         if (j == 3 && p.peek_str(3) == "end") {
           events->push_back(true);
-          tokens->push_back({p.i, p.i + j, Token::Keyword});
+          tokens->push_back({p.i, p.i + j, io::Token::Keyword});
           p.advance(3);
           return false;
         }
         if (j == 5 && p.peek_str(5) == "class") {
-          tokens->push_back({p.i, p.i + j, Token::Keyword});
+          tokens->push_back({p.i, p.i + j, io::Token::Keyword});
           p.advance(5);
           p.current().flags =
             (p.current().flags & ~RubyState::RubyInternalState::NAME_MASK)
@@ -966,7 +981,7 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           return false;
         }
         if (j == 6 && p.peek_str(6) == "module") {
-          tokens->push_back({p.i, p.i + j, Token::Keyword});
+          tokens->push_back({p.i, p.i + j, io::Token::Keyword});
           p.advance(6);
           p.current().flags =
             (p.current().flags & ~RubyState::RubyInternalState::NAME_MASK)
@@ -974,28 +989,33 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
           return false;
         }
         if (j == 3 && p.peek_str(3) == "def") {
-          tokens->push_back({p.i, p.i + j, Token::Keyword});
+          tokens->push_back({p.i, p.i + j, io::Token::Keyword});
           p.advance(3);
           p.current().flags =
             (p.current().flags & ~RubyState::RubyInternalState::NAME_MASK)
             | RubyState::RubyInternalState::DEF_NAME;
           return false;
         }
+        if (j > 3 && p.peek_str(3) == "to_") {
+          p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
+          tokens->push_back({p.i, p.i + j, io::Token::Function});
+          p.advance(j);
+        }
         uint32_t start = p.i;
         if (p.peek(j) == ':') {
           p.advance(j);
           p.advance();
-          tokens->push_back({start, p.i, Token::Label});
+          tokens->push_back({start, p.i, io::Token::Label});
           return false;
         } else if (p.peek(j - 1) == '!' || p.peek(j - 1) == '?') {
           p.advance(j);
-          tokens->push_back({start, p.i, Token::Function});
+          tokens->push_back({start, p.i, io::Token::Function});
           return false;
         } else {
           p.advance(j);
           j = 0;
           if (p.peek(j) == '(' || p.peek(j) == '{') {
-            tokens->push_back({start, p.i, Token::Function});
+            tokens->push_back({start, p.i, io::Token::Function});
             return false;
           } else if (p.peek(j) == ' ' || p.peek(j) == '\t') {
             j++;
@@ -1008,8 +1028,9 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
             return false;
           if (p.peek(j) == '&'
               || p.peek(j) == '%'
-              || p.peek(j) == ':') {
+              || p.peek(j) == ':'
-            if (p.peek(j + 1) == ' ' || p.peek(j + 1) == '>')
+              || p.peek(j) == '?') {
+            if (p.peek(j + 1) == ' ' || p.peek(j + 1) == '\0')
               return false;
           } else if (p.peek(j) == '-') {
             if (p.peek(j + 1) != '>')
@@ -1025,22 +1046,22 @@ bool handle_syntax(RubyParser &p, std::vector<Token> *tokens, std::vector<ParseE
             || p.peek(j) == '*'
             || p.peek(j) == '/'
             || p.peek(j) == '='
-            || p.peek(j) == '?'
             || p.peek(j) == '|'
             || p.peek(j) == '^'
             || p.peek(j) == '<'
             || p.peek(j) == '>'
+            || p.peek(j) == '#'
           ) {
             return false;
           }
-          tokens->push_back({start, p.i, Token::Function});
+          tokens->push_back({start, p.i, io::Token::Function});
           return false;
         }
       }
     } else {
       uint32_t op_len;
       if ((op_len = tries.operator_trie.longest_match(p.peek_str(p.len() - p.i)))) {
-        tokens->push_back({p.i, p.i + op_len, Token::Operator});
+        tokens->push_back({p.i, p.i + op_len, io::Token::Operator});
         p.advance(op_len);
         p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
         p.set_op_last = true;
@@ -1056,17 +1077,17 @@ void ruby_parse(
   void **v_state,
   std::string_view line,
   bool fl,
-  std::vector<Token> *tokens,
+  std::vector<io::Token> *tokens,
   std::vector<ParseEvent> *events
 ) {
   RubyParser p(v_state, line);
-  while (p.i <= p.len()) {
+  while (p.i < p.len()) {
     p.op_last = p.set_op_last;
     p.set_op_last = false;
     if (p.current().state == RubyState::RubyInternalState::END)
       return;
     if (p.current().state == RubyState::RubyInternalState::COMMENT) {
-      tokens->push_back({p.i, p.len(), Token::Comment});
+      tokens->push_back({p.i, p.len(), io::Token::Comment});
       if (p.i == 0 && p.peek_str(4) == "=end") {
         p.current().state = RubyState::RubyInternalState::NONE;
         events->push_back(true);
@@ -1098,6 +1119,13 @@ void ruby_parse(
       return;
     if (!handle_syntax(p, tokens, events))
       p.current().flags &= ~RubyState::RubyInternalState::NEWLINE;
+    if (p.peek() == '\0') {
+      if (p.ending)
+        p.current().flags |= RubyState::RubyInternalState::EXPECTING_EXPRESSION;
+      p.advance();
+      p.current().flags &= ~RubyState::RubyInternalState::NEWLINE;
+      break;
+    }
     continue;
   }
   if (p.ending)

src/internal/theme/theme.cc

@@ -5,150 +5,248 @@ Theme::Theme() {
   hl.fill({
     .fg = 0xF0F0F0,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   });
 }
 
-Highlight Theme::get(internal::syntax::Token token) const {
+io::Highlight Theme::get(const io::Token::Kind &token) const {
-  return hl[token.type];
+  return hl[token];
 }
 
 Theme Theme::default_theme() {
   Theme theme;
-  theme.hl[internal::syntax::Token::Shebang] = {
+
-    .fg = 0x7DCFFF,
+  constexpr uint32_t PINK = 0xECA1AE;
+  constexpr uint32_t GREY = 0xAFB7D3;
+  constexpr uint32_t BLUE = 0x799EDB;
+  constexpr uint32_t LIGHT_BLUE = 0xB5BFFE;
+  constexpr uint32_t RED = 0xF38CAA;
+  constexpr uint32_t YELLOW = 0xF9E3B1;
+  constexpr uint32_t PURPLE = 0xA286C7;
+  constexpr uint32_t CYAN = 0x89DBEA;
+  constexpr uint32_t LIGHT_GREEN = 0x98CE94;
+  constexpr uint32_t TURQUOISE = 0x00BDB7;
+  constexpr uint32_t BRIGHT_GREEN = 0x4AF6CA;
+
+  theme.hl[io::Token::TempCurrent] = {
+    .fg = PINK,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Error] = {
+  theme.hl[io::Token::BufferName] = {
-    .fg = 0xEF5168,
+    .fg = PURPLE,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Comment] = {
+  theme.hl[io::Token::AddressSeperator] = {
-    .fg = 0xAAAAAA,
+    .fg = GREY,
     .bg = 0x000000,
-    .flags = Highlight::Italic,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::String] = {
+  theme.hl[io::Token::AddressSymbol] = {
+    .fg = BLUE,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Mark] = {
+    .fg = LIGHT_BLUE,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::RubyFunction] = {
+    .fg = RED,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::RubyArg] = {
+    .fg = LIGHT_GREEN,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Shell] = {
+    .fg = BRIGHT_GREEN,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::File] = {
+    .fg = GREY,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Suffix] = {
+    .fg = TURQUOISE,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Color1] = {
+    .fg = 0x7AA2F7,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Color2] = {
     .fg = 0xAAD94C,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Escape] = {
+  theme.hl[io::Token::Color3] = {
+    .fg = 0xFF9E64,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Color4] = {
+    .fg = 0xBB9AF7,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Color5] = {
+    .fg = 0xFF757F,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Warning] = {
+    .fg = 0xF1C55A,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Data] = {
+    .fg = GREY,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Shebang] = {
     .fg = 0x7DCFFF,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Interpolation] = {
+  theme.hl[io::Token::Error] = {
-    .fg = 0x7DCFFF,
-    .bg = 0x000000,
-    .flags = Highlight::None,
-  };
-  theme.hl[internal::syntax::Token::Regexp] = {
-    .fg = 0xD2A6FF,
-    .bg = 0x000000,
-    .flags = Highlight::None,
-  };
-  theme.hl[internal::syntax::Token::Number] = {
-    .fg = 0xE6C08A,
-    .bg = 0x000000,
-    .flags = Highlight::None,
-  };
-  theme.hl[internal::syntax::Token::True] = {
-    .fg = 0x7AE93C,
-    .bg = 0x000000,
-    .flags = Highlight::None,
-  };
-  theme.hl[internal::syntax::Token::False] = {
     .fg = 0xEF5168,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Char] = {
+  theme.hl[io::Token::Comment] = {
+    .fg = 0xAAAAAA,
+    .bg = 0x000000,
+    .flags = io::Highlight::Italic,
+  };
+  theme.hl[io::Token::String] = {
+    .fg = 0xAAD94C,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Escape] = {
+    .fg = 0x7DCFFF,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Interpolation] = {
+    .fg = 0x7DCFFF,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Regexp] = {
+    .fg = 0x7DCFFF,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Number] = {
+    .fg = 0xE6C08A,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::True] = {
+    .fg = 0x7AE93C,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::False] = {
+    .fg = 0xEF5168,
+    .bg = 0x000000,
+    .flags = io::Highlight::None,
+  };
+  theme.hl[io::Token::Char] = {
     .fg = 0xFFAF70,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Keyword] = {
+  theme.hl[io::Token::Keyword] = {
     .fg = 0xFF8F40,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::KeywordOperator] = {
+  theme.hl[io::Token::KeywordOperator] = {
     .fg = 0xF07178,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Operator] = {
+  theme.hl[io::Token::Operator] = {
     .fg = 0xFFFFFF,
     .bg = 0x000000,
-    .flags = Highlight::Italic,
+    .flags = io::Highlight::Italic,
   };
-  theme.hl[internal::syntax::Token::Function] = {
+  theme.hl[io::Token::Function] = {
     .fg = 0xFFAF70,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Type] = {
+  theme.hl[io::Token::Type] = {
     .fg = 0xF07178,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Constant] = {
+  theme.hl[io::Token::Constant] = {
     .fg = 0x7DCFFF,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::VariableInstance] = {
+  theme.hl[io::Token::VariableInstance] = {
     .fg = 0x95E6CB,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::VariableGlobal] = {
+  theme.hl[io::Token::VariableGlobal] = {
     .fg = 0xF07178,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Annotation] = {
+  theme.hl[io::Token::Annotation] = {
     .fg = 0x7DCFFF,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Directive] = {
+  theme.hl[io::Token::Directive] = {
     .fg = 0xFF8F40,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Label] = {
+  theme.hl[io::Token::Label] = {
     .fg = 0xD2A6FF,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Brace1] = {
+  theme.hl[io::Token::Brace1] = {
-    .fg = 0xD2A6FF,
+    .fg = BRIGHT_GREEN,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Brace2] = {
+  theme.hl[io::Token::Brace2] = {
-    .fg = 0xFFAFAF,
+    .fg = YELLOW,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Brace3] = {
+  theme.hl[io::Token::Brace3] = {
     .fg = 0xFFFF00,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Brace4] = {
+  theme.hl[io::Token::Brace4] = {
-    .fg = 0x0FFF0F,
+    .fg = CYAN,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
-  theme.hl[internal::syntax::Token::Brace5] = {
+  theme.hl[io::Token::Brace5] = {
     .fg = 0xFF0F0F,
     .bg = 0x000000,
-    .flags = Highlight::None,
+    .flags = io::Highlight::None,
   };
   return theme;
 }
@@ -156,7 +254,6 @@ Theme Theme::default_theme() {
 Theme Theme::from_name(std::string_view name) {
   if (name == "default")
     return default_theme();
-  throw std::runtime_error("Unknown theme: " + std::string(name));
+  throw ed_error("Unknown theme: " + std::string(name));
 }
-
 } // namespace bed::internal::theme

src/internal/ui/command/command.cc

@@ -1,67 +1,8 @@
 #include "internal/ui/command.h"
 #include "bed.h"
+#include "internal/parser/parser.h"
 
 namespace bed::internal::ui {
-/*template <typename F>
-static void for_each_cluster(std::string_view s, F &&f) {
-  unicode_width_state_t state;
-  unicode_width_init(&state);
-  size_t i = 0;
-  while (i < s.size()) {
-    unsigned char c = static_cast<unsigned char>(s[i]);
-    size_t bytes = 1;
-    int width = 0;
-    if (c < 128) {
-      width = unicode_width_process(&state, c);
-    } else {
-      uint_least32_t cp;
-      size_t decoded = grapheme_decode_utf8(s.data() + i, s.size() - i, &cp);
-      bytes = decoded > 0 ? decoded : 1;
-      width = unicode_width_process(&state, cp);
-    }
-    if (width < 0)
-      width = 0;
-    f(i, bytes, width);
-    i += bytes;
-  }
-}
-
-static int display_width(std::string_view s) {
-  int w = 0;
-  for_each_cluster(s, [&](size_t, size_t, int cw) { w += cw; });
-  return w;
-}
-
-static uint16_t count_clusters(std::string_view s) {
-  uint16_t n = 0;
-  for_each_cluster(s, [&](size_t, size_t, int) { ++n; });
-  return n;
-}
-
-static std::vector<uint16_t> wrap_offsets(std::string_view line, uint16_t avail) {
-  std::vector<uint16_t> offsets{0};
-  int col = 0;
-  for_each_cluster(line, [&](uint16_t i, uint16_t, int w) {
-    if (col + w > avail && col > 0) {
-      offsets.push_back(i);
-      col = 0;
-    }
-    col += w;
-  });
-  return offsets;
-}
-
-// The word under/before `byte_pos`, split on plain ASCII spaces. Used to
-// pick what prefix to hand the suggestion trie.
-// TODO: change to use libgrapheme word break here.
-static std::string current_word(const std::string &line, size_t byte_pos) {
-  size_t start = (byte_pos == 0) ? std::string::npos : line.rfind(' ', byte_pos - 1);
-  start = (start == std::string::npos) ? 0 : start + 1;
-  if (byte_pos < start)
-    byte_pos = start;
-  return line.substr(start, byte_pos - start);
-}*/
-
 CommandIO::CommandIO(BEd &bed) : bed(bed) {
   if (bed.prompt_mode)
     prompt = bed.prompt(bed);
@@ -69,6 +10,17 @@ CommandIO::CommandIO(BEd &bed) : bed(bed) {
 }
 
 std::pair<std::string, bool> CommandIO::run() {
+  if (!bed.io.interactive())
+    return run_pipe();
+  return run_terminal();
+}
+
+std::pair<std::string, bool> CommandIO::run_pipe() {
+  bed.io.write(prompt);
+  return bed.io.read_pipe();
+}
+
+std::pair<std::string, bool> CommandIO::run_terminal() {
   auto [row, col] = bed.io.cursor_position();
   auto [rows, cols] = bed.io.terminal_size();
   if (row > rows)
@@ -151,7 +103,22 @@ void CommandIO::redraw() {
   bed.io.write("\x1b[2K", 4);
   bed.io.move_cursor(start, 1);
   bed.io.write(prompt);
-  bed.io.write(cmd);
+  const auto tokens = parser::Parser::get_highlight(cmd, bed);
+  uint32_t pos = 0;
+  for (const auto &token : tokens) {
+    const uint32_t tstart = token.start;
+    const uint32_t tend = token.end;
+    if (tstart > cmd.size())
+      break;
+    if (pos < tstart)
+      bed.io.write(cmd.data() + pos, tstart - pos);
+    bed.io.apply(token.type);
+    bed.io.write(cmd.data() + tstart, tend - tstart);
+    bed.io.reset();
+    pos = tend;
+  }
+  if (pos < cmd.size())
+    bed.io.write(cmd.data() + pos, cmd.size() - pos);
   bed.io.move_cursor(start, prompt.size() + cursor + 1);
 }
 } // namespace bed::internal::ui

src/internal/ui/text_mode/text_mode.cc

@@ -1,26 +1,170 @@
 #include "internal/ui/text_mode.h"
 #include "bed.h"
 
-namespace bed::internal::ui {
+namespace bed::internal::ui::text_mode {
-TextMode::TextMode(BEd &bed) : bed(bed) {
+// TODO: make this into a proper layout engine, in order to avoid so many expensive calculations repeatedly.
-  cursor = 0;
+
+template <typename F>
+static void for_each_cluster(std::string_view s, F &&f) {
+  size_t cluster_start = 0;
+  while (cluster_start < s.size()) {
+    size_t cluster_len =
+      grapheme_next_character_break_utf8(
+        s.data() + cluster_start,
+        s.size() - cluster_start
+      );
+    if (cluster_len == 0)
+      break;
+    size_t cluster_end = cluster_start + cluster_len;
+    unicode_width_state_t state;
+    unicode_width_init(&state);
+    size_t i = cluster_start;
+    int width = 0;
+    while (i < cluster_end) {
+      uint_least32_t cp;
+      size_t decoded =
+        grapheme_decode_utf8(s.data() + i, cluster_end - i, &cp);
+      if (decoded == 0)
+        decoded = 1;
+      int w = unicode_width_process(&state, cp);
+      if (w > width)
+        width = w;
+      i += decoded;
+    }
+    if (width < 0)
+      width = 0;
+    if (!f(cluster_start, cluster_len, width))
+      return;
+    cluster_start = cluster_end;
+  }
+}
+
+static size_t previous_cluster(std::string_view line, size_t col) {
+  size_t previous = 0;
+  for_each_cluster(line, [&](size_t start, size_t len, int) {
+    if (start >= col)
+      return false;
+    previous = start;
+    return start + len < col;
+  });
+  return previous;
+}
+
+static size_t next_cluster(std::string_view line, size_t col) {
+  size_t next = line.size();
+  for_each_cluster(line, [&](size_t start, size_t len, int) {
+    if (start >= col) {
+      next = start + len;
+      return false;
+    }
+    return true;
+  });
+  return next;
+}
+
+static size_t visual_column(std::string_view line, size_t col) {
+  size_t width = 0;
+  for_each_cluster(line, [&](size_t start, size_t, int cluster_width) {
+    if (start >= col)
+      return false;
+    width += cluster_width;
+    return true;
+  });
+  return width;
+}
+
+static size_t byte_column(std::string_view line, size_t target_width) {
+  size_t width = 0;
+  size_t col = 0;
+  for_each_cluster(line, [&](size_t start, size_t len, int cluster_width) {
+    if (width + cluster_width > target_width)
+      return false;
+    width += cluster_width;
+    col = start + len;
+    if (width == target_width)
+      return false;
+    return true;
+  });
+  return col;
+}
+
+static std::vector<std::pair<size_t, size_t>> wrap_line(std::string_view line, size_t avail) {
+  std::vector<std::pair<size_t, size_t>> result;
+  if (avail == 0) {
+    result.push_back({0, line.size()});
+    return result;
+  }
+  size_t begin = 0;
+  size_t end = 0;
+  int col = 0;
+  for_each_cluster(line, [&](size_t i, size_t len, int width) {
+    if (col > 0 && col + width > int(avail)) {
+      result.push_back({begin, i});
+      begin = i;
+      col = 0;
+    }
+    col += width;
+    end = i + len;
+    return true;
+  });
+  if (begin < line.size() || result.empty())
+    result.push_back({begin, end});
+  return result;
+}
+
+void TextMode::layout_lines() {
+  vase::Iterator it(vase, 0, Direction::Forward);
+  lines.clear();
+  while (it.next()) {
+    auto wrapped = wrap_line(it.line, term_width);
+    lines.push_back({});
+    for (auto [begin, end] : wrapped)
+      lines.back().push_back({begin, end - begin});
+  }
+}
+
+TextMode::TextMode(
+  BEd &bed, vase::Shard *vase, syntax::Language *lang, void *state
+) : vase(vase), bed(bed) {
+  if (lang)
+    parser.emplace(*lang, vase, state);
+  cursor = vase::eof_point(vase);
 }
 
 std::pair<vase::Shard *, bool> TextMode::run() {
+  if (!bed.io.interactive())
+    return run_pipe();
+  return run_terminal();
+}
+
+std::pair<vase::Shard *, bool> TextMode::run_pipe() {
+  vase::Shard::release(vase);
+  vase = nullptr;
+  while (true) {
+    auto [str, eof] = bed.io.read_pipe();
+    if (str == "." || eof)
+      break;
+    vase = vase::insert(&bed.append, vase, &cursor, str.data(), str.length());
+    vase = vase::insert(&bed.append, vase, &cursor, '\n');
+  }
+  return {vase, false};
+}
+
+std::pair<vase::Shard *, bool> TextMode::run_terminal() {
   auto [row, col] = bed.io.cursor_position();
   auto [rows, cols] = bed.io.terminal_size();
   if (row > rows)
-    throw fatal_error("Invalid cursor position.", 1);
+    throw ed_error("Invalid cursor position.");
   start = row;
   height = rows - row + 1;
   term_width = cols;
   term_height = rows;
-  cmd.clear();
-  cursor = 0;
   redraw();
   bool running = true;
+  bool cancelled = false;
 
   while (running) {
+    redraw();
     io::KeyEvent res = bed.io.read_key();
     switch (res.type) {
     case io::KeyEvent::KeyType::EOF_:
@@ -35,110 +179,161 @@ std::pair<vase::Shard *, bool> TextMode::run() {
       case io::KeyEvent::Modifier::ALT:
       case io::KeyEvent::Modifier::CTRL_ALT:
       case io::KeyEvent::Modifier::CTRL:
+        if (res.text[0] == 'c') {
+          cancelled = true;
+          running = false;
+        }
         break;
       case io::KeyEvent::Modifier::NONE:
         if (res.text[0] == '\b' || res.text[0] == 0x7f) {
-          if (cursor > 0) {
+          if (!vase)
-            cmd.erase(--cursor, 1);
+            break;
+          if (cursor.row || cursor.col) {
+            auto last = cursor;
+            if (last.col) {
+              vase::Iterator it(vase, last.row, Direction::Forward);
+              if (!it.next())
+                throw ed_error("Invalid cursor position.");
+              last.col = previous_cluster(it.line, last.col);
+            } else if (last.row) {
+              --last.row;
+              vase::Iterator it(vase, last.row, Direction::Forward);
+              if (!it.next())
+                throw ed_error("Invalid cursor position.");
+              last.col = it.line.size();
+            } else {
+              break;
+            }
+            vase::Range r = {last, cursor};
+            vase = vase::erase(vase, r);
+            if (parser) {
+              if (cursor.row == last.row) {
+                parser->dirty(vase, cursor.row, 1);
+              } else {
+                parser->erase(vase, cursor.row, 1);
+                parser->dirty(vase, last.row, 1);
+              }
+            }
+            cursor = last;
           }
         } else if (res.text[0] == '\n') {
-          cmd.insert(cursor++, 1, '\n');
+          if (vase && cursor.row == vase->lines) {
-          grow();
+            vase::Iterator it(vase, vase->lines, Direction::Backward);
+            if (it.next() && it.line == ".") {
+              vase = vase::erase(vase, vase->lines + 1, vase->lines + 1);
+              running = false;
+              break;
+            }
+          }
+          vase = vase::insert(&bed.append, vase, &cursor, '\n');
+          if (parser)
+            parser->insert(vase, cursor.row - 1, 1);
         } else {
-          cmd.insert(cursor, res.text);
+          vase = vase::insert(&bed.append, vase, &cursor, res.text.data(), res.text.size());
-          cursor += res.text.size();
+          if (parser)
+            parser->dirty(vase, cursor.row, 1);
         }
         break;
       }
       break;
-    case io::KeyEvent::KeyType::PASTE:
+    case io::KeyEvent::KeyType::PASTE: {
-      cmd.insert(cursor, res.text);
+      auto lines = std::count(res.text.begin(), res.text.end(), '\n');
-      cursor += res.text.size();
+      vase = vase::insert(&bed.append, vase, &cursor, res.text.data(), res.text.size());
-      grow();
+      if (parser) {
-      break;
+        if (lines) {
+          parser->insert(vase, cursor.row - lines, lines);
+          parser->dirty(vase, cursor.row - lines, 1);
+        } else {
+          parser->dirty(vase, cursor.row, 1);
+        }
+      }
+    } break;
     case io::KeyEvent::KeyType::SPECIAL:
+      if (!vase)
+        break;
       switch (res.special_key) {
       case io::KeyEvent::SpecialKey::UNKNOWN:
         break;
-      case io::KeyEvent::SpecialKey::RIGHT:
+      case io::KeyEvent::SpecialKey::RIGHT: {
-        if (cursor < cmd.size())
+        vase::Iterator it(vase, cursor.row, Direction::Forward);
-          ++cursor;
+        if (!it.next())
-        break;
+          throw ed_error("Invalid cursor position.");
-      case io::KeyEvent::SpecialKey::LEFT:
+        cursor.col = next_cluster(it.line, cursor.col);
-        if (cursor > 0)
+      } break;
-          --cursor;
+      case io::KeyEvent::SpecialKey::LEFT: {
-        break;
+        vase::Iterator it(vase, cursor.row, Direction::Forward);
+        if (!it.next())
+          throw ed_error("Invalid cursor position.");
+        cursor.col = previous_cluster(it.line, cursor.col);
+      } break;
       case io::KeyEvent::SpecialKey::UP: {
-        size_t line_start =
+        if (!cursor.row)
-          cmd.rfind('\n', cursor == 0 ? 0 : cursor - 1);
-        if (line_start == std::string::npos)
-          line_start = 0;
-        else
-          ++line_start;
-        size_t col = cursor - line_start;
-        if (line_start == 0)
           break;
-        size_t prev_end = line_start - 1;
+        vase::Iterator current(vase, cursor.row, Direction::Forward);
-        size_t prev_start =
+        if (!current.next())
-          cmd.rfind('\n', prev_end == 0 ? 0 : prev_end - 1);
+          throw ed_error("Invalid cursor position.");
-        if (prev_start == std::string::npos)
+        size_t wanted = visual_column(current.line, cursor.col);
-          prev_start = 0;
+        --cursor.row;
-        else
+        vase::Iterator previous(vase, cursor.row, Direction::Forward);
-          ++prev_start;
+        if (!previous.next())
-        size_t prev_len = prev_end - prev_start;
+          throw ed_error("Invalid cursor position.");
-        cursor = prev_start + std::min(col, prev_len);
+        cursor.col = byte_column(previous.line, wanted);
-        break;
+      } break;
-      }
       case io::KeyEvent::SpecialKey::DOWN: {
-        size_t line_start =
+        if (cursor.row >= vase->lines)
-          cmd.rfind('\n', cursor == 0 ? 0 : cursor - 1);
-        if (line_start == std::string::npos)
-          line_start = 0;
-        else
-          ++line_start;
-        size_t col = cursor - line_start;
-        size_t line_end = cmd.find('\n', cursor);
-        if (line_end == std::string::npos)
-          line_end = cmd.size();
-        if (line_end == cmd.size())
           break;
-        size_t next_start = line_end + 1;
+        vase::Iterator current(vase, cursor.row, Direction::Forward);
-        size_t next_end = cmd.find('\n', next_start);
+        if (!current.next())
-        if (next_end == std::string::npos)
+          throw ed_error("Invalid cursor position.");
-          next_end = cmd.size();
+        size_t wanted = visual_column(current.line, cursor.col);
-        size_t next_len = next_end - next_start;
+        ++cursor.row;
-        cursor = next_start + std::min(col, next_len);
+        vase::Iterator previous(vase, cursor.row, Direction::Forward);
-        break;
+        if (!previous.next())
-      }
+          throw ed_error("Invalid cursor position.");
+        cursor.col = byte_column(previous.line, wanted);
+      } break;
       case io::KeyEvent::SpecialKey::DELETE:
-        if (cursor < cmd.size())
+        auto next = cursor;
-          cmd.erase(cursor, 1);
+        vase::Iterator it(vase, cursor.row, Direction::Forward);
-        break;
+        if (!it.next())
+          break;
+        if (cursor.col < it.line.size()) {
+          vase::Iterator it(vase, next.row, Direction::Forward);
+          if (!it.next())
+            throw ed_error("Invalid cursor position.");
+          next.col = next_cluster(it.line, cursor.col);
+        } else if (cursor.row < vase->lines) {
+          ++next.row;
+          next.col = 0;
+        } else {
+          break;
+        }
+        vase::Range r = {cursor, next};
+        vase = vase::erase(vase, r);
+        if (parser) {
+          if (cursor.row != next.row)
+            parser->erase(vase, cursor.row + 1, 1);
+          parser->dirty(vase, cursor.row, 1);
+        }
       }
       break;
     }
-    redraw();
-    if (cmd.size() >= 3 && cmd.compare(cmd.size() - 3, 3, "\n.\n") == 0) {
-      cmd.erase(cmd.size() - 3);
-      cursor = cmd.size();
-      running = false;
-    }
-    if (cmd.size() == 2 && cmd.compare(0, 2, ".\n") == 0) {
-      cmd.clear();
-      cursor = 0;
-      running = false;
-    }
   }
 
-  size_t total_lines = cmd.size() ? 1 + std::count(cmd.begin(), cmd.end(), '\n') : 0;
+  layout_lines();
-  uint16_t last_row = start + total_lines;
+  size_t required_height = 0;
+  for (const auto &elem : lines)
+    required_height += elem.size();
+  uint16_t last_row = start + required_height;
   bed.io.move_cursor(last_row, 1);
-  bed.io.write("\n", 1);
+  bed.io.write("\n");
-  return {vase::Shard::from_string(cmd.data(), cmd.length(), true), false};
+  return {vase, cancelled};
 }
 
 void TextMode::grow() {
-  size_t required_height = 1 + std::count(cmd.begin(), cmd.end(), '\n');
+  size_t required_height = 0;
+  for (const auto &elem : lines)
+    required_height += elem.size();
   auto [rows, cols] = bed.io.terminal_size();
   term_height = rows;
   term_width = cols;
@@ -149,46 +344,85 @@ void TextMode::grow() {
     return;
   bed.io.move_cursor(rows, 1);
   for (long i = 0; i < overflow; ++i)
-    bed.io.write("\n", 1);
+    bed.io.write("\n");
   start -= overflow;
   if (start < 1)
     start = 1;
 }
 
 void TextMode::redraw() {
-  auto [rows, cols] = bed.io.terminal_size();
+  layout_lines();
-  term_height = rows;
+  grow();
-  term_width = cols;
+  bed.io.write("\x1b[?25l", 6);
   for (uint16_t i = 0; i < height; ++i) {
     bed.io.move_cursor(start + i, 1);
     bed.io.write("\x1b[2K", 4);
   }
-  size_t line = 0;
-  size_t line_start = 0;
-  for (size_t i = 0; i < cursor; ++i) {
-    if (cmd[i] == '\n') {
-      ++line;
-      line_start = i + 1;
-    }
-  }
-  size_t col = cursor - line_start;
-  size_t pos = 0;
   uint16_t screen_line = start;
-  while (pos <= cmd.size()) {
+  uint64_t line = 0;
-    size_t end = cmd.find('\n', pos);
+  size_t cursor_screen_row = start;
-    if (end == std::string::npos)
+  size_t cursor_screen_col = 0;
-      end = cmd.size();
+  vase::Iterator it(vase, line, Direction::Forward);
-    if (screen_line < start + height) {
+  while (it.next()) {
-      size_t len = std::min(end - pos, size_t(term_width));
+    if (screen_line >= start + height)
-      bed.io.move_cursor(screen_line, 1);
-      bed.io.write(cmd.substr(pos, len));
-    }
-    if (end == cmd.size())
       break;
-    pos = end + 1;
+    std::string_view l(it.line);
-    ++screen_line;
+    const std::vector<io::Token> *tokens = nullptr;
+    if (parser && parser->lines.size() > line)
+      tokens = &parser->lines[line].second;
+    size_t token_index = 0;
+    for (auto &row : lines[line]) {
+      const size_t row_start = row.start;
+      const size_t row_end = row.start + row.length;
+      bed.io.move_cursor(screen_line, 1);
+      if (tokens) {
+        while (token_index < tokens->size() && (*tokens)[token_index].end <= row_start)
+          ++token_index;
+        size_t pos = row_start;
+        for (size_t i = token_index; i < tokens->size(); ++i) {
+          const auto &token = (*tokens)[i];
+          if (token.start >= row_end)
+            break;
+          const size_t begin = std::max<size_t>(
+            token.start,
+            row_start
+          );
+          const size_t end = std::min<size_t>(
+            token.end,
+            row_end
+          );
+          if (begin > pos)
+            bed.io.write(l.data() + pos, begin - pos);
+          if (begin < end) {
+            bed.io.apply(token.type);
+            bed.io.write(l.data() + begin, end - begin);
+            bed.io.reset();
+            pos = end;
+          }
+        }
+        if (pos < row_end)
+          bed.io.write(l.data() + pos, row_end - pos);
+        bed.io.reset();
+      } else {
+        bed.io.write(l.substr(row_start, row.length));
+      }
+      if (line == cursor.row && cursor.col >= row_start && cursor.col <= row_end) {
+        cursor_screen_row = screen_line;
+        cursor_screen_col = 0;
+        for_each_cluster(
+          l.substr(row_start, cursor.col - row_start),
+          [&](size_t, size_t, int width) {
+            cursor_screen_col += width;
+            return true;
+          }
+        );
+      }
+      ++screen_line;
+    }
+    ++line;
   }
-  size_t vis_col = std::min(col, size_t(term_width - 1));
+  cursor_screen_col = std::min(cursor_screen_col, size_t(term_width - 1));
-  bed.io.move_cursor(start + line, vis_col + 1);
+  bed.io.move_cursor(cursor_screen_row, cursor_screen_col + 1);
+  bed.io.write("\x1b[?25h", 6);
 }
-} // namespace bed::internal::ui
+} // namespace bed::internal::ui::text_mode

src/internal/vase/regex/api.cc

@@ -90,12 +90,7 @@ Shard *substitute(
   if (matches.empty())
     return root;
   std::vector<ReplacePart> replace_parts = parse_replace(ap, replace);
-  struct Edit {
+  uint64_t current_line = 1;
-    uint64_t line;
-    uint64_t old_lines;
-    uint64_t new_lines;
-  };
-  uint64_t orig_line = start;
   int64_t line_delta = 0;
   std::vector<Shard *> pieces;
   pieces.reserve(matches.size() * 2 + 1);
@@ -109,7 +104,7 @@ Shard *substitute(
       Shard::release(remaining);
       pieces.push_back(keep);
       remaining = rest;
-      orig_line += keep ? keep->lines : 0;
+      current_line += keep ? keep->lines : 0;
     }
     auto [dropped, rest2] = Shard::split(remaining, match.end - match.start);
     Shard::release(remaining);
@@ -152,13 +147,11 @@ Shard *substitute(
       }
     }
     Shard::release(dropped);
-    if (old_lines || new_lines) {
+    uint64_t report_line = (uint64_t)((int64_t)current_line + line_delta);
-      uint64_t report_line = (uint64_t)((int64_t)orig_line + line_delta);
+    if (on_edit)
-      if (on_edit)
+      on_edit(report_line, old_lines, new_lines);
-        on_edit(report_line, old_lines, new_lines);
+    line_delta += (int64_t)new_lines - (int64_t)old_lines;
-      line_delta += (int64_t)new_lines - (int64_t)old_lines;
+    current_line += old_lines;
-    }
-    orig_line += old_lines;
     cursor = match.end;
   }
   pieces.push_back(remaining);

src/internal/vase/shard.cc

@@ -202,7 +202,7 @@ Shard *Shard::build(Shard **pieces, uint64_t lo, uint64_t hi) {
   return node;
 }
 
-Shard *Shard::from_command(const char *cmd, bool posix_ending) {
+Shard *Shard::from_command(const char *cmd) {
   auto o = new OriginalStorage("/tmp");
   int dest_fd = o->fd;
   if (dest_fd == -1) {
@@ -278,21 +278,21 @@ Shard *Shard::from_command(const char *cmd, bool posix_ending) {
     io::IO::enable_raw();
     return nullptr;
   }
-  if (posix_ending) {
+  if (ending[1] == '\n') {
-    if (ending[1] == '\n') {
+    Petal *last = (Petal *)pieces.back();
-      Petal *last = (Petal *)pieces.back();
+    last->lines--;
-      last->lines--;
+    last->length--;
-      last->length--;
+    if (last->length == 0) {
-      if (last->length == 0) {
+      Shard::release(last);
-        Shard::release(last);
+      pieces.pop_back();
-        pieces.pop_back();
+      last = nullptr;
-        last = nullptr;
+      if (!pieces.empty())
-        if (!pieces.empty())
+        last = (Petal *)pieces.back();
-          last = (Petal *)pieces.back();
+      else
-      }
+        return nullptr;
-      if (last && ending[0] == '\r')
-        last->length--;
     }
+    if (last && ending[0] == '\r')
+      last->length--;
   }
   o->initialize();
   io::IO::enable_raw();
@@ -301,7 +301,7 @@ Shard *Shard::from_command(const char *cmd, bool posix_ending) {
   return build(pieces.data(), 0, pieces.size());
 }
 
-Shard *Shard::from_file(const std::filesystem::path &path, bool posix_ending) {
+Shard *Shard::from_file(const std::filesystem::path &path) {
   auto o = new OriginalStorage("/tmp");
   int dest_fd = o->fd;
   if (dest_fd == -1) {
@@ -311,10 +311,10 @@ Shard *Shard::from_file(const std::filesystem::path &path, bool posix_ending) {
   int src_fd = open(path.c_str(), O_RDONLY);
   if (src_fd == -1) {
     delete o;
-    return nullptr;
+    throw ed_error("Couldn't open file.");
   }
   uint64_t total = std::filesystem::file_size(path);
-  if (posix_ending && total > 0) {
+  if (total > 0) {
     char last;
     if (pread(src_fd, &last, 1, (off_t)(total - 1)) != 1) {
       delete o;
@@ -379,7 +379,7 @@ Shard *Shard::from_file(const std::filesystem::path &path, bool posix_ending) {
   return build(pieces.data(), 0, pieces.size());
 }
 
-Shard *Shard::from_string(const char *data, uint64_t len, bool posix_ending) {
+Shard *Shard::from_string(const char *data, uint64_t len) {
   auto o = new OriginalStorage("/tmp");
   int dest_fd = o->fd;
   if (dest_fd == -1 || data == nullptr) {
@@ -387,7 +387,7 @@ Shard *Shard::from_string(const char *data, uint64_t len, bool posix_ending) {
     return nullptr;
   }
   uint64_t total = len;
-  if (posix_ending && total > 0) {
+  if (total > 0) {
     if (data[total - 1] == '\n') {
       total--;
       if (total > 0 && data[total - 1] == '\r')

src/internal/vase/vase.cc

@@ -74,14 +74,7 @@ void _insert(AppendStorage *ap, Shard **root, Point *point, const char *data, ui
     ++lines;
     last_line = ++data;
   }
-  uint64_t col = 0;
+  uint64_t col = end - last_line;
-  uint64_t remaining = end - last_line;
-  while (remaining) {
-    uint64_t n = grapheme_next_character_break_utf8(last_line, remaining);
-    last_line += n;
-    remaining -= n;
-    ++col;
-  }
   if (lines) {
     point->row += lines;
     point->col = col;
@@ -122,6 +115,22 @@ Shard *replace(AppendStorage *ap, Shard *root, Range range, const char *data, ui
   return insert(ap, root, &range.start, data, len);
 }
 
+Point eof_point(Shard *root) {
+  if (!root)
+    return {0, 0};
+  Point result{
+    .row = root->lines,
+    .col = 0
+  };
+  PetalIterator it(root, Direction::Forward);
+  it.seek_line(root->lines);
+  const char *data;
+  uint64_t len;
+  while (it.next(&data, &len))
+    result.col += len;
+  return result;
+}
+
 uint64_t offset_of(Shard *root, Point point) {
   return offset_of(root, point.row) + point.col;
 }
@@ -173,8 +182,6 @@ Shard *insert(AppendStorage *ap, Shard *root, Shard *text, uint64_t line) {
     Shard::retain(text);
     return text;
   }
-  if (!text)
-    return root;
   if (line > root->lines + 1)
     throw ed_error("line out of range");
   Shard::retain(text);

src/main.cc

@@ -4,8 +4,7 @@
 int main(int argc, char *argv[]) {
   std::vector<std::string> args(argv, argv + argc);
   try {
-    bed::internal::io::IO io = bed::internal::io::IO();
+    bed::BEd ed(args);
-    bed::BEd ed(args, io);
     ed.run();
   } catch (bed::fatal_error &e) {
     if (e.code)