Fix minor bugs and add language detection.

ARCHITECTURE.md

@@ -0,0 +1,149 @@
+# Base.
+
+### `PCH`
+
+Brings in all the external and standard headers. <br/>
+Apart from c++std stuff we have:
+- `mruby` and almost all it's subheaders.
+- `pcre2` (For regex.)
+- `grapheme.h` - The suckless libgrapheme (to find utf8 boundaries)
+- `unicode_width.h` - To find terminal width of unicode char's.
+
+*Maybe try removing some of them ive probably stopped using.*
+
+### `definitions.h`
+
+- Defines the `bed` namespace
+- Forward declaration of `BEd` type needed by all.
+- `fatal_error` definition, a msg, code (quits application on being thrown)
+- `ed_error`. just a message (stops the current cycle of the ed repl.)
+
+### `bed.h`
+
+**As this is based heavily on the internals, and so ill look into it at the end.**
+
+# `Internal`
+
+## `Vase`
+
+- This module doesn't depend on any other.
+- It's namespace is `bed::internal::vase`
+- Vase is the module to do with the actual storage and handling of a disk backed immutable and reference counted avl piece tree.
+- It is a O(log n) per edit text datatype.
+
+### `constants.h`
+
+Defines `PETAL_SIZE_MAX` as `32 * 1024`.
+- `PETAL_SIZE_MAX` limits teh size of a petal which makes multiple O(n) in petal searches O(1) as it is now bounded.
+
+### `storage/`
+
+A storage is something that stores the actual text, the text is actually stored in storages which are referenced by the trees.
+
+#### `storage.h`
+
+Defines the virtual `Storage` class.
+It has the methods:
+- `const char *read(uint64_t pos)`: reads text from position `pos` and returns a pointer to it.
+- `uint64_t length()`: returns the length of the storage.
+- `void retain() / release()`: Methods to help with refcounted storages.
+
+#### `original.h`
+
+Defines the `Original` type of storage.
+This is a refcounted storage.
+
+It has the fields:
+- `atomic_uint64_t refs{0}`: refcount
+- `const char *buf`: pointer to the actual text.
+- `uint64_t len`
+- `int fd = -1`: the file descriptor of teh undelying disk storage.
+
+It adds the method `initialize` over the base class.
+
+`src/vase/storage/original.cc`.
+
+The constructor uses `mkstemp` to make a temp file, then unlinks it to hide it from the fs but keeps the fd alive.
+
+`initialize` then `mmap`s the fd into a non modifyable pointer managed by the kernel then closes the fd itself.
+- This is what gives us a pointer to the file contents that the kernel can load and evict from memory as needed, and therefore if we have say a 500MB file loaded, but the cursor is arount 200MB and the user is only editing a couple lines around that, the rest of the file will not be using your memory, it might be loaded but if needed the kernel can evict it.
+
+The call site fills the fd before calling initialze, it cannot be modified after that.
+
+#### `append.h`
+
+Defines the `Append` type of storage.
+This is not actually refcounted, it is to be owned by the application just once and reused all throughout.
+It contains the actively typed stuff.
+
+It has the fields:
+- `const char *buf`: pointer to the actual text.
+- `uint64_t allocated_capacity`: the amount of bytees that can be written to it without needing to increase the size.
+- `uint64_t current_size`: Stores the size of actual text stored in it. (the cursor)
+- `int fd = -1`: the file descriptor of the underlying disk storage.
+
+It has the methods `append` (with 2 overloads) and private `grow` over the base class.
+`retain` and `release` do nothing.
+
+`src/vase/storage/append.cc`.
+
+The constructor uses `mkstemp` an `unlink` similar to the original storage.
+It then `ftruncate`s the file to `2^30` bytes or 1GiB this makes writing 1gb to the file possible but does'nt neccasarily use up 1gb on the disk.
+But when `mmap`ing we set `PROT_READ | PROT_WRITE` and `MAP_SHARED` to be able to write edits to disk.
+
+`grow` works by doubling the capacity `ftruncat`ing the fd to the new capacity and then on `linux` it uses `mremap` to remap to the new file size, but otherwise we `munmap` and `mmap` into the new size.
+
+The append methods append text into the buffer (either a single char or char * + len), they may grow the storage size. (the char*+len version uses memcpy).
+
+### `shard.h`
+
+Defines the actual peice tree.
+
+The tree is a polymorhic set of classes.
+
+The base class `Shard` has:
+- `enum Kind : uint8_t kind`: Branch or Petal.
+- `uint16_t height`: the tree height (for avl balancing.)
+- `atomic_uint32_t refs`: the refcount.
+- `uint64_t length`: the length in bytes of this (sub)tree.
+- `uint64_t lines`: the number of `\n` in this subtree.
+- It starts refcount at 1.
+
+The `Branch` class adds:
+- `Shard *left/right` the subtrees.
+- its constructor takes l, r and sets length/lines `(l->length + r->length)`
+- and height `1 + std::max(l->height, r->height)`
+
+The `Petal` (leaf) class adds:
+- `Storage *` a pointer to the storage used.
+- `uint64_t pos` the position in the storage its text starts at.
+- it takes `(uint64_t length, uint64_t lines, Storage *source, uint64_t pos)`
+- it retains `source`. (usefull for refcounted sotrages.)
+
+A `nullptr` `Shard*` is an empty peice of text (i.e. valid).
+
+`src/internal/vase/shard.cc`: defines a set of static function on the Shard class. (All the applications of it.)
+
+the operations methods are:
+- `retain/release` retain/releases them, if freeing a leaf it also releases the storage and for branches it releases both subtrees. nullptr's are ignored.
+- `balance/rotate_(right/left) and height/balance factor` are for doing simple avl balancing operations.
+- `concat` borrows 2 pointers to `Shard` and returns an owned Shard*.
+- `split` borrows a `Shard` and returns a `pair<Shard*, Shard*>` of 2 owned shards.
+- `merge_leaves` works similar to concat but it tries to merge the middle bit if they have their physical representation in order. for example if i have a shard `hell` and i type the letter `o` and because i'd been typing in order the Append storage will have text `hello` we can merge them into a single petal.
+- `append` similar to concat, but uses merge_leaves, useful for small appends like when typing.
+- `build` takes a pointer to an array of `Shard*` and start and end then makes a single avl balanced `Shard*` out of it it is generally faster than calling concat on each peice when we have many.
+
+Then for building a Shard* in the first place we have:
+- `from_command(const char *cmd)` runs the \0 terminated command string and reads its STDOUT and returns an owned Shard*.
+- `from_file(const std::filesystem::path &path)` reads the file at path.
+- `from_string(const char *data, uint64_t len)` loads the string given into a new storage, (normally inserting text can work by inserting into the append storage but this function loads it into a `OriginalStorage` object.).
+These methods all clip the final newline if it exists.
+And they also create a new `OriginalStorage` object.
+
+### `vase.h`
+
+### `iterators/`
+
+#### `line.h`
+
+#### `petal.h`

README.md

@@ -56,9 +56,8 @@ It should support:
 - Error handling.
 - And more.
 
-Not done yet.
+### TODO:
-
-### TODO immediately:
 
 - Make "g" command work.
-- properly handle escapes for %q ' etc in ruby parser (rn everything is escapable.)
+- Switch to proper posix style Shard objects (to distinguish 1 empty line from no lines.)
+- Use proper ruby binding context to keep locals across <pre>`</pre> function calls.

include/bed.h

@@ -20,6 +20,8 @@ struct BEd {
   internal::functions::Function eof_op;
   std::array<std::optional<internal::functions::Suffix>, 26> suffixes;
   internal::theme::Theme theme;
+  std::unordered_map<std::string, std::string> extention_to_language;
+  std::unordered_map<std::string, std::string> filename_to_language;
   std::unordered_map<std::string, internal::syntax::Language *> languages;
   internal::io::IO io;
   internal::vase::AppendStorage append{"/tmp"};

include/internal/buffer/decl.h

@@ -1,6 +1,7 @@
 #pragma once
 
 #include "definitions.h"
+#include "internal/syntax/c/parser.h"
 #include "internal/syntax/parser.h"
 #include "internal/syntax/ruby/parser.h"
 #include "internal/theme/theme.h"
@@ -32,7 +33,7 @@ struct Buffer {
   virtual bool waste() = 0;
   virtual uint64_t lines() = 0;
   virtual uint64_t bytes() = 0;
-  virtual void set_filename(std::filesystem::path path) = 0;
+  virtual void set_filename(BEd &ctx, std::filesystem::path path) = 0;
   virtual std::filesystem::path filename() = 0;
   virtual void saved_hook() = 0;
   virtual void load(BEd &ctx, vase::Shard *text) = 0;

include/internal/buffer/types/clip.h

@@ -14,7 +14,7 @@ struct ClipBuffer : Buffer {
   uint64_t lines() override;
   uint64_t bytes() override;
   void load(BEd &ctx, vase::Shard *text) override;
-  void set_filename(std::filesystem::path path) override;
+  void set_filename(BEd &ctx, std::filesystem::path path) override;
   std::filesystem::path filename() override;
   vase::Shard *copy(uint64_t start_line, uint64_t end_line) override;
   void substitute(

include/internal/buffer/types/generic.h

@@ -34,7 +34,7 @@ struct GenericBuffer : ShardBuffer {
   void saved_hook() override;
   bool waste() override;
   void load(BEd &ctx, vase::Shard *text) override;
-  void set_filename(std::filesystem::path path) override;
+  void set_filename(BEd &ctx, std::filesystem::path path) override;
   std::filesystem::path filename() override;
   void substitute(
     BEd &ctx, uint64_t start_line, uint64_t end_line,

include/internal/buffer/types/readonly.h

@@ -19,7 +19,7 @@ struct ReadonlyBuffer : ShardBuffer {
   void load(BEd &, vase::Shard *) override {
     throw ed_error("This buffer is read-only.");
   }
-  void set_filename(std::filesystem::path) override {}
+  void set_filename(BEd &, std::filesystem::path) override {}
   std::filesystem::path filename() override {
     return {};
   }

include/internal/syntax/c/decl.h

@@ -0,0 +1,30 @@
+#pragma once
+
+#include "../decl.h"
+#include "pch.h"
+#include "tries.h"
+
+namespace bed::internal::syntax::c {
+struct alignas(2) CState {
+  struct CInternalState {
+    uint16_t brace_level;
+    enum : uint8_t {
+      NONE,
+      STRING,
+      CHAR,
+      MULTI_COMMENT,
+      INLINE_COMMENT
+    } state;
+    uint8_t flags = 0;
+  };
+
+  uint8_t top;
+  // the stack (RubyInternalState * top)
+
+  inline CInternalState *stack() {
+    return (CInternalState *)(this + 1);
+  }
+};
+
+Language lang_c();
+} // namespace bed::internal::syntax::c

include/internal/syntax/c/parser.h

@@ -0,0 +1,65 @@
+#pragma once
+
+#include "decl.h"
+#include "pch.h"
+
+namespace bed::internal::syntax::c {
+struct CParser {
+  void **v_state;
+  CState *state;
+  std::string_view line;
+  bool still_empty = true;
+  uint32_t i = 0;
+
+  CParser(void **v_state, std::string_view line)
+      : v_state(v_state), state((CState *)*v_state), line(line) {}
+
+  uint32_t len() const {
+    return line.size();
+  }
+
+  CState::CInternalState &current() {
+    return state->stack()[state->top - 1];
+  }
+
+  char peek(int32_t offset = 0) {
+    uint32_t pos = i + offset;
+    return pos < line.size() ? line[pos] : '\0';
+  }
+
+  std::string_view peek_str(uint32_t len) {
+    return line.substr(i, len);
+  }
+
+  void advance() {
+    ++i;
+  }
+
+  void advance(uint32_t n) {
+    i += n;
+  }
+
+  void push_state() {
+    state->top++;
+    state = (CState *)realloc(
+      state,
+      sizeof(CState)
+        + sizeof(CState::CInternalState) * state->top
+    );
+    current() = {
+      .brace_level = 1,
+      .state = CState::CInternalState::NONE,
+      .flags = 0
+    };
+  }
+
+  void pop_state() {
+    state->top--;
+  }
+};
+
+void c_parse(
+  void **v_state, std::string_view line, bool first_line,
+  std::vector<io::Token> *tokens, std::vector<ParseEvent> *events
+);
+} // namespace bed::internal::syntax::c

include/internal/syntax/c/tries.h

@@ -0,0 +1,34 @@
+#pragma once
+
+#include "internal/trie/trie.h"
+#include "pch.h"
+
+namespace bed::internal::syntax::c {
+const static std::vector<std::string> types = {
+  "int",
+  "char",
+  "long",
+  "void",
+  "auto"
+};
+
+const static std::vector<std::string> type_modifier = {
+  "signed",
+  "unsigned",
+  "long"
+};
+
+const static std::vector<std::string> math_operators = {
+  "+",
+  "-",
+  "*"
+};
+
+struct CTries {
+  trie::Trie<void> types_trie;
+  CTries() {
+    for (auto &keyword : types)
+      types_trie.insert(keyword);
+  }
+};
+} // namespace bed::internal::syntax::c

include/internal/syntax/ruby/decl.h

@@ -24,7 +24,8 @@ struct alignas(2) RubyState {
       DEF_NAME = 0b10,
       MODULE_NAME = 0b11
     };
-    static constexpr uint8_t NEWLINE = 1 << 5;
+    static constexpr uint8_t NEWLINE = 1 << 4;
+    static constexpr uint8_t ALLOW_ESCAPE = 1 << 5;
     static constexpr uint8_t ALLOW_INTERPOLATION = 1 << 6;
     static constexpr uint8_t EXPECTING_EXPRESSION = 1 << 7;
     uint8_t flags = 0;

include/internal/vase/shard.h

@@ -6,6 +6,9 @@
 #include "storage/storage.h"
 
 namespace bed::internal::vase {
+// TODO: rewrite with invariant, text ends with a newline, as right now both 1 real and empty line and no text mean teh same thing,
+//       which causes a ton of issues.
+
 struct Shard {
   enum struct Kind : uint8_t {
     Branch,
@@ -24,9 +27,9 @@ struct Shard {
   static void retain(Shard *n);
   static void release(Shard *n);
 
-  static Shard *from_file(const std::filesystem::path &path, bool posix_ending);
+  static Shard *from_file(const std::filesystem::path &path);
-  static Shard *from_string(const char *data, uint64_t len, bool posix_ending);
+  static Shard *from_string(const char *data, uint64_t len);
-  static Shard *from_command(const char *cmd, bool posix_ending);
+  static Shard *from_command(const char *cmd);
 
   static std::pair<Shard *, Shard *> split(Shard *n, uint64_t offset);
   static Shard *concat(Shard *a, Shard *b);

src/bed/bed.cc

@@ -47,6 +47,14 @@ BEd::BEd(std::vector<std::string> args)
   internal::functions::Suffix::register_suffixes(*this);
   internal::scripting::register_basic(*this);
   languages["ruby"] = new internal::syntax::Language(internal::syntax::ruby::lang_ruby());
+  extention_to_language[".rb"] = "ruby";
+  languages["c"] = new internal::syntax::Language(internal::syntax::c::lang_c());
+  extention_to_language[".h"] = "c";
+  extention_to_language[".hh"] = "c";
+  extention_to_language[".h++"] = "c";
+  extention_to_language[".c"] = "c";
+  extention_to_language[".cc"] = "c";
+  extention_to_language[".c++"] = "c";
   buffers["clip"] = new internal::buffer::ClipBuffer("clip");
   current() = {"default", 0};
   try {

src/internal/buffer/generic.cc

@@ -249,11 +249,17 @@ void GenericBuffer::load(BEd &ctx, vase::Shard *text) {
     ctx.prev().end = text->lines + 1;
   }
   ctx.current() = {name, lines()};
+  syntax::Language *lang = parse.lang;
   syntax::release(parse);
-  parse = syntax::make_parser(root, lines(), nullptr);
+  if (save_path.has_extension()) {
+    auto it = ctx.extention_to_language.find(save_path.extension().string());
+    if (it != ctx.extention_to_language.end())
+      lang = ctx.languages[it->second];
+  }
+  parse = syntax::make_parser(root, lines(), lang);
 }
 
-void GenericBuffer::set_filename(std::filesystem::path path) {
+void GenericBuffer::set_filename(BEd &, std::filesystem::path path) {
   save_path = path;
 }
 

src/internal/buffer/special/clip.cc

@@ -11,14 +11,14 @@ bool ClipBuffer::waste() {
 void ClipBuffer::saved_hook() {}
 
 uint64_t ClipBuffer::lines() {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   uint64_t lines = s ? s->lines + 1 : 0;
   vase::Shard::release(s);
   return lines;
 }
 
 uint64_t ClipBuffer::bytes() {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   uint64_t length = s ? s->length + 1 : 0;
   vase::Shard::release(s);
   return length;
@@ -47,7 +47,7 @@ void ClipBuffer::load(BEd &ctx, vase::Shard *text) {
   }
 }
 
-void ClipBuffer::set_filename(std::filesystem::path) {}
+void ClipBuffer::set_filename(BEd &, std::filesystem::path) {}
 
 std::filesystem::path ClipBuffer::filename() {
   return "";
@@ -57,7 +57,7 @@ void ClipBuffer::append(BEd &ctx, vase::Shard *text, uint64_t line) {
   ctx.prev().buffername = name;
   ctx.prev().start = line + 1;
   ctx.prev().end = line + (text ? text->lines + 1 : 0);
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::insert(&ctx.append, s, text, line);
   clip_write(s);
   vase::Shard::release(s);
@@ -65,7 +65,7 @@ void ClipBuffer::append(BEd &ctx, vase::Shard *text, uint64_t line) {
 }
 
 void ClipBuffer::remove(BEd &ctx, uint64_t start_line, uint64_t end_line) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::erase(s, start_line, end_line);
   clip_write(s);
   ctx.prev().buffername = name;
@@ -85,7 +85,7 @@ void ClipBuffer::replace(BEd &ctx, vase::Shard *text, uint64_t start_line, uint6
   ctx.prev().end = start_line + text->lines;
   uint64_t new_count = text->lines + 1;
   uint64_t old_count = end_line - start_line + 1;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::replace(s, text, start_line, end_line);
   clip_write(s);
   vase::Shard::release(s);
@@ -99,7 +99,7 @@ void ClipBuffer::replace(BEd &ctx, vase::Shard *text, uint64_t start_line, uint6
 }
 
 void ClipBuffer::join(BEd &ctx, uint64_t start_line, uint64_t end_line) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::join(s, start_line, end_line);
   clip_write(s);
   vase::Shard::release(s);
@@ -116,7 +116,7 @@ void ClipBuffer::substitute(
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
   ctx.prev().end = end_line;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   s = vase::substitute(
     &ctx.append,
     s,
@@ -138,21 +138,21 @@ void ClipBuffer::substitute(
 }
 
 vase::Shard *ClipBuffer::copy(uint64_t start_line, uint64_t end_line) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   vase::Shard *o = vase::copy(s, start_line, end_line);
   vase::Shard::release(s);
   return o;
 }
 
 uint64_t ClipBuffer::find_next(std::string_view pattern, uint64_t start) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   uint64_t line = vase::find_next(s, pattern, start);
   vase::Shard::release(s);
   return line;
 }
 
 uint64_t ClipBuffer::find_prev(std::string_view pattern, uint64_t start) {
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   uint64_t line = vase::find_prev(s, pattern, start);
   vase::Shard::release(s);
   return line;
@@ -176,7 +176,7 @@ void ClipBuffer::print(BEd &ctx, uint64_t start_line, uint64_t end_line) {
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
   ctx.prev().end = end_line;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   vase::Iterator it(s, start_line - 1, Direction::Forward);
   while (it.next() && start_line++ <= end_line)
     ctx.io.write_line(it.line);
@@ -190,7 +190,7 @@ void ClipBuffer::number_print(BEd &ctx, uint64_t start_line, uint64_t end_line)
   uint8_t width = 1;
   for (uint64_t n = end_line; n >= 10; n /= 10)
     ++width;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   vase::Iterator it(s, start_line - 1, Direction::Forward);
   while (it.next() && start_line <= end_line)
     ctx.io.write_line(std::format("{:>{}}\t{}", start_line++, width, it.line));
@@ -201,7 +201,7 @@ void ClipBuffer::list_print(BEd &ctx, uint64_t start_line, uint64_t end_line) {
   ctx.prev().buffername = name;
   ctx.prev().start = start_line;
   ctx.prev().end = end_line;
-  auto s = vase::Shard::from_command("xclip -selection clipboard -o", true);
+  auto s = vase::Shard::from_command("xclip -selection clipboard -o");
   vase::Iterator it(s, start_line - 1, Direction::Forward);
   while (it.next() && start_line++ <= end_line)
     ctx.io.write_line(list_string(it.line));

src/internal/functions/extended.cc

@@ -197,7 +197,7 @@ void Function::register_extented(BEd &ctx) {
         ctx.io.write("=> ", 3);
         auto parser = syntax::MiniParser(
           *ctx.languages["ruby"],
-          vase::Shard::from_string(line.data(), line.size(), true),
+          vase::Shard::from_string(line.data(), line.size()),
           nullptr
         );
         const auto &tokens = parser.lines[0].second;

src/internal/functions/posix.cc

@@ -144,17 +144,17 @@ void Function::register_posix(BEd &ctx) {
         vase::Shard *s = nullptr;
         if (std::holds_alternative<std::filesystem::path>(arg)) {
           auto path = std::get<std::filesystem::path>(arg);
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
-          buf.set_filename(path);
+          buf.set_filename(ctx, path);
         } else if (std::holds_alternative<ShellArg>(arg)) {
           auto cmd = std::get<ShellArg>(arg).cmd;
           ctx.escape_command(cmd, buf.filename().string());
-          s = vase::Shard::from_command(cmd.c_str(), true);
+          s = vase::Shard::from_command(cmd.c_str());
         } else {
           auto path = buf.filename();
           if (path.empty())
             throw ed_error("Need filename.");
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
         };
         try {
           buf.load(ctx, s);
@@ -190,17 +190,17 @@ void Function::register_posix(BEd &ctx) {
         vase::Shard *s = nullptr;
         if (std::holds_alternative<std::filesystem::path>(arg)) {
           auto path = std::get<std::filesystem::path>(arg);
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
-          buf.set_filename(path);
+          buf.set_filename(ctx, path);
         } else if (std::holds_alternative<ShellArg>(arg)) {
           auto cmd = std::get<ShellArg>(arg).cmd;
           ctx.escape_command(cmd, buf.filename().string());
-          s = vase::Shard::from_command(cmd.c_str(), true);
+          s = vase::Shard::from_command(cmd.c_str());
         } else {
           auto path = buf.filename();
           if (path.empty())
             throw ed_error("Need filename.");
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
         };
         try {
           buf.load(ctx, s);
@@ -234,7 +234,7 @@ void Function::register_posix(BEd &ctx) {
         auto &addr = std::get<std::string>(addr_);
         auto &buf = ctx.buffer(addr);
         if (std::holds_alternative<std::filesystem::path>(arg))
-          buf.set_filename(std::get<std::filesystem::path>(arg));
+          buf.set_filename(ctx, std::get<std::filesystem::path>(arg));
         else if (std::holds_alternative<ShellArg>(arg))
           throw ed_error("Can't save shell command as save path.");
         if (buf.filename().empty())
@@ -588,18 +588,18 @@ void Function::register_posix(BEd &ctx) {
         vase::Shard *s = nullptr;
         if (std::holds_alternative<std::filesystem::path>(arg)) {
           auto path = std::get<std::filesystem::path>(arg);
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
           if (buf.filename().empty())
-            buf.set_filename(path);
+            buf.set_filename(ctx, path);
         } else if (std::holds_alternative<ShellArg>(arg)) {
           auto cmd = std::get<ShellArg>(arg).cmd;
           ctx.escape_command(cmd, buf.filename().string());
-          s = vase::Shard::from_command(cmd.c_str(), true);
+          s = vase::Shard::from_command(cmd.c_str());
         } else {
           auto path = buf.filename();
           if (path.empty())
             throw ed_error("Need filename.");
-          s = vase::Shard::from_file(path, true);
+          s = vase::Shard::from_file(path);
         };
         try {
           buf.append(ctx, s, addr.number);
@@ -740,7 +740,7 @@ void Function::register_posix(BEd &ctx) {
           if (std::holds_alternative<std::filesystem::path>(arg)) {
             auto path = std::get<std::filesystem::path>(arg);
             vase::write_file(path, text);
-            buf.set_filename(path);
+            buf.set_filename(ctx, path);
           } else if (std::holds_alternative<ShellArg>(arg)) {
             auto cmd = std::get<ShellArg>(arg).cmd;
             ctx.escape_command(cmd, buf.filename().string());

src/internal/syntax/c/core.cc

@@ -0,0 +1,37 @@
+#include "internal/syntax/c/parser.h"
+
+namespace bed::internal::syntax::c {
+Language lang_c() {
+  return Language{
+    .none_state = []() {
+      CState *st = (CState *)malloc(
+        sizeof(CState)
+        + sizeof(CState::CInternalState)
+      );
+      st->top = 1;
+      st->stack()[0] = {
+        .brace_level = 1,
+        .state = CState::CInternalState::NONE,
+        .flags = 0
+      };
+      return st; },
+    .parse = c_parse,
+    .copy = [](void *v_i_st) {
+      CState *i_st = (CState *)v_i_st;
+      uint32_t bytes = sizeof(CState)
+                       + sizeof(CState::CInternalState) * i_st->top;
+      CState *o_st = (CState *)malloc(bytes);
+      memcpy(o_st, i_st, bytes);
+      return o_st; },
+    .equal = [](void *v_a_st, void *v_b_st) {
+      CState *a_st = (CState *)v_a_st;
+      uint32_t a_bytes = sizeof(CState)
+                         + sizeof(CState::CInternalState) * a_st->top;
+      CState *b_st = (CState *)v_b_st;
+      uint32_t b_bytes = sizeof(CState)
+                         + sizeof(CState::CInternalState) * b_st->top;
+      return a_bytes == b_bytes && memcmp(a_st, b_st, a_bytes) == 0; },
+    .destroy = [](void *v_st) { free(v_st); },
+  };
+}
+} // namespace bed::internal::syntax::c

src/internal/syntax/c/parse.cc

@@ -0,0 +1,396 @@
+#include "internal/syntax/c/parser.h"
+
+namespace bed::internal::syntax::c {
+inline bool is_hex(char c) {
+  return ('0' <= c && c <= '9')
+         || ('a' <= c && c <= 'f')
+         || ('A' <= c && c <= 'F');
+};
+
+inline bool identifier_start_char(char c) {
+  return (c & 0x80)
+         || ('a' <= c && c <= 'z')
+         || ('A' <= c && c <= 'Z')
+         || c == '_';
+}
+
+inline bool identifier_char(char c) {
+  return (c & 0x80)
+         || ('a' <= c && c <= 'z')
+         || ('A' <= c && c <= 'Z')
+         || ('0' <= c && c <= '9')
+         || c == '_';
+}
+
+void handle_multiline_comment(CParser &p, std::vector<io::Token> *tokens, std::vector<ParseEvent> *events) {
+  uint32_t start = p.i;
+  while (p.i < p.len()) {
+    if (p.peek() == '*' && p.peek(1) == '/') {
+      p.i += 2;
+      p.current().state = CState::CInternalState::NONE;
+      tokens->push_back({start, p.i, io::Token::Comment});
+      events->push_back(true);
+      return;
+    }
+    p.advance();
+  }
+  tokens->push_back({start, p.i, io::Token::Comment});
+}
+
+void handle_inline_comment(CParser &p, std::vector<io::Token> *tokens, std::vector<ParseEvent> *) {
+  uint32_t start = p.i;
+  p.i = p.len();
+  tokens->push_back({start, p.i, io::Token::Comment});
+  if (p.line.empty() || p.line.back() != '\\')
+    p.current().state = CState::CInternalState::NONE;
+}
+
+bool handle_escapes(CParser &p, std::vector<io::Token> *tokens, uint32_t &start, bool string) {
+  if (p.peek() != '\\')
+    return false;
+  if (string)
+    tokens->push_back({start, p.i, io::Token::String});
+  else
+    tokens->push_back({start, p.i, io::Token::Char});
+  start = p.i;
+  p.advance();
+  if (p.peek() == 'x') {
+    p.advance();
+    while (is_hex(p.peek()))
+      p.advance();
+  } else if (p.peek() == 'u') {
+    p.advance();
+    for (int i = 0; i < 4 && is_hex(p.peek()); i++)
+      p.advance();
+  } else if (p.peek() == 'U') {
+    p.advance();
+    for (int i = 0; i < 8 && is_hex(p.peek()); i++)
+      p.advance();
+  } else if ('0' <= p.peek() && p.peek() <= '7') {
+    p.advance();
+    if ('0' <= p.peek() && p.peek() <= '7')
+      p.advance();
+    if ('0' <= p.peek() && p.peek() <= '7')
+      p.advance();
+  } else {
+    p.advance();
+  }
+  tokens->push_back({start, p.i, io::Token::Escape});
+  start = p.i;
+  return true;
+}
+
+void handle_string(CParser &p, std::vector<io::Token> *tokens, bool string) {
+  uint32_t start = p.i;
+  while (p.i < p.len()) {
+    if (handle_escapes(p, tokens, start, string))
+      continue;
+    if (p.peek() == (string ? '"' : '\'')) {
+      p.advance();
+      if (string)
+        tokens->push_back({start, p.i, io::Token::String});
+      else
+        tokens->push_back({start, p.i, io::Token::Char});
+      p.current().state = CState::CInternalState::NONE;
+      return;
+    }
+    p.advance();
+  }
+  if (p.i >= p.len()) {
+    if (string)
+      tokens->push_back({start, p.len(), io::Token::String});
+    else
+      tokens->push_back({start, p.len(), io::Token::Char});
+    if (p.line.empty() || p.line.back() != '\\')
+      p.current().state = CState::CInternalState::NONE;
+  }
+}
+
+void handle_number(CParser &p, std::vector<io::Token> *tokens) {
+  bool integer = true;
+  uint32_t start = p.i;
+  if (p.peek() == '0') {
+    p.advance();
+    if (p.peek() == 'x' || p.peek() == 'X') {
+      p.advance();
+      if (!(is_hex(p.peek()) || p.peek() == '.'))
+        return;
+      while (is_hex(p.peek())) {
+        p.advance();
+        if (p.peek() == '\'') {
+          p.advance();
+          if (!(is_hex(p.peek())))
+            return;
+        }
+      }
+      bool need_p = false;
+      if (p.peek() == '.') {
+        need_p = true;
+        p.advance();
+      }
+      while (is_hex(p.peek())) {
+        p.advance();
+        if (p.peek() == '\'') {
+          p.advance();
+          if (!(is_hex(p.peek())))
+            return;
+        }
+      }
+      if (p.peek() == 'p' || p.peek() == 'P') {
+        integer = false;
+        p.advance();
+        if (p.peek() == '-' || p.peek() == '+')
+          p.advance();
+        if (!('0' <= p.peek() && p.peek() <= '9'))
+          return;
+        p.advance();
+        while ('0' <= p.peek() && p.peek() <= '9') {
+          p.advance();
+          if (p.peek() == '\'') {
+            p.advance();
+            if (!('0' <= p.peek() && p.peek() <= '9'))
+              return;
+          }
+        }
+      } else if (need_p) {
+        return;
+      }
+    } else if (p.peek() == 'b' || p.peek() == 'B') {
+      p.advance();
+      if (p.peek() != '0' && p.peek() != '1')
+        return;
+      p.advance();
+      while (p.peek() == '0' || p.peek() == '1') {
+        p.advance();
+        if (p.peek() == '\'') {
+          p.advance();
+          if (!(p.peek() == '0' || p.peek() == '1'))
+            return;
+        }
+      }
+    } else {
+      while ('0' <= p.peek() && p.peek() <= '7') {
+        p.advance();
+        if (p.peek() == '\'') {
+          p.advance();
+          if (!('0' <= p.peek() && p.peek() <= '7'))
+            return;
+        }
+      }
+      if (p.peek() == '.') {
+        integer = false;
+        p.advance();
+      }
+      while ('0' <= p.peek() && p.peek() <= '7') {
+        p.advance();
+        if (p.peek() == '\'') {
+          p.advance();
+          if (!('0' <= p.peek() && p.peek() <= '7'))
+            return;
+        }
+      }
+      if (p.peek() == 'e' || p.peek() == 'E') {
+        integer = false;
+        p.advance();
+        if (p.peek() == '-' || p.peek() == '+')
+          p.advance();
+        if (!('0' <= p.peek() && p.peek() <= '9'))
+          return;
+        p.advance();
+        while ('0' <= p.peek() && p.peek() <= '9') {
+          p.advance();
+          if (p.peek() == '\'') {
+            p.advance();
+            if (!('0' <= p.peek() && p.peek() <= '9'))
+              return;
+          }
+        }
+      }
+    }
+  } else if (p.peek() == '.') {
+    integer = false;
+    p.advance();
+    goto after_decimal;
+  } else {
+    p.advance();
+    while ('0' <= p.peek() && p.peek() <= '9') {
+      p.advance();
+      if (p.peek() == '\'') {
+        p.advance();
+        if (!('0' <= p.peek() && p.peek() <= '9'))
+          return;
+      }
+    }
+    if (p.peek() == '.') {
+      integer = false;
+      p.advance();
+    }
+  after_decimal:
+    while ('0' <= p.peek() && p.peek() <= '9') {
+      p.advance();
+      if (p.peek() == '\'') {
+        p.advance();
+        if (!('0' <= p.peek() && p.peek() <= '9'))
+          return;
+      }
+    }
+    if (p.peek() == 'e' || p.peek() == 'E') {
+      integer = false;
+      p.advance();
+      if (p.peek() == '-' || p.peek() == '+')
+        p.advance();
+      if (!('0' <= p.peek() && p.peek() <= '9'))
+        return;
+      p.advance();
+      while ('0' <= p.peek() && p.peek() <= '9') {
+        p.advance();
+        if (p.peek() == '\'') {
+          p.advance();
+          if (!('0' <= p.peek() && p.peek() <= '9'))
+            return;
+        }
+      }
+    }
+  }
+  if (integer) {
+    if (p.peek() == 'u' || p.peek() == 'U') {
+      p.advance();
+      if (p.peek() == 'w') {
+        p.advance();
+        if (p.peek() != 'b')
+          return;
+        p.advance();
+      }
+      if (p.peek() == 'W') {
+        p.advance();
+        if (p.peek() != 'B')
+          return;
+        p.advance();
+      }
+      if (p.peek() == 'l' || p.peek() == 'L')
+        p.advance();
+      if (p.peek() == 'l' || p.peek() == 'L')
+        p.advance();
+    } else if (p.peek() == 'l' || p.peek() == 'L') {
+      p.advance();
+      if (p.peek() == 'l' || p.peek() == 'L')
+        p.advance();
+      if (p.peek() == 'u' || p.peek() == 'U')
+        p.advance();
+    } else if (p.peek_str(2) == "wb" || p.peek_str(2) == "WB") {
+      p.advance(2);
+    } else if (p.peek() == 'f' || p.peek() == 'F' || p.peek() == 'l' || p.peek() == 'L') {
+      p.advance();
+    } else if ((p.peek() == 'd' || p.peek() == 'D')
+               && (p.peek(1) == 'f' || p.peek(1) == 'F' || p.peek(1) == 'd' || p.peek(1) == 'D' || p.peek(1) == 'l' || p.peek(1) == 'L')) {
+      p.advance(2);
+    }
+  } else {
+    if (p.peek() == 'f' || p.peek() == 'F' || p.peek() == 'l' || p.peek() == 'L')
+      p.advance();
+    else if ((p.peek() == 'd' || p.peek() == 'D')
+             && (p.peek(1) == 'f' || p.peek(1) == 'F' || p.peek(1) == 'd' || p.peek(1) == 'D' || p.peek(1) == 'l' || p.peek(1) == 'L'))
+      p.advance(2);
+  }
+  if (identifier_char(p.peek()) || p.peek() == '.')
+    return;
+  tokens->push_back({start, p.i, io::Token::Number});
+}
+
+void handle_syntax(CParser &p, std::vector<io::Token> *tokens, std::vector<ParseEvent> *events) {
+  if (p.peek() == '/' && p.peek(1) == '/') {
+    p.current().state = CState::CInternalState::INLINE_COMMENT;
+    return;
+  }
+  if (p.peek() == '/' && p.peek(1) == '*') {
+    p.current().state = CState::CInternalState::MULTI_COMMENT;
+    events->push_back(false);
+    return;
+  }
+  switch (p.peek()) {
+  case '"':
+    p.current().state = CState::CInternalState::STRING;
+    p.advance();
+    tokens->push_back({p.i - 1, p.i, io::Token::String});
+    return;
+  case '\'':
+    p.current().state = CState::CInternalState::CHAR;
+    p.advance();
+    tokens->push_back({p.i - 1, p.i, io::Token::Char});
+    return;
+  case 'u': {
+    uint32_t j = 1;
+    if (p.peek(j) == '8')
+      j++;
+    if (p.peek(j) == '"') {
+      p.current().state = CState::CInternalState::STRING;
+      tokens->push_back({p.i, p.i + j, io::Token::Annotation});
+      p.advance(j + 1);
+      tokens->push_back({p.i - 1, p.i, io::Token::String});
+      return;
+    } else if (p.peek(j) == '\'') {
+      p.current().state = CState::CInternalState::CHAR;
+      tokens->push_back({p.i, p.i + j, io::Token::Annotation});
+      p.advance(j + 1);
+      tokens->push_back({p.i - 1, p.i, io::Token::Char});
+      return;
+    }
+    goto def;
+  }
+  case 'U':
+  case 'L': {
+    if (p.peek(1) == '"') {
+      p.current().state = CState::CInternalState::STRING;
+      tokens->push_back({p.i, p.i + 1, io::Token::Annotation});
+      p.advance(2);
+      tokens->push_back({p.i - 1, p.i, io::Token::String});
+      return;
+    } else if (p.peek(1) == '\'') {
+      p.current().state = CState::CInternalState::CHAR;
+      tokens->push_back({p.i, p.i + 1, io::Token::Annotation});
+      p.advance(2);
+      tokens->push_back({p.i - 1, p.i, io::Token::Char});
+      return;
+    }
+    goto def;
+  }
+  default:
+  def:
+    if (('0' <= p.peek() && p.peek() <= '9') || p.peek() == '.') {
+      handle_number(p, tokens);
+      return;
+    }
+    p.advance();
+  }
+}
+
+void c_parse(
+  void **v_state,
+  std::string_view line,
+  bool,
+  std::vector<io::Token> *tokens,
+  std::vector<ParseEvent> *events
+) {
+  CParser p(v_state, line);
+  while (p.i < p.len()) {
+    if (p.current().state == CState::CInternalState::MULTI_COMMENT) {
+      handle_multiline_comment(p, tokens, events);
+      continue;
+    }
+    if (p.current().state == CState::CInternalState::INLINE_COMMENT) {
+      handle_inline_comment(p, tokens, events);
+      return;
+    }
+    if (p.current().state == CState::CInternalState::STRING) {
+      handle_string(p, tokens, true);
+      continue;
+    }
+    if (p.current().state == CState::CInternalState::CHAR) {
+      handle_string(p, tokens, false);
+      continue;
+    }
+    handle_syntax(p, tokens, events);
+    continue;
+  }
+}
+} // namespace bed::internal::syntax::c

src/internal/syntax/ruby/parse.cc

@@ -35,8 +35,21 @@ inline uint8_t utf8_codepoint_width(unsigned char c) {
 }
 
 bool handle_escapes(RubyParser &p, std::vector<io::Token> *tokens, uint32_t &start, bool string = true) {
-  // TODO: properly handle escapes for %q ' etc.
+  if (p.peek() != '\\')
-  if (p.peek() == '\\') {
+    return false;
+  if (!(p.current().flags & RubyState::RubyInternalState::ALLOW_ESCAPE)) {
+    p.advance();
+    if (p.peek() != '\'' && p.peek() != '\\')
+      return false;
+    if (string)
+      tokens->push_back({start, p.i - 1, io::Token::String});
+    else
+      tokens->push_back({start, p.i - 1, io::Token::Regexp});
+    p.advance();
+    tokens->push_back({p.i - 2, p.i, io::Token::Escape});
+    start = p.i;
+    return true;
+  }
   if (string)
     tokens->push_back({start, p.i, io::Token::String});
   else
@@ -100,8 +113,6 @@ bool handle_escapes(RubyParser &p, std::vector<io::Token> *tokens, uint32_t &sta
   start = p.i;
   return true;
 }
-  return false;
-};
 
 bool handle_heredoc(RubyParser &p, std::vector<io::Token> *tokens, std::vector<ParseEvent> *events) {
   uint8_t *heredocs = p.state->heredocs();
@@ -669,6 +680,7 @@ bool handle_syntax(RubyParser &p, std::vector<io::Token> *tokens, std::vector<Pa
     p.current().delim_start = '\'';
     p.current().delim_end = '\'';
     p.current().flags &= ~RubyState::RubyInternalState::ALLOW_INTERPOLATION;
+    p.current().flags &= ~RubyState::RubyInternalState::ALLOW_ESCAPE;
     p.advance();
     return false;
   }
@@ -679,6 +691,7 @@ bool handle_syntax(RubyParser &p, std::vector<io::Token> *tokens, std::vector<Pa
     p.current().delim_start = '"';
     p.current().delim_end = '"';
     p.current().flags |= RubyState::RubyInternalState::ALLOW_INTERPOLATION;
+    p.current().flags |= RubyState::RubyInternalState::ALLOW_ESCAPE;
     p.advance();
     return false;
   }
@@ -704,6 +717,7 @@ bool handle_syntax(RubyParser &p, std::vector<io::Token> *tokens, std::vector<Pa
     char delim_start = '\0';
     char delim_end = '\0';
     bool allow_interp = true;
+    bool allow_escape = true;
     int prefix_len = 1;
     bool is_regexp = false;
     switch (type) {
@@ -717,6 +731,7 @@ bool handle_syntax(RubyParser &p, std::vector<io::Token> *tokens, std::vector<Pa
     case 'I':
     case 'W':
       allow_interp = true;
+      allow_escape = true;
       prefix_len = 2;
       break;
     case 'w':
@@ -724,10 +739,12 @@ bool handle_syntax(RubyParser &p, std::vector<io::Token> *tokens, std::vector<Pa
     case 'i':
     case 's':
       allow_interp = false;
+      allow_escape = false;
       prefix_len = 2;
       break;
     default:
       allow_interp = true;
+      allow_escape = true;
       prefix_len = 1;
       break;
     }
@@ -769,6 +786,8 @@ bool handle_syntax(RubyParser &p, std::vector<io::Token> *tokens, std::vector<Pa
     p.current().delim_end = delim_end;
     if (allow_interp)
       p.current().flags |= RubyState::RubyInternalState::ALLOW_INTERPOLATION;
+    if (allow_escape)
+      p.current().flags |= RubyState::RubyInternalState::ALLOW_ESCAPE;
     p.current().lit_brace_level = 1;
     p.advance(prefix_len + 1);
     return false;
@@ -981,6 +1000,7 @@ bool handle_syntax(RubyParser &p, std::vector<io::Token> *tokens, std::vector<Pa
           p.current().flags &= ~RubyState::RubyInternalState::EXPECTING_EXPRESSION;
           tokens->push_back({p.i, p.i + j, io::Token::Function});
           p.advance(j);
+          return false;
         }
         uint32_t start = p.i;
         if (p.peek(j) == ':') {

src/internal/vase/shard.cc

@@ -202,7 +202,7 @@ Shard *Shard::build(Shard **pieces, uint64_t lo, uint64_t hi) {
   return node;
 }
 
-Shard *Shard::from_command(const char *cmd, bool posix_ending) {
+Shard *Shard::from_command(const char *cmd) {
   auto o = new OriginalStorage("/tmp");
   int dest_fd = o->fd;
   if (dest_fd == -1) {
@@ -278,7 +278,6 @@ Shard *Shard::from_command(const char *cmd, bool posix_ending) {
     io::IO::enable_raw();
     return nullptr;
   }
-  if (posix_ending) {
   if (ending[1] == '\n') {
     Petal *last = (Petal *)pieces.back();
     last->lines--;
@@ -295,7 +294,6 @@ Shard *Shard::from_command(const char *cmd, bool posix_ending) {
     if (last && ending[0] == '\r')
       last->length--;
   }
-  }
   o->initialize();
   io::IO::enable_raw();
   if (pieces.size() == 1)
@@ -303,7 +301,7 @@ Shard *Shard::from_command(const char *cmd, bool posix_ending) {
   return build(pieces.data(), 0, pieces.size());
 }
 
-Shard *Shard::from_file(const std::filesystem::path &path, bool posix_ending) {
+Shard *Shard::from_file(const std::filesystem::path &path) {
   auto o = new OriginalStorage("/tmp");
   int dest_fd = o->fd;
   if (dest_fd == -1) {
@@ -313,10 +311,10 @@ Shard *Shard::from_file(const std::filesystem::path &path, bool posix_ending) {
   int src_fd = open(path.c_str(), O_RDONLY);
   if (src_fd == -1) {
     delete o;
-    return nullptr;
+    throw ed_error("Couldn't open file.");
   }
   uint64_t total = std::filesystem::file_size(path);
-  if (posix_ending && total > 0) {
+  if (total > 0) {
     char last;
     if (pread(src_fd, &last, 1, (off_t)(total - 1)) != 1) {
       delete o;
@@ -381,7 +379,7 @@ Shard *Shard::from_file(const std::filesystem::path &path, bool posix_ending) {
   return build(pieces.data(), 0, pieces.size());
 }
 
-Shard *Shard::from_string(const char *data, uint64_t len, bool posix_ending) {
+Shard *Shard::from_string(const char *data, uint64_t len) {
   auto o = new OriginalStorage("/tmp");
   int dest_fd = o->fd;
   if (dest_fd == -1 || data == nullptr) {
@@ -389,7 +387,7 @@ Shard *Shard::from_string(const char *data, uint64_t len, bool posix_ending) {
     return nullptr;
   }
   uint64_t total = len;
-  if (posix_ending && total > 0) {
+  if (total > 0) {
     if (data[total - 1] == '\n') {
       total--;
       if (total > 0 && data[total - 1] == '\r')

src/internal/vase/vase.cc

@@ -179,7 +179,10 @@ Shard *insert(AppendStorage *ap, Shard *root, Shard *text, uint64_t line) {
   if (!root) {
     if (line != 0)
       throw ed_error("line out of range");
+    if (text)
       Shard::retain(text);
+    else
+      text = newline(ap);
     return text;
   }
   if (line > root->lines + 1)