Add indentation engine and other fixes

.clangd

@@ -2,3 +2,4 @@ CompileFlags:
   Add: [-I/home/syed/main/crib/include, -I/home/syed/main/crib/libs, c++20]
   Remove: []
   Compiler: clang++
+

README.md

@@ -251,7 +251,7 @@ Activated by `:` or `;`.
 - move lines up/down
 - folding on a selected range
 - yank/cut/paste via system clipboard
-- auto-indent on new lines
+- per-language smart auto-indent on new line insert
 - bracket/quote auto-pairing
 - hooks jumping (bookmarking)
 - color hex code highlighting
@@ -292,7 +292,10 @@ Activated by `:` or `;`.
 - diagnostics
 - autocompletion
 - hover docs (with markdown support)
-- formatting *(very few lsp's support this - try to configure a few more which can but need configuration and for others need to add support for external formatters)*
+- formatting support
+    - Full file formatting on save
+    - Ontype formatting when inserting special characters defined by the language server
+    - *(few lsp's actually support this - try to configure a few more which can but need configuration and for others need to add support for external formatters)*
 - A list of all supported lsp's can be found [here](#lsps).
 
 **A lot lot more to come**

TODO.md

@@ -4,36 +4,45 @@ Copyright 2025 Syed Daanish
 
 ### Critical Fixes
 
-* [ ] **Critical Crash:** Fix bug where closing immediately while LSP is still loading hangs and then segfaults (especially on slow ones like fish-lsp).
+##### Check each lsp with each of the features implemented
-* [ ] **Navigation Bug:** Fix bug where `Alt+Up` at EOF adds an extra line.
+
 * [ ] **LSP Bug:** Check why `fish-lsp` is behaving so off with completions filtering.
 * [ ] **File IO:** Normalize/validate unicode on file open (enforce UTF-8, handle other types gracefully).
-
+* [ ] **Critical Crash:** Fix bug where closing immediately while LSP is still loading hangs and then segfaults (especially on slow ones like fish-lsp).
+* [ ] **Navigation Bug:** Fix bug where `Alt+Up` at EOF adds an extra line.
+* [ ] **Modularize handle_events functions:** The function is over 700 lines with a lot of repeating blocks. Split into smaller functions.
+* [ ] **Editor Indentation Fix:**
+    * [ ] Keep cache of language maps in engine to reduce lookup time.
+    * [ ] In indents add function to support tab which indents if before any content and inserts a pure \t otherwise.
+    * [ ] And backspace which undents if before any content.
+    * [ ] Add block indentation support.
+    * [ ] Ignore comments/strings (maybe as-set by tree-sitter) when auto-indenting.
+    * [ ] Just use span cursor to avoid strings/comments.. And use another map for c-style single line block and add stuff like operators to it.
+    * [ ] These will dedent when the block immediately after them is dedented
+    * [ ] Dont dedent is ending is valid starting is invalid but also empty
+    * [ ] Just leave asis if starting is empty
+* [ ] **Readme:** Update readme to show indetation mechanics.
+* [ ] **LSP Bug:** Try to find out why emojis are breaking lsp edits. (check the ruby sample)
+* [ ] **UI Refinement:**
+    * [ ] Allow completion list to be scrolled; show only `x` max items.
+    * [ ] Finish autocomplete box style functions.
+* [ ] **Documentation UI:** Capture `Ctrl+h` / `Ctrl+l` for scrolling documentation windows.
+* [ ] **Redo hooks and folding as proper engines**: With functions to checkstate/cursor like function and edits application.
+* [ ] Do trextmate like regex grammar parsing with lsp symbols for semantic highlighting.
+    * Probably remove tre--sitter or just keep it for context tree.
+    * Making bracket matching andignoring strings/comments easier.
 
 ### Core Editing Mechanics
 
 * [ ] **Undo/Redo:** Add support for undo/redo history.
-* [ ] **Indentation Engine (Major Task):**
+
-    * **Startup:**
+* [ ] **Auto brace selection:** Add support for auto brace selection.
-        1. Scan file: Check for lines with 2+ spaces. Least count = indent. If tabs, use tabs.
-        2. Fallback: Use table of file types or default to 2 spaces.
-        3. Store as: `1 = tab`, `2+ = n spaces`.
-        4. Apply: Use this for indent/unindent actions.
-        5. Newline: Follow indent of previous line immediately (ignore default).
-    * **Indent/Unindent:**
-        * Add support for indent/unindent actions.
-        * that use indentation of previous line that is not comment or string or whitespace/blank.
-        * and try auto indent one level extra if previous line ends with colon or bracket start.
-        * and dedent one level extra if previous line ends with bracket end.
-    * **Newline:** Add support for newline actions similar to indent.
 
 * [ ] **Tree-sitter Indent:** Attempt to allow Tree-sitter to handle indentation if possible.
 
 * [ ] **Scrolling:** Add logic where selecting at the end of the screen scrolls down (and vice versa).
     * *Implementation:* Update `main.cc` to send drag events to the selected editor.
 
-* [ ] **Documentation UI:** Capture `Ctrl+h` / `Ctrl+l` for scrolling documentation windows.
-
 
 ### UX
 
@@ -43,10 +52,6 @@ Copyright 2025 Syed Daanish
     * [ ] Stop filtering case-sensitive.
     * [ ] Normalize completion edits if local filtering is used.
 
-* [ ] **UI Refinement:**
-    * [ ] Allow completion list to be scrolled; show only `x` max items.
-    * [ ] Finish autocomplete box style functions.
-
 * [ ] **LSP Features:**
     * [ ] Add LSP jumping support (Go to Definition, Hover).
     * [ ] Add LSP rename support.
@@ -110,6 +115,7 @@ Copyright 2025 Syed Daanish
 * [ ] **Performance:**
     * [ ] Switch JSON parser to `RapidJSON` (or similar high-performance lib).
     * [ ] Decrease usage of `std::string` in UI, LSP, and warnings.
+    * [ ] Also for vectors into managed memory especially for completions/lsp-stuff.
 
 * [ ] **Folding:** Redo folding system and its relation to `move_line_*` functions.
 

include/config.h

@@ -88,6 +88,7 @@ static const std::unordered_map<uint8_t, LSP> kLsps = {
           "--stdio",
           nullptr,
       }}},
+#define LUA_LS 12
     {12,
      {"lua-language-server",
       {
@@ -204,7 +205,6 @@ static const std::unordered_map<std::string, Language> kLanguages = {
     {"query", {"query", LANG(query), 0, 0x7E57C2, " "}},
     {"regex", {"regex", LANG(regex), 0, 0x9E9E9E, ".*"}},
     {"ini", {"ini", LANG(ini), 0, 0x6d8086, " "}},
-
 };
 
 static const std::unordered_map<std::string, std::string> kExtToLang = {
@@ -257,6 +257,7 @@ static const std::unordered_map<std::string, std::string> kExtToLang = {
     {"toml", "toml"},
     {"yaml", "yaml"},
     {"yml", "yaml"},
+    {"clangd", "yaml"},
 };
 
 static const std::unordered_map<std::string, std::string> kMimeToLang = {

include/editor/completions.h

@@ -14,6 +14,7 @@ struct CompletionItem {
   std::optional<std::string> documentation;
   bool is_markup = false;
   bool deprecated = false;
+  bool asis = true;
   std::string sort;
   std::string filter;
   bool snippet = false;
@@ -37,6 +38,7 @@ struct CompletionSession {
   HoverBox hover;
   uint32_t doc = UINT32_MAX;
   std::atomic<bool> hover_dirty = false;
+  int version;
 
   CompletionSession() : box(this) {}
 };

include/editor/editor.h

@@ -2,6 +2,7 @@
 #define EDITOR_H
 
 #include "editor/completions.h"
+#include "editor/indents.h"
 #include "editor/spans.h"
 #include "io/knot.h"
 #include "io/sysio.h"
@@ -52,8 +53,9 @@ struct Editor {
   HoverBox hover;
   bool diagnostics_active;
   DiagnosticBox diagnostics;
-  int lsp_version = 1;
+  std::atomic<int> lsp_version = 1;
   CompletionSession completion;
+  IndentationEngine indents;
 };
 
 Editor *new_editor(const char *filename_arg, Coord position, Coord size);
@@ -72,8 +74,6 @@ void cursor_right(Editor *editor, uint32_t number);
 void scroll_up(Editor *editor, int32_t number);
 void scroll_down(Editor *editor, uint32_t number);
 void ensure_cursor(Editor *editor);
-void indent_line(Editor *editor, uint32_t row);
-void dedent_line(Editor *editor, uint32_t row);
 void ensure_scroll(Editor *editor);
 void handle_editor_event(Editor *editor, KeyEvent event);
 void edit_erase(Editor *editor, Coord pos, int64_t len);
@@ -93,9 +93,6 @@ void word_boundaries_exclusive(Editor *editor, Coord coord, uint32_t *prev_col,
 std::vector<Fold>::iterator find_fold_iter(Editor *editor, uint32_t line);
 bool add_fold(Editor *editor, uint32_t start, uint32_t end);
 bool remove_fold(Editor *editor, uint32_t line);
-uint32_t leading_indent(const char *line, uint32_t len);
-uint32_t get_indent(Editor *editor, Coord cursor);
-bool closing_after_cursor(const char *line, uint32_t len, uint32_t col);
 void editor_lsp_handle(Editor *editor, json msg);
 void apply_lsp_edits(Editor *editor, std::vector<TextEdit> edits, bool move);
 void completion_resolve_doc(Editor *editor);

include/editor/indents.h

@@ -0,0 +1,152 @@
+#ifndef EDITOR_INDENTS_H
+#define EDITOR_INDENTS_H
+
+#include "utils/utils.h"
+
+static const std::unordered_map<std::string, uint8_t> kLangtoIndent = {
+    {"make", 1}, {"yaml", 2}};
+
+// this indents the newline one level when the line (on the curser before \n  is
+// inserted) matches this at its end (stripped of whitespace)
+static const std::unordered_map<std::string, const std::vector<std::string>>
+    kLangtoBlockStartsEnd = {
+        {"bash", {"then", "do", "in", "{", "(", "\\", "&&", "||", "|"}},
+        {"c", {"{", "(", ":"}},
+        {"cpp", {"{", "(", ":"}},
+        {"h", {"{", "(", ":"}},
+        {"css", {"{", "("}},
+        {"fish", {"{", "(", "^", "&&", "||", "|"}},
+        {"go", {"{", "(", ":"}},
+        {"gomod", {"{", "(", ":"}},
+        {"haskell", {"do", "where", "then", "else", "of"}},
+        {"javascript", {"{", "(", "[", ":"}},
+        {"typescript", {"{", "(", "[", ":"}},
+        {"json", {"{", "[", ":"}},
+        {"jsonc", {"{", "[", ":"}},
+        {"ruby", {"then", "else", "begin", "{", "(", "["}},
+        {"lua", {"then", "do", "else", "repeat", "{", "(", "["}},
+        {"python", {":", "(", "[", "{"}},
+        {"rust", {"{", "(", "[", ":"}},
+        {"php", {"{", "(", "[", ":"}},
+        {"nginx", {"{"}},
+        {"yaml", {":"}},
+        {"sql", {"("}},
+        {"make", {":"}},
+        {"gdscript", {":", "(", "[", "{"}},
+};
+
+// this indents the newline one level when the line (on the curser before \n is
+// inserted) matches this at its start (stripped of whitespace)
+static const std::unordered_map<std::string, const std::vector<std::string>>
+    kLangtoBlockStartsStart = {
+        {"c", {"if", "for", "while"}},
+        {"cpp", {"if", "for", "while"}},
+        {"h", {"if", "for", "while"}},
+        {"fish", {"if", "else", "for", "while", "switch", "case", "function"}},
+        {"javascript", {"if", "for", "while"}},
+        {"typescript", {"if", "for", "while"}},
+        {"ruby",
+         {"if", "do", "when", "rescue", "class", "module", "def", "unless",
+          "until", "elsif", "ensure"}},
+        {"lua", {"function"}},
+        {"nginx", {"{"}},
+};
+
+// This dedents the line (under the cursor before \n is inserted) when the line
+// matches this fully (stripped of whitespace)
+static const std::unordered_map<std::string, const std::vector<std::string>>
+    kLangtoBlockEndsFull = {
+        {"bash", {"fi", "done", "esac", "}", ")"}},
+        {"c", {"}", ")"}},
+        {"cpp", {"}", ")"}},
+        {"h", {"}", ")"}},
+        {"css", {"}", ")"}},
+        {"fish", {"end"}},
+        {"go", {"}", ")"}},
+        {"gomod", {"}", ")"}},
+        {"javascript", {"}", ")", "]"}},
+        {"typescript", {"}", ")", "]"}},
+        {"json", {"}", "]"}},
+        {"jsonc", {"}", "]"}},
+        {"ruby", {"end", "else", "}", ")", "]"}},
+        {"lua", {"else", "}", ")", "]"}},
+        {"python", {"}", ")", "]", "else:"}},
+        {"rust", {"}", ")", "]"}},
+        {"php",
+         {"}", ")", "]", "else:", "endif;", "endfor;", "endwhile;",
+          "endswitch;", "endcase;", "endfunction;"}},
+        {"nginx", {"}"}},
+        {"sql", {")"}},
+        {"gdscript", {"}", ")", "]"}},
+};
+
+// This dedents the line (under the cursor before \n is inserted) when the line
+// matches this at its start (stripped of whitespace)
+static const std::unordered_map<std::string, const std::vector<std::string>>
+    kLangtoBlockEndsStart = {
+        {"c", {"case", "default:", "} else"}},
+        {"cpp", {"case", "default:", "} else"}},
+        {"h", {"case", "default:", "} else"}},
+        {"fish", {"else if"}},
+        {"go", {"case", "default:", "} else"}},
+        {"gomod", {"}", ")"}},
+        {"javascript", {"case", "default:"}},
+        {"typescript", {"case", "default:"}},
+        {"json", {"}", "]"}},
+        {"python", {"elif"}},
+        {"jsonc", {"}", "]"}},
+        {"ruby", {"when", "elsif", "rescue", "ensure"}},
+        {"lua", {"end", "elseif", "until"}},
+        {"rust", {"case", "default:", "} else"}},
+        {"php", {"case", "default:", "} else"}},
+};
+
+struct IndentationEngine {
+  // tabs = 1, spaces = 2+
+  uint8_t indent = 0;
+  struct Editor *editor = nullptr;
+
+  void compute_indent(Editor *n_editor);
+  void insert_new_line(Coord cursor);
+  void insert_tab(Coord cursor);
+  uint32_t set_indent(uint32_t row, int64_t indent_level);
+  uint32_t indent_line(uint32_t row);
+  uint32_t dedent_line(uint32_t row);
+  void indent_block(uint32_t start, uint32_t end);
+  void dedent_block(uint32_t start, uint32_t end);
+  // fixes a autocomplete block's indentation
+  char *block_to_asis(Coord cursor, std::string source, uint32_t *out_len);
+
+private:
+  // TODO: Ignore comments/strings too
+  // returns the indent level of the line itself or of the previous non-empty
+  uint32_t indent_expected(uint32_t row);
+  // returns the indent level of the line
+  uint32_t indent_real(char *line, uint32_t len);
+};
+
+inline static bool ends_with(const std::string &str,
+                             const std::string &suffix) {
+  const size_t str_len = str.size();
+  const size_t suf_len = suffix.size();
+  if (suf_len > str_len)
+    return false;
+  for (size_t i = 0; i < suf_len; i++)
+    if (str[str_len - suf_len + i] != suffix[i])
+      return false;
+  return true;
+}
+
+inline static bool starts_with(const std::string &str,
+                               const std::string &prefix) {
+  const size_t str_len = str.size();
+  const size_t pre_len = prefix.size();
+  if (pre_len > str_len)
+    return false;
+  for (size_t i = 0; i < pre_len; i++)
+    if (str[i] != prefix[i])
+      return false;
+  return true;
+}
+
+#endif

include/lsp/lsp.h

@@ -67,7 +67,7 @@ static json client_capabilities = {
           {"resolveSupport", {{"properties", {"documentation"}}}},
           {"insertReplaceSupport", true},
           {"labelDetailsSupport", true},
-          {"insertTextModeSupport", {{"valueSet", {1}}}},
+          {"insertTextModeSupport", {{"valueSet", {1, 2}}}},
           {"deprecatedSupport", true}}},
         {"completionItemKind",
          {{"valueSet", {1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11, 12, 13,

include/utils/utils.h

@@ -61,6 +61,7 @@ struct Match {
 
 #define MAX(a, b) ((a) > (b) ? (a) : (b))
 #define MIN(a, b) ((a) < (b) ? (a) : (b))
+#define ABS(x) ((x) < 0 ? -(x) : (x))
 #define UNUSED(x) (void)(x)
 #define USING(x) UNUSED(sizeof(x))
 
@@ -69,14 +70,16 @@ std::string percent_encode(const std::string &s);
 std::string percent_decode(const std::string &s);
 uint32_t count_clusters(const char *line, size_t len, size_t from, size_t to);
 std::string trim(const std::string &s);
+std::string substitute_fence(const std::string &documentation,
+                             const std::string &lang);
 
 int display_width(const char *str, size_t len);
 uint32_t get_visual_col_from_bytes(const char *line, uint32_t len,
                                    uint32_t byte_limit);
 uint32_t get_bytes_from_visual_col(const char *line, uint32_t len,
                                    uint32_t target_visual_col);
-int utf8_byte_offset_to_utf16(const char *s, size_t byte_pos);
+uint32_t utf8_byte_offset_to_utf16(const char *s, uint32_t byte_pos);
-size_t utf16_offset_to_utf8(const char *s, int utf16_pos);
+uint32_t utf16_offset_to_utf8(const char *s, uint32_t utf16_pos);
 
 void log(const char *fmt, ...);
 

samples/python.py

@@ -12,6 +12,7 @@ NotImplemented
 Ellipsis
 __name__  # builtin constant
 
+
 # ==============================
 # Imports
 # ==============================
@@ -71,7 +72,7 @@ GLOBAL_VAR = 3
 
 # Builtin variable references
 self = "something"
-cls = "class"
+cls = "dj"
 
 # ==============================
 # Control flow

samples/ruby.rb

@@ -18,7 +18,7 @@ cjk_samples = [
   "測試中文字串",
   "한국어 테스트",
   "ひらがなカタカナ😀混合",
-  "大量の文字列🚀🚀🚀",
+  "大量の文字列🚀🚀🚀"
 ]
 
 # Ruby regex with unicode

scripts/exit.sh

@@ -1 +1 @@
-kitty @ --to="$KITTY_LISTEN_ON" set-spacing padding=8 margin=0 2>/dev/null || true
+# kitty @ --to="$KITTY_LISTEN_ON" set-spacing padding=8 margin=0 2>/dev/null || true

scripts/init.sh

@@ -1 +1 @@
-kitty @ --to="$KITTY_LISTEN_ON" set-spacing padding=0 margin=0 2>/dev/null || true
+# kitty @ --to="$KITTY_LISTEN_ON" set-spacing padding=0 margin=0 2>/dev/null || true

src/editor/completions.cc

@@ -5,7 +5,6 @@
 #include "lsp/lsp.h"
 #include "main.h"
 #include "utils/utils.h"
-#include <regex>
 
 inline static std::string completion_prefix(Editor *editor) {
   Coord hook = editor->completion.hook;
@@ -58,6 +57,7 @@ void completion_request(Editor *editor) {
   editor->completion.items.clear();
   editor->completion.visible.clear();
   editor->completion.select = 0;
+  editor->completion.version = editor->lsp_version;
   LSPPending *pending = new LSPPending();
   pending->editor = editor;
   pending->method = "textDocument/completion";
@@ -67,6 +67,7 @@ void completion_request(Editor *editor) {
     std::vector<json> items_json;
     std::vector<char> end_chars_def;
     int insert_text_format = 1;
+    int insert_text_mode = 1;
     if (message.contains("result")) {
       auto &result = message["result"];
       if (result.is_array()) {
@@ -87,6 +88,9 @@ void completion_request(Editor *editor) {
           if (defs.contains("insertTextFormat") &&
               defs["insertTextFormat"].is_number())
             insert_text_format = defs["insertTextFormat"].get<int>();
+          if (defs.contains("insertTextMode") &&
+              defs["insertTextMode"].is_number())
+            insert_text_mode = defs["insertTextMode"].get<int>();
           if (defs.contains("textEdit"))
             if (defs["textEdit"].is_array())
               for (auto &c : defs["textEdit"]) {
@@ -119,10 +123,11 @@ void completion_request(Editor *editor) {
               "markdown";
           std::string documentation =
               item_json["documentation"]["value"].get<std::string>();
+          if (documentation.size() > 1024)
+            item.is_markup = false;
           if (item.is_markup) {
-            static const std::regex fence_no_lang("```(\\s*\\n)");
+            item.documentation =
-            item.documentation = std::regex_replace(
+                substitute_fence(documentation, editor->lang.name);
-                documentation, fence_no_lang, "```" + editor->lang.name + "$1");
           } else {
             item.documentation = documentation;
           }
@@ -193,6 +198,8 @@ void completion_request(Editor *editor) {
       item.snippet = insert_text_format == 2;
       if (item_json.contains("insertTextFormat"))
         item.snippet = item_json["insertTextFormat"].get<int>() == 2;
+      if (item_json.contains("insertTextMode"))
+        item.asis = item_json["insertTextMode"].get<int>() == 1;
       if (item_json.contains("commitCharacters"))
         for (auto &c : item_json["commitCharacters"])
           if (c.is_string() && c.get<std::string>().size() == 1)
@@ -301,9 +308,6 @@ void handle_completion(Editor *editor, KeyEvent event) {
           else
             completion_request(editor);
         }
-      } else {
-        editor->completion.trigger = 3;
-        completion_request(editor);
       }
     } else {
       editor->completion.active = false;
@@ -351,10 +355,11 @@ void completion_resolve_doc(Editor *editor) {
             "markdown";
         std::string documentation =
             message["result"]["documentation"]["value"].get<std::string>();
+        if (documentation.size() > 1024)
+          item.is_markup = false;
         if (item.is_markup) {
-          static const std::regex fence_no_lang("```(\\s*\\n)");
+          item.documentation =
-          item.documentation = std::regex_replace(
+              substitute_fence(documentation, editor->lang.name);
-              documentation, fence_no_lang, "```" + editor->lang.name + "$1");
         } else {
           item.documentation = documentation;
         }
@@ -372,7 +377,9 @@ void complete_accept(Editor *editor) {
   if (!editor->completion.active || editor->completion.box.hidden)
     return;
   auto &item = editor->completion.items[editor->completion.select];
-  // TODO: support snippets here maybe?
+  // TODO: support snippets and asis here
+  //       once indentation engine is implemented
+  if (editor->completion.version != editor->lsp_version) {
     int delta_col = 0;
     TextEdit &e = item.edits[0];
     if (e.end.row == editor->cursor.row) {
@@ -387,6 +394,7 @@ void complete_accept(Editor *editor) {
         }
       }
     }
+  }
   apply_lsp_edits(editor, item.edits, true);
   editor->completion.active = false;
 }

src/editor/edit.cc

@@ -286,10 +286,10 @@ void edit_replace(Editor *editor, Coord start, Coord end, const char *text,
   uint32_t start_byte =
       line_to_byte(editor->root, start.row, nullptr) + start.col;
   uint32_t end_byte = line_to_byte(editor->root, end.row, nullptr) + end.col;
-  lock.unlock();
   char *buf = read(editor->root, start_byte, end_byte - start_byte);
   if (!buf)
     return;
+  lock.unlock();
   uint32_t erase_len =
       count_clusters(buf, end_byte - start_byte, 0, end_byte - start_byte);
   free(buf);

src/editor/editor.cc

@@ -38,6 +38,7 @@ Editor *new_editor(const char *filename_arg, Coord position, Coord size) {
   }
   if (len <= (1024 * 28))
     request_add_to_lsp(editor->lang, editor);
+  editor->indents.compute_indent(editor);
   return editor;
 }
 
@@ -69,14 +70,15 @@ void save_file(Editor *editor) {
   if (!editor || !editor->root)
     return;
   std::shared_lock lock(editor->knot_mtx);
+  int version = editor->lsp_version;
   char *str = read(editor->root, 0, editor->root->char_count);
   if (!str)
     return;
+  lock.unlock();
   std::ofstream out(editor->filename);
   out.write(str, editor->root->char_count);
   out.close();
   free(str);
-  lock.unlock();
   if (editor->lsp) {
     json save_msg = {{"jsonrpc", "2.0"},
                      {"method", "textDocument/didSave"},
@@ -95,8 +97,10 @@ void save_file(Editor *editor) {
       LSPPending *pending = new LSPPending();
       pending->editor = editor;
       pending->method = "textDocument/formatting";
-      pending->callback = [save_msg](Editor *editor, std::string,
+      pending->callback = [save_msg, version](Editor *editor, std::string,
                                               json message) {
+        if (version != editor->lsp_version)
+          return;
         auto &edits = message["result"];
         if (edits.is_array()) {
           std::vector<TextEdit> t_edits;
@@ -113,10 +117,11 @@ void save_file(Editor *editor) {
           }
           apply_lsp_edits(editor, t_edits, false);
           ensure_scroll(editor);
-          std::unique_lock lock(editor->knot_mtx);
+          std::shared_lock lock(editor->knot_mtx);
           char *str = read(editor->root, 0, editor->root->char_count);
           if (!str)
             return;
+          lock.unlock();
           std::ofstream out(editor->filename);
           out.write(str, editor->root->char_count);
           free(str);

src/editor/events.cc

@@ -2,6 +2,8 @@
 #include "editor/folds.h"
 #include "lsp/lsp.h"
 #include "main.h"
+#include "utils/utils.h"
+#include <cstdint>
 
 void handle_editor_event(Editor *editor, KeyEvent event) {
   static std::chrono::steady_clock::time_point last_click_time =
@@ -21,11 +23,11 @@ void handle_editor_event(Editor *editor, KeyEvent event) {
     case SCROLL:
       switch (event.mouse_direction) {
       case SCROLL_UP:
-        scroll_up(editor, 10);
+        scroll_up(editor, 4);
         ensure_cursor(editor);
         break;
       case SCROLL_DOWN:
-        scroll_down(editor, 10);
+        scroll_down(editor, 4);
         ensure_cursor(editor);
         break;
       case SCROLL_LEFT:
@@ -355,13 +357,17 @@ void handle_editor_event(Editor *editor, KeyEvent event) {
         ensure_cursor(editor);
         break;
       case '>':
-      case '.':
+      case '.': {
-        indent_line(editor, editor->cursor.row);
+        uint32_t delta = editor->indents.indent_line(editor->cursor.row);
-        break;
+        editor->cursor.col = start.col + delta;
+        editor->cursor.row = start.row;
+      } break;
       case '<':
-      case ',':
+      case ',': {
-        dedent_line(editor, editor->cursor.row);
+        uint32_t delta = editor->indents.dedent_line(editor->cursor.row);
-        break;
+        editor->cursor.col = MAX((int64_t)start.col - delta, 0);
+        editor->cursor.row = start.row;
+      } break;
       case CTRL('s'):
         save_file(editor);
         break;
@@ -385,33 +391,7 @@ void handle_editor_event(Editor *editor, KeyEvent event) {
           edit_insert(editor, editor->cursor, (char *)"  ", 2);
           cursor_right(editor, 2);
         } else if (event.c[0] == '\n' || event.c[0] == '\r') {
-          uint32_t line_len = 0;
+          editor->indents.insert_new_line(editor->cursor);
-          LineIterator *it = begin_l_iter(editor->root, editor->cursor.row);
-          char *line = next_line(it, &line_len);
-          bool closing = false;
-          if (line && line_len > 0 && line[line_len - 1] == '\n')
-            line_len--;
-          uint32_t indent = get_indent(editor, editor->cursor);
-          if (line) {
-            if (indent == 0)
-              indent = leading_indent(line, line_len);
-            closing = closing_after_cursor(line, line_len, editor->cursor.col);
-          }
-          free(it->buffer);
-          free(it);
-          uint32_t closing_indent =
-              indent >= INDENT_WIDTH ? indent - INDENT_WIDTH : 0;
-          std::string insert_text("\n");
-          insert_text.append(indent, ' ');
-          Coord new_cursor = {editor->cursor.row + 1, indent};
-          if (closing) {
-            insert_text.push_back('\n');
-            insert_text.append(closing_indent, ' ');
-          }
-          edit_insert(editor, editor->cursor, insert_text.data(),
-                      insert_text.size());
-          editor->cursor = new_cursor;
-          editor->cursor_preffered = UINT32_MAX;
         } else if (event.c[0] == CTRL('W')) {
           uint32_t prev_col_byte, prev_col_cluster;
           word_boundaries(editor, editor->cursor, &prev_col_byte, nullptr,
@@ -424,8 +404,15 @@ void handle_editor_event(Editor *editor, KeyEvent event) {
           char c = event.c[0];
           uint32_t col = editor->cursor.col;
           LineIterator *it = begin_l_iter(editor->root, editor->cursor.row);
+          if (!it)
+            return;
           uint32_t len;
           char *line = next_line(it, &len);
+          if (!line) {
+            free(it->buffer);
+            free(it);
+            return;
+          }
           bool skip_insert = false;
           if (line && col < len) {
             char next = line[col];
@@ -465,6 +452,67 @@ void handle_editor_event(Editor *editor, KeyEvent event) {
               edit_insert(editor, editor->cursor, &c, 1);
               cursor_right(editor, 1);
             }
+            if (editor->lsp && editor->lsp->allow_formatting_on_type) {
+              for (char ch : editor->lsp->format_chars) {
+                if (ch == c) {
+                  LineIterator *it =
+                      begin_l_iter(editor->root, editor->cursor.row);
+                  if (!it)
+                    return;
+                  char *line = next_line(it, nullptr);
+                  if (!line) {
+                    free(it->buffer);
+                    free(it);
+                    return;
+                  }
+                  uint32_t col =
+                      utf8_byte_offset_to_utf16(line, editor->cursor.col);
+                  free(it->buffer);
+                  free(it);
+                  int version = editor->lsp_version;
+                  json message = {
+                      {"jsonrpc", "2.0"},
+                      {"method", "textDocument/onTypeFormatting"},
+                      {"params",
+                       {{"textDocument", {{"uri", editor->uri}}},
+                        {"position",
+                         {{"line", editor->cursor.row}, {"character", col}}},
+                        {"ch", std::string(1, c)},
+                        {"options",
+                         {{"tabSize", 2},
+                          {"insertSpaces", true},
+                          {"trimTrailingWhitespace", true},
+                          {"trimFinalNewlines", true}}}}}};
+                  LSPPending *pending = new LSPPending();
+                  pending->editor = editor;
+                  pending->method = "textDocument/onTypeFormatting";
+                  pending->callback = [version](Editor *editor, std::string,
+                                                json message) {
+                    if (version != editor->lsp_version)
+                      return;
+                    auto &edits = message["result"];
+                    if (edits.is_array()) {
+                      std::vector<TextEdit> t_edits;
+                      t_edits.reserve(edits.size());
+                      for (auto &edit : edits) {
+                        TextEdit t_edit;
+                        t_edit.text = edit.value("newText", "");
+                        t_edit.start.row = edit["range"]["start"]["line"];
+                        t_edit.start.col = edit["range"]["start"]["character"];
+                        t_edit.end.row = edit["range"]["end"]["line"];
+                        t_edit.end.col = edit["range"]["end"]["character"];
+                        utf8_normalize_edit(editor, &t_edit);
+                        t_edits.push_back(t_edit);
+                      }
+                      apply_lsp_edits(editor, t_edits, false);
+                      ensure_scroll(editor);
+                    }
+                  };
+                  lsp_send(editor->lsp, message, pending);
+                  break;
+                }
+              }
+            }
           }
         } else if (event.c[0] == 0x7F || event.c[0] == 0x08) {
           Coord prev_pos = editor->cursor;

src/editor/indents.cc

@@ -1,87 +1,376 @@
 #include "editor/editor.h"
+#include "io/knot.h"
+#include "lsp/lsp.h"
 
-uint32_t leading_indent(const char *line, uint32_t len) {
+void IndentationEngine::compute_indent(Editor *n_editor) {
-  uint32_t indent = 0;
+  editor = n_editor;
+  uint32_t line_idx = 0;
+  LineIterator *it = begin_l_iter(editor->root, 0);
+  if (!it)
+    return;
+  auto is_set = kLangtoIndent.find(editor->lang.name);
+  indent = is_set != kLangtoIndent.end() ? is_set->second : 0;
+  while (indent == 0 && line_idx < 64) {
+    uint32_t len;
+    char *line = next_line(it, &len);
+    if (!line)
+      break;
+    line_idx++;
+    if (len > 0 && line[len - 1] == '\n')
+      --len;
+    if (len == 0)
+      continue;
+    if (line[0] == '\t') {
+      indent = 1;
+    } else if (line[0] == ' ') {
       for (uint32_t i = 0; i < len; i++) {
-    if (line[i] == ' ')
+        if (line[i] == ' ') {
-      indent++;
+          indent += 1;
-    else if (line[i] == '\t')
+        } else {
-      indent += INDENT_WIDTH;
+          if (indent == 1)
-    else
+            indent = 0;
           break;
         }
-  return indent;
+      }
+    }
+  }
+  if (indent == 0)
+    indent = 2;
+  free(it->buffer);
+  free(it);
 }
 
-uint32_t get_indent(Editor *editor, Coord cursor) {
+uint32_t IndentationEngine::indent_real(char *line, uint32_t len) {
-  if (!editor)
+  uint32_t spaces = 0;
+  uint32_t tabs = 0;
+  for (uint32_t i = 0; i < len; i++) {
+    if (line[i] == ' ') {
+      spaces += 1;
+    } else if (line[i] == '\t') {
+      tabs += 1;
+    } else {
+      break;
+    }
+  }
+  return tabs ? tabs : spaces / indent;
+}
+
+uint32_t IndentationEngine::indent_expected(uint32_t row) {
+  std::shared_lock lock(editor->knot_mtx);
+  uint32_t line_idx = row;
+  if (row == 0)
+    return 0;
+  LineIterator *it = begin_l_iter(editor->root, row - 1);
+  if (!it)
     return 0;
-  LineIterator *it = begin_l_iter(editor->root, cursor.row);
   next_line(it, nullptr);
-  uint32_t line_len;
+  uint32_t c_indent = 0;
-  char *line;
+  while (line_idx--) {
-  while ((line = prev_line(it, &line_len)) != nullptr) {
+    uint32_t len;
-    if (line_len == 0)
+    char *line = prev_line(it, &len);
+    if (!line)
+      break;
+    if (len > 0 && line[len - 1] == '\n')
+      --len;
+    if (len == 0)
       continue;
-    uint32_t indent = leading_indent(line, line_len);
+    c_indent = indent_real(line, len);
+    auto is_end_set = kLangtoBlockStartsEnd.find(editor->lang.name);
+    auto is_start_set = kLangtoBlockStartsStart.find(editor->lang.name);
+    bool is_end = false;
+    if (is_end_set != kLangtoBlockStartsEnd.end())
+      for (auto end : is_end_set->second)
+        if (ends_with(line, end)) {
+          c_indent++;
+          is_end = true;
+          break;
+        }
+    if (!is_end && is_start_set != kLangtoBlockStartsStart.end())
+      for (auto end : is_start_set->second)
+        if (starts_with(line, end)) {
+          c_indent++;
+          break;
+        }
+    break;
+  }
   free(it->buffer);
   free(it);
-    return indent;
+  return c_indent;
 }
+
+uint32_t IndentationEngine::set_indent(uint32_t row, int64_t new_indent) {
+  std::shared_lock lock(editor->knot_mtx);
+  LineIterator *it = begin_l_iter(editor->root, row);
+  if (!it)
+    return 0;
+  uint32_t len;
+  char *line = next_line(it, &len);
+  if (!line) {
     free(it->buffer);
     free(it);
     return 0;
   }
-
+  if (len > 0 && line[len - 1] == '\n')
-bool closing_after_cursor(const char *line, uint32_t len, uint32_t col) {
+    --len;
-  uint32_t i = col;
+  lock.unlock();
-  while (i < len && (line[i] == ' ' || line[i] == '\t'))
+  if (new_indent <= 0)
-    i++;
+    new_indent = 0;
-  if (i >= len)
+  uint32_t ws_len = 0;
-    return false;
+  while (ws_len < len && (line[ws_len] == ' ' || line[ws_len] == '\t'))
-  return line[i] == '}' || line[i] == ']' || line[i] == ')';
+    ws_len++;
+  std::string new_ws;
+  if (indent == 1)
+    new_ws.assign(new_indent, '\t');
+  else
+    new_ws.assign(new_indent * indent, ' ');
+  Coord start = {row, 0};
+  Coord end = {row, ws_len};
+  edit_replace(editor, start, end, new_ws.c_str(), new_ws.length());
+  free(it->buffer);
+  free(it);
+  return len - ws_len + (new_indent * indent);
 }
 
-void indent_line(Editor *editor, uint32_t row) {
+uint32_t IndentationEngine::indent_line(uint32_t row) {
-  if (!editor)
+  std::shared_lock lock(editor->knot_mtx);
-    return;
   LineIterator *it = begin_l_iter(editor->root, row);
-  uint32_t line_len;
+  if (!it)
-  char *line = next_line(it, &line_len);
+    return 0;
+  uint32_t len;
+  char *line = next_line(it, &len);
+  if (!line) {
+    free(it->buffer);
+    free(it);
+    return 0;
+  }
+  lock.unlock();
+  if (len > 0 && line[len - 1] == '\n')
+    --len;
+  uint32_t new_indent = indent_real(line, len) + 1;
+  uint32_t ws_len = 0;
+  while (ws_len < len && (line[ws_len] == ' ' || line[ws_len] == '\t'))
+    ws_len++;
+  std::string new_ws;
+  if (indent == 1)
+    new_ws.assign(new_indent, '\t');
+  else
+    new_ws.assign(new_indent * indent, ' ');
+  edit_replace(editor, {row, 0}, {row, ws_len}, new_ws.c_str(),
+               new_indent * indent);
+  free(it->buffer);
+  free(it);
+  return (uint32_t)ABS((int64_t)ws_len - (new_indent * indent));
+}
+
+uint32_t IndentationEngine::dedent_line(uint32_t row) {
+  std::shared_lock lock(editor->knot_mtx);
+  LineIterator *it = begin_l_iter(editor->root, row);
+  if (!it)
+    return 0;
+  uint32_t len;
+  char *line = next_line(it, &len);
+  if (!line) {
+    free(it->buffer);
+    free(it);
+    return 0;
+  }
+  lock.unlock();
+  if (len > 0 && line[len - 1] == '\n')
+    --len;
+  int64_t new_indent = (int64_t)indent_real(line, len) - 1;
+  if (new_indent < 0)
+    new_indent = 0;
+  uint32_t ws_len = 0;
+  while (ws_len < len && (line[ws_len] == ' ' || line[ws_len] == '\t'))
+    ws_len++;
+  std::string new_ws;
+  if (indent == 1)
+    new_ws.assign(new_indent, '\t');
+  else
+    new_ws.assign(new_indent * indent, ' ');
+  edit_replace(editor, {row, 0}, {row, ws_len}, new_ws.c_str(),
+               new_indent * indent);
+  free(it->buffer);
+  free(it);
+  return (uint32_t)ABS((int64_t)ws_len - (new_indent * indent));
+}
+
+void IndentationEngine::insert_new_line(Coord cursor) {
+  std::string formatted;
+  std::shared_lock lock(editor->knot_mtx);
+  LineIterator *it = begin_l_iter(editor->root, cursor.row);
+  if (!it)
+    return;
+  uint32_t len;
+  char *line = next_line(it, &len);
   if (!line) {
     free(it->buffer);
     free(it);
     return;
   }
-  char *spaces = (char *)malloc(INDENT_WIDTH);
+  lock.unlock();
-  memset(spaces, ' ', INDENT_WIDTH);
+  if (len > 0 && line[len - 1] == '\n')
-  Coord cursor = editor->cursor;
+    --len;
-  edit_insert(editor, {row, 0}, spaces, INDENT_WIDTH);
+  if (cursor.col >= len) {
-  editor->cursor = cursor;
+    auto is_end_full = kLangtoBlockEndsFull.find(editor->lang.name);
-  free(spaces);
+    auto is_end_start = kLangtoBlockEndsStart.find(editor->lang.name);
+    bool end_matched = false;
+    if (is_end_full != kLangtoBlockEndsFull.end())
+      for (auto end : is_end_full->second)
+        if (end == trim(line)) {
+          cursor.col =
+              set_indent(cursor.row, (int64_t)indent_expected(cursor.row) - 1);
+          end_matched = true;
+          break;
+        }
+    if (!end_matched && is_end_start != kLangtoBlockEndsStart.end())
+      for (auto end : is_end_start->second)
+        if (starts_with(trim(line), end)) {
+          cursor.col =
+              set_indent(cursor.row, (int64_t)indent_expected(cursor.row) - 1);
+          break;
+        }
+    lock.lock();
     free(it->buffer);
     free(it);
-}
+    it = begin_l_iter(editor->root, cursor.row);
-
+    if (!it)
-void dedent_line(Editor *editor, uint32_t row) {
-  if (!editor)
       return;
-  LineIterator *it = begin_l_iter(editor->root, row);
+    line = next_line(it, &len);
-  uint32_t line_len;
-  char *line = next_line(it, &line_len);
     if (!line) {
       free(it->buffer);
       free(it);
       return;
     }
-  uint32_t indent = leading_indent(line, line_len);
+    if (len > 0 && line[len - 1] == '\n')
-  if (indent == 0) {
+      --len;
+    lock.unlock();
+  }
+  std::string ending = trim(std::string(line + cursor.col, len - cursor.col));
+  std::string before = trim(std::string(line, cursor.col));
+  uint32_t c_indent = indent_real(line, len);
+  if (!ending.empty()) {
+    bool ending_valid = false;
+    bool starting_valid = false;
+    auto is_end_full = kLangtoBlockEndsFull.find(editor->lang.name);
+    auto is_end_start = kLangtoBlockEndsStart.find(editor->lang.name);
+    auto is_end_set = kLangtoBlockStartsEnd.find(editor->lang.name);
+    auto is_start_set = kLangtoBlockStartsStart.find(editor->lang.name);
+    if (is_end_full != kLangtoBlockEndsFull.end())
+      for (auto end : is_end_full->second)
+        if (ending == end) {
+          ending_valid = true;
+          break;
+        }
+    if (!ending_valid && is_end_start != kLangtoBlockEndsStart.end())
+      for (auto end : is_end_start->second)
+        if (starts_with(ending, end)) {
+          ending_valid = true;
+          break;
+        }
+    if (is_end_set != kLangtoBlockStartsEnd.end())
+      for (auto end : is_end_set->second)
+        if (ends_with(before, end)) {
+          c_indent++;
+          starting_valid = true;
+          break;
+        }
+    if (!starting_valid && is_start_set != kLangtoBlockStartsStart.end())
+      for (auto end : is_start_set->second)
+        if (starts_with(before, end)) {
+          c_indent++;
+          break;
+        }
+    if (ending_valid && starting_valid)
+      ending = "\n" +
+               (indent == 1 ? std::string(c_indent, '\t')
+                            : std::string(c_indent * indent, ' ')) +
+               ending;
+    else if (ending_valid)
+      c_indent--;
+  }
+  auto is_end_set = kLangtoBlockStartsEnd.find(editor->lang.name);
+  auto is_start_set = kLangtoBlockStartsStart.find(editor->lang.name);
+  bool is_end = false;
+  if (is_end_set != kLangtoBlockStartsEnd.end())
+    for (auto end : is_end_set->second)
+      if (ends_with(before, end)) {
+        c_indent++;
+        is_end = true;
+        break;
+      }
+  if (!is_end && is_start_set != kLangtoBlockStartsStart.end())
+    for (auto end : is_start_set->second)
+      if (starts_with(before, end)) {
+        c_indent++;
+        break;
+      }
+  formatted = "\n" +
+              (indent == 1 ? std::string(c_indent, '\t')
+                           : std::string(c_indent * indent, ' ')) +
+              ending;
+  Coord new_cursor = {cursor.row + 1, c_indent * indent};
+  edit_replace(editor, cursor, {cursor.row, len}, formatted.data(),
+               formatted.size());
+  editor->cursor = new_cursor;
+  editor->cursor_preffered = UINT32_MAX;
+  free(it->buffer);
+  free(it);
+  if (!editor->lsp || !editor->lsp->allow_formatting_on_type)
+    return;
+  for (char ch : editor->lsp->format_chars) {
+    if (ch == '\n') {
+      LineIterator *it = begin_l_iter(editor->root, editor->cursor.row);
+      if (!it)
+        return;
+      char *line = next_line(it, nullptr);
+      if (!line) {
         free(it->buffer);
         free(it);
         return;
       }
-  uint32_t remove = indent >= INDENT_WIDTH ? INDENT_WIDTH : indent;
+      uint32_t col = utf8_byte_offset_to_utf16(line, editor->cursor.col);
-  edit_erase(editor, {row, 0}, remove);
       free(it->buffer);
       free(it);
+      int version = editor->lsp_version;
+      json message = {
+          {"jsonrpc", "2.0"},
+          {"method", "textDocument/onTypeFormatting"},
+          {"params",
+           {{"textDocument", {{"uri", editor->uri}}},
+            {"position", {{"line", editor->cursor.row}, {"character", col}}},
+            {"ch", std::string(1, ch)},
+            {"options",
+             {{"tabSize", 2},
+              {"insertSpaces", true},
+              {"trimTrailingWhitespace", true},
+              {"trimFinalNewlines", true}}}}}};
+      LSPPending *pending = new LSPPending();
+      pending->editor = editor;
+      pending->method = "textDocument/onTypeFormatting";
+      pending->callback = [version](Editor *editor, std::string, json message) {
+        if (version != editor->lsp_version)
+          return;
+        auto &edits = message["result"];
+        if (edits.is_array()) {
+          std::vector<TextEdit> t_edits;
+          t_edits.reserve(edits.size());
+          for (auto &edit : edits) {
+            TextEdit t_edit;
+            t_edit.text = edit.value("newText", "");
+            t_edit.start.row = edit["range"]["start"]["line"];
+            t_edit.start.col = edit["range"]["start"]["character"];
+            t_edit.end.row = edit["range"]["end"]["line"];
+            t_edit.end.col = edit["range"]["end"]["character"];
+            utf8_normalize_edit(editor, &t_edit);
+            t_edits.push_back(t_edit);
+          }
+          apply_lsp_edits(editor, t_edits, false);
+          ensure_scroll(editor);
+        }
+      };
+      lsp_send(editor->lsp, message, pending);
+      break;
+    }
+  }
 }

src/editor/lsp.cc

@@ -1,5 +1,6 @@
 #include "editor/decl.h"
 #include "editor/editor.h"
+#include "utils/utils.h"
 
 void apply_lsp_edits(Editor *editor, std::vector<TextEdit> edits, bool move) {
   if (!edits.size())
@@ -43,13 +44,28 @@ void editor_lsp_handle(Editor *editor, json msg) {
     for (size_t i = 0; i < diagnostics.size(); i++) {
       json d = diagnostics[i];
       VWarn w;
-      // HACK: convert back to utf-8 but as this is only visually affecting it
-      //       is not worth getting the line string from the rope.
       w.line = d["range"]["start"]["line"];
       w.start = d["range"]["start"]["character"];
+      std::shared_lock lock(editor->knot_mtx);
+      LineIterator *it = begin_l_iter(editor->root, w.line);
+      if (!it)
+        continue;
+      uint32_t len;
+      char *line = next_line(it, &len);
+      if (!line) {
+        free(it->buffer);
+        free(it);
+        continue;
+      }
+      if (len > 0 && line[len - 1] == '\n')
+        --len;
+      lock.unlock();
+      w.start = utf16_offset_to_utf8(line, w.start);
       uint32_t end = d["range"]["end"]["character"];
       if (d["range"]["end"]["line"] == w.line)
-        w.end = end;
+        w.end = utf16_offset_to_utf8(line, end);
+      free(it->buffer);
+      free(it);
       std::string text = trim(d["message"].get<std::string>());
       w.text_full = text;
       auto pos = text.find('\n');

src/io/renderer.cc

@@ -1,4 +1,5 @@
 #include "io/sysio.h"
+#include <cstdint>
 
 uint32_t rows, cols;
 bool show_cursor = 0;
@@ -143,7 +144,7 @@ void render() {
       if (content_changed) {
         if (first_change_col == -1) {
           first_change_col = col;
-          if (first_change_col > 0) {
+          if (first_change_col > 0)
             for (int64_t back = 1; back <= 4 && first_change_col - back >= 0;
                  ++back) {
               ScreenCell &prev_cell =
@@ -154,8 +155,7 @@ void render() {
               }
             }
         }
-        }
+        last_change_col = MIN(cols + 1, col + 4);
-        last_change_col = col;
       }
     }
     if (first_change_col == -1)
@@ -253,10 +253,18 @@ void render() {
         out.push_back(' ');
       } else {
         if (!new_cell.utf8.empty()) {
-          if (new_cell.utf8[0] == '\t')
+          if (new_cell.utf8[0] == '\t') {
             out.append("    ");
-          else if (new_cell.utf8[0] != '\x1b')
+          } else if (new_cell.utf8[0] != '\x1b') {
             out.append(new_cell.utf8);
+          } else if (new_cell.utf8[0] == '\x1b') {
+            ScreenCell *cell = &new_cell;
+            int back = 0;
+            while ((int)col - back >= 0 && cell->utf8[0] == '\x1b')
+              cell = &screen[row * cols + col - ++back];
+            if (width >= cell->width)
+              out.append(" ");
+          }
         } else {
           out.push_back(' ');
         }

src/lsp/process.cc

@@ -59,7 +59,7 @@ std::shared_ptr<LSPInstance> get_or_init_lsp(uint8_t lsp_id) {
     LSPPending *pending = new LSPPending();
     pending->method = "initialize";
     pending->editor = nullptr;
-    pending->callback = [lsp](Editor *, std::string, json msg) {
+    pending->callback = [lsp, lsp_id](Editor *, std::string, json msg) {
       if (msg.contains("result") && msg["result"].contains("capabilities")) {
         auto &caps = msg["result"]["capabilities"];
         if (caps.contains("textDocumentSync")) {
@@ -73,7 +73,8 @@ std::shared_ptr<LSPInstance> get_or_init_lsp(uint8_t lsp_id) {
           }
         }
         lsp->allow_formatting = caps.value("documentFormattingProvider", false);
-        if (caps.contains("documentOnTypeFormattingProvider")) {
+        if (lsp_id != LUA_LS /* Lua ls gives terrible ontype formatting */ &&
+            caps.contains("documentOnTypeFormattingProvider")) {
           auto &fmt = caps["documentOnTypeFormattingProvider"];
           if (fmt.is_object()) {
             if (fmt.contains("firstTriggerCharacter")) {
@@ -93,7 +94,6 @@ std::shared_ptr<LSPInstance> get_or_init_lsp(uint8_t lsp_id) {
             lsp->allow_formatting_on_type = fmt.get<bool>();
           }
         }
-
         if (caps.contains("hoverProvider")) {
           auto &hover = caps["hoverProvider"];
           lsp->allow_hover =

src/ui/hover.cc

@@ -7,8 +7,6 @@ void HoverBox::clear() {
   is_markup = false;
   size = {0, 0};
   cells.clear();
-  highlights.clear();
-  hover_spans.clear();
 }
 
 void HoverBox::scroll(int32_t number) {
@@ -30,9 +28,11 @@ void HoverBox::render_first(bool scroll) {
     std::vector<Span> injected_spans;
     TSSetBase ts = TSSetBase{};
     if (is_markup) {
+      highlights.clear();
       highlights.reserve(1024);
       base_spans.reserve(1024);
       injected_spans.reserve(1024);
+      hover_spans.clear();
       hover_spans.reserve(1024);
       std::string query_path = get_exe_dir() + "/../grammar/hover.scm";
       ts.language = LANG(markdown)();
@@ -85,6 +85,8 @@ void HoverBox::render_first(bool scroll) {
                 uint32_t start = ts_node_start_byte(inj_cap.node);
                 uint32_t end = ts_node_end_byte(inj_cap.node);
                 if (Highlight *hl = safe_get(inj_ts.query_map, inj_cap.index)) {
+                  if (highlights.size() >= 1000)
+                    continue;
                   highlights.push_back(*hl);
                   Highlight *hl_f = &highlights.back();
                   injected_spans.push_back({start, end, hl_f});
@@ -98,6 +100,8 @@ void HoverBox::render_first(bool scroll) {
             continue;
           }
           if (Highlight *hl = safe_get(ts.query_map, cap.index)) {
+            if (highlights.size() >= 1000)
+              continue;
             highlights.push_back(*hl);
             Highlight *hl_f = &highlights.back();
             base_spans.push_back({start, end, hl_f});

src/utils/text.cc

@@ -42,6 +42,39 @@ std::string percent_encode(const std::string &s) {
   return out;
 }
 
+std::string substitute_fence(const std::string &documentation,
+                             const std::string &lang) {
+  int errorcode;
+  PCRE2_SIZE erroroffset;
+  const char *pattern = "\n```\n(.*?)```\n";
+  pcre2_code *re =
+      pcre2_compile((PCRE2_SPTR)pattern, PCRE2_ZERO_TERMINATED, PCRE2_DOTALL,
+                    &errorcode, &erroroffset, nullptr);
+  if (!re)
+    return documentation;
+  PCRE2_SIZE outlen = 0;
+  std::string replacement = "```" + lang + "\n$1```";
+  int rc = pcre2_substitute(
+      re, (PCRE2_SPTR)documentation.c_str(), documentation.size(), 0,
+      PCRE2_SUBSTITUTE_GLOBAL | PCRE2_SUBSTITUTE_OVERFLOW_LENGTH, nullptr,
+      nullptr, (PCRE2_SPTR)replacement.c_str(), replacement.size(), nullptr,
+      &outlen);
+  if (rc < 0) {
+    pcre2_code_free(re);
+    return documentation;
+  }
+  std::string out(outlen, '\0');
+  rc = pcre2_substitute(re, (PCRE2_SPTR)documentation.c_str(),
+                        documentation.size(), 0, PCRE2_SUBSTITUTE_GLOBAL,
+                        nullptr, nullptr, (PCRE2_SPTR)replacement.c_str(),
+                        replacement.size(), (PCRE2_UCHAR *)out.data(), &outlen);
+  pcre2_code_free(re);
+  if (rc < 0)
+    return documentation;
+  out.resize(outlen);
+  return out;
+}
+
 std::string trim(const std::string &s) {
   size_t start = s.find_first_not_of(" \t\n\r");
   if (start == std::string::npos)

src/utils/unicode.cc

@@ -86,9 +86,9 @@ uint32_t count_clusters(const char *line, size_t len, size_t from, size_t to) {
   return count;
 }
 
-int utf8_byte_offset_to_utf16(const char *s, size_t byte_pos) {
+uint32_t utf8_byte_offset_to_utf16(const char *s, uint32_t byte_pos) {
-  int utf16_units = 0;
+  uint32_t utf16_units = 0;
-  size_t i = 0;
+  uint32_t i = 0;
   while (i < byte_pos) {
     unsigned char c = s[i];
     if ((c & 0x80) == 0x00) {
@@ -108,9 +108,9 @@ int utf8_byte_offset_to_utf16(const char *s, size_t byte_pos) {
   return utf16_units;
 }
 
-size_t utf16_offset_to_utf8(const char *s, int utf16_pos) {
+uint32_t utf16_offset_to_utf8(const char *s, uint32_t utf16_pos) {
-  int utf16_units = 0;
+  uint32_t utf16_units = 0;
-  size_t i = 0;
+  uint32_t i = 0;
   while (utf16_units < utf16_pos) {
     unsigned char c = s[i];
     if ((c & 0x80) == 0x00) {