Cleanup

README.md

@@ -21,7 +21,7 @@ curl https://syedm.dev/crib | sh
 ```
 
 Currently only for Linux.<br>
-*Tested with arch linux and ubuntu and void*<br>
+*Tested with arch linux, ubuntu and void*<br>
 
 ## Building
 

TODO.md

@@ -2,35 +2,45 @@ Copyright 2025 Syed Daanish
 
 # TODO
 
-## Next few super long boring things to do::
-* redo lsp threads such that no mutex needed for any rope stuff (done)
-    - Also make the classes own the methods in lsp (done)
-    - This will mean that parsers/renderers and keystrokes will not need to be individually locked (done)
-    - And so it will be much faster (done)
-    - At which point the main thread can also be blocked on user input or lsp responses and still be faster
-* Add a superclass for editor called Window (which can be popup or tiled) (done)
-* Add a recursive tiling class for windows (done)
-* Handled by a single renderer that calls and renders each window (done)
-* Make editor's functions into its own methods (classify it) (done)
-    - While at it
-        - Seperate system functions into a class that branches to support local / ssh / server modes.
-        - Even lsp shouldnt be directly controlled because it can branch on local and server modes
-            - check wolfSSH stuff for remote editing
-            - Thick and thin mode
-                - thin mode only shares screen diffing over the network - uses server settings / themes
-                - thick only shared fileio and lsp data with local used for all else - uses local settings / themes
-    - Redo hooks as a engine of its own.
-    - And factorize renderer into its own class (and make it just return an array of the render without knowing teh x,y)
-        - which is just managed by the renderer
-    - which is reused by scrollers/ensurers too
-    - this will then allow inlay hints to be possible
-    - and also make VAI easier to implement
+- And factorize renderer into its own class
+    - which is just managed by the renderer
+    - and adjusment.cc will call it
+- which is reused by scrollers/ensurers too (for knowing screen wrapped end)
+- this will then allow inlay hints to be possible
+- and also make VAI easier to implement
 * Allow keybinds to be set in ruby
 
-* then the fun part:
+* ruby file for project specific runs alongside any system wide ones
+
+- Seperate system functions into a class that branches to support local / ssh / server modes.
+- Even lsp shouldnt be directly controlled because it can branch on local and server modes
+    - check libssh2 or child process stuff for remote editing (remote mode)
+    - Thick and thin mode (server mode) (through ssh or port) (for website only port + webhook)
+        - thin mode only shares screen diffing over the network - uses server settings / themes / shell
+            - only one user connects at a time
+            - provides data as a websocket with tui or gui/online modes where tui shares diffing as is (can be connected to by gui or website)
+            - and gui/online modes try to (if not then render onto a canvas) forward window.render_gui() and dedicated windowing system (i.e windowing is html divs)
+            - and window.render_gui() returns a markup code (custom markup)
+                - and for events i could either leave tui as-is and for markup have special event handling or i
+                - implement the widget system in both and have the same event handling then
+            - actually for true cross platform i need to make a ui lib with widgets etc. so it is workable on all platforms (so they all need only a single ui lib)
+        - thick only shared fileio and lsp data with local used for all else - uses local settings / themes
+            - multiple users connect at once but maybe either make them not be allowed to open same file or make vscode like multiediting (unsure how exactly)
+            - they all share same lsp instances
+            - they all have the same shell access
+        - allow having an instance that forwards thick mode to thin mode (to connect online mode to a thick server for code collaboration)
 * Then allow ruby code to create and handle windows as extentions
 * Then 3 inbuilt extentions being file manager, theme picker, tab selector
 
+* make another tile type called tile tabbed that has a tab switcher and shows only one of its children at once
+* it takes focus and takes some keybinds for switching and forwards the rest (it also forwards the info and adds one for itself - the selected/total tabs)
+* also implement info strings for all the stuff that take focus
+
+* Mode is a feild on a editor and there is no global "mode system" except in editors (enxtentions do their own thing)
+    * make mode normal and not atomic if it is truly unused across threads
+    * this means keybinds can be set to use the editor as insert mode only and so behave like vscode
+
+* split lsp stuff into multiple files (not a monolithic class as it is now)
 * Extentions can also be used as file openers (for like databases . diffing . images . audio etc)
 * Local cache for state management (like undo history, cursor positions etc) (location can be set in config)
 * make sure to write inbuilt extentions in cpp and not ruby
@@ -110,8 +120,6 @@ Copyright 2025 Syed Daanish
 
 * [ ] **Auto brace selection:** Add support for auto brace selection.
 
-* [ ] **Tree-sitter Indent:** Attempt to allow Tree-sitter to handle indentation if possible.
-
 ### UX
 
 * [ ] **Completion Filtering:**
@@ -120,7 +128,7 @@ Copyright 2025 Syed Daanish
 
 * [ ] **Basic Autocomplete:** Keep a list of words in the current buffer for non-LSP fallback.
 
-### Major Features
+### Features
 
 * [ ] **Search & Replace:**
     * [ ] Add Search/Replace UI.
@@ -154,3 +162,14 @@ Copyright 2025 Syed Daanish
     * [ ] Switch JSON parser to `RapidJSON` (or similar high-performance lib).
     * [ ] Decrease usage of `std::string` in UI, LSP, warnings etc.
     * [ ] Also for vectors into managed memory especially for completions/lsp-stuff.
+
+
+
+
+<!-- The pits of hell: -->
+<!-- * Replace perfectly working superfast rope with a structure that stores for small files everything -->
+<!--     - but for large ones it stores only what the user viewed / edited (by line number) -->
+<!--     - the rest is from the disk and not entirely loaded into memory (keep as option maybe they want everything in memory?) -->
+<!--     - also for the line mode still go through entire file to get and store line offsets -->
+<!--     - also it stores everything prolly still as rope tho (rope of lines) (or sparse map) -->
+<!--     - it stores the line text as-is which is only unicode / crlf normalized during usage -->

include/editor/editor.h

@@ -1,13 +1,15 @@
 #ifndef EDITOR_H
 #define EDITOR_H
 
+#include "editor/hooks.h"
 #include "editor/indents.h"
+#include "editor/visual.h"
+#include "extentions/diagnostics.h"
 #include "extentions/hover.h"
 #include "io/knot.h"
 #include "io/sysio.h"
 #include "syntax/extras.h"
 #include "syntax/parser.h"
-#include "ui/diagnostics.h"
 #include "utils/utils.h"
 #include "windows/decl.h"
 
@@ -31,7 +33,6 @@ struct Editor : Window {
   Coord size = {0, 0};
   Coord scroll = {0, 0};
   Language lang = {};
-  uint32_t hooks[94] = {0};
   bool jumper_set = false;
   std::vector<VWarn> warnings = {};
   bool warnings_dirty = false;
@@ -39,9 +40,10 @@ struct Editor : Window {
   std::shared_mutex lsp_mtx;
   std::atomic<struct LSPInstance *> lsp = nullptr;
   HoverBox *hover_popup = init_hover();
-  bool diagnostics_active = false;
+  DiagnosticBox *diagnostic_popup = init_diagnostic();
   std::atomic<int> lsp_version = 1;
   IndentationEngine indents = {};
+  HookEngine hooks = {};
   Parser *parser = nullptr;
   ExtraHighlighter extra_hl = {};
   bool is_css_color = false;
@@ -84,7 +86,6 @@ struct Editor : Window {
   void backspace_edit();
   void delete_prev_word();
   void delete_next_word();
-  void clear_hooks_at_line(uint32_t line);
   void cursor_prev_word();
   void cursor_next_word();
   void select_all();
@@ -158,19 +159,6 @@ struct Editor : Window {
     free(it->buffer);
     free(it);
   }
-
-  inline void apply_hook_insertion(uint32_t line, uint32_t rows) {
-    for (auto &hook : this->hooks)
-      if (hook > line)
-        hook += rows;
-  }
-
-  inline void apply_hook_deletion(uint32_t removal_start,
-                                  uint32_t removal_end) {
-    for (auto &hook : this->hooks)
-      if (hook > removal_start)
-        hook -= removal_end - removal_start + 1;
-  }
 };
 
 #endif

include/editor/hooks.h

@@ -0,0 +1,91 @@
+#ifndef EDITOR_HOOKS_H
+#define EDITOR_HOOKS_H
+
+#include "utils/utils.h"
+
+struct HookEngine {
+  uint32_t hooks[94];
+  std::vector<std::pair<uint32_t, char>> v;
+  std::vector<std::pair<uint32_t, char>>::iterator it;
+
+  HookEngine() {
+    for (int i = 0; i < 94; i++)
+      hooks[i] = UINT32_MAX;
+    v.reserve(94);
+  }
+
+  inline void edit(uint32_t line, uint32_t removed_rows,
+                   uint32_t inserted_rows) {
+
+    int64_t delta = (int64_t)inserted_rows - (int64_t)removed_rows;
+    for (int i = 0; i < 94; i++) {
+      uint32_t &h = hooks[i];
+      if (h == UINT32_MAX)
+        continue;
+      if (h < line)
+        continue;
+      if (removed_rows > 0 && h < line + removed_rows) {
+        h = UINT32_MAX;
+        continue;
+      }
+      h = (uint32_t)((int64_t)h + delta);
+    }
+  }
+
+  inline void set(char c, uint32_t line) {
+    if (c < '!' || c > '~')
+      return;
+    int idx = c - '!';
+    for (int i = 0; i < 94; i++) {
+      if (hooks[i] == line) {
+        hooks[i] = UINT32_MAX;
+        break;
+      }
+    }
+    hooks[idx] = line;
+  }
+
+  inline bool get(char c, uint32_t *out_line) const {
+    if (c < '!' || c > '~')
+      return false;
+    uint32_t h = hooks[c - '!'];
+    if (h == UINT32_MAX)
+      return false;
+    *out_line = h;
+    return true;
+  }
+
+  inline void clear_line(uint32_t line) {
+    for (int i = 0; i < 94; i++)
+      if (hooks[i] == line)
+        hooks[i] = UINT32_MAX;
+  }
+
+  inline void clear(char c) {
+    if (c < '!' || c > '~')
+      return;
+    hooks[c - '!'] = UINT32_MAX;
+  }
+
+  void start_iter(uint32_t scroll_line) {
+    for (int i = 0; i < 94; i++)
+      if (hooks[i] != UINT32_MAX)
+        v.push_back({hooks[i], (char)('!' + i)});
+    std::sort(v.begin(), v.end(),
+              [](const auto &a, const auto &b) { return a.first < b.first; });
+    it = std::lower_bound(
+        v.begin(), v.end(), scroll_line,
+        [](const auto &p, uint32_t line) { return p.first < line; });
+  }
+
+  char next(uint32_t line_index) {
+    if (it != v.end() && it->first == line_index) {
+      char c = it->second;
+      it++;
+      return c;
+    }
+    return '\0';
+  }
+};
+
+#endif

include/editor/visual.h

@@ -0,0 +1,113 @@
+#include "editor/decl.h"
+#include "io/knot.h"
+#include "io/sysio.h"
+#include "utils/utils.h"
+
+struct GraphemeIterator {
+  char *line{nullptr};
+  uint32_t len{0};
+
+  GraphemeIterator(char *line, uint32_t len) : line(line), len(len) {}
+
+  char *next(uint32_t *o_len, uint32_t *o_width) {
+    if (!line || len == 0)
+      return nullptr;
+    char *cur = line;
+    uint32_t g = grapheme_next_character_break_utf8(cur, len);
+    if (o_width)
+      *o_width = display_width(cur, g);
+    if (o_len)
+      *o_len = g;
+    line += g;
+    len -= g;
+    return cur;
+  }
+};
+
+struct VisualIterator {
+  LineIterator *it{nullptr};
+  uint32_t line_index{UINT32_MAX};
+  uint32_t offset{0};
+  uint32_t len{0};
+  bool first{true};
+  char *line{nullptr};
+  uint32_t render_width{io::cols};
+  std::stack<std::pair<Coord, Coord>> prev_stack;
+
+  VisualIterator(Knot *root, Coord pos, uint32_t width)
+      : it(begin_l_iter(root, pos.row)), line_index(pos.row - 1),
+        offset(pos.col), render_width(width) {}
+
+  std::pair<Coord, Coord> prev() {
+    if (!it)
+      return {{UINT32_MAX, UINT32_MAX}, {UINT32_MAX, UINT32_MAX}};
+    if (prev_stack.empty())
+      return _prev();
+    auto ret = prev_stack.top();
+    offset = ret.first.col;
+    prev_stack.pop();
+    return ret;
+  }
+
+  std::pair<Coord, Coord> _prev() {
+    if (!it)
+      return {{UINT32_MAX, UINT32_MAX}, {UINT32_MAX, UINT32_MAX}};
+    line_index--;
+    line = prev_line(it, &len);
+    if (!line)
+      return {{UINT32_MAX, UINT32_MAX}, {UINT32_MAX, UINT32_MAX}};
+    if (len > 0 && line[len - 1] == '\n')
+      len--;
+    offset = first ? offset : 0;
+    first = false;
+    GraphemeIterator g(line + offset, len - offset);
+    uint32_t o_len, o_width, rendered = 0, advance = 0;
+    while (g.next(&o_len, &o_width)) {
+      if (rendered + o_width > render_width) {
+        prev_stack.push({{line_index, offset}, {line_index, offset + advance}});
+        offset += advance;
+        rendered = 0;
+        advance = 0;
+      }
+      advance += o_len;
+      rendered += o_width;
+    }
+    return {{line_index, offset}, {line_index, offset + advance}};
+  }
+
+  std::pair<Coord, Coord> next() {
+    if (!it)
+      return {{UINT32_MAX, UINT32_MAX}, {UINT32_MAX, UINT32_MAX}};
+    if (!line) {
+      line_index++;
+      line = next_line(it, &len);
+      if (!line)
+        return {{UINT32_MAX, UINT32_MAX}, {UINT32_MAX, UINT32_MAX}};
+      if (len > 0 && line[len - 1] == '\n')
+        len--;
+      offset = first ? offset : 0;
+      first = false;
+    }
+    GraphemeIterator g(line + offset, len - offset);
+    uint32_t o_len, o_width, rendered = 0, advance = 0;
+    while (g.next(&o_len, &o_width)) {
+      if (rendered + o_width > render_width) {
+        offset += advance;
+        return {{line_index, offset - advance}, {line_index, offset}};
+      }
+      advance += o_len;
+      rendered += o_width;
+    }
+    offset += advance;
+    if (offset >= len)
+      line = nullptr;
+    return {{line_index, offset}, {line_index, offset + advance}};
+  }
+
+  ~VisualIterator() {
+    if (!it)
+      return;
+    free(it->buffer);
+    free(it);
+  }
+};

include/extentions/diagnostics.h

@@ -0,0 +1,22 @@
+#ifndef EXTENTION_DIAGNOSTICS_H
+#define EXTENTION_DIAGNOSTICS_H
+
+#include "editor/decl.h"
+#include "io/sysio.h"
+#include "pch.h"
+#include "utils/utils.h"
+#include "windows/decl.h"
+
+struct DiagnosticBox : Popup {
+  std::vector<VWarn> warnings;
+
+  DiagnosticBox() { this->hidden = true; }
+  void clear() { this->warnings.clear(); }
+  void render(std::vector<ScreenCell> &buffer, Coord size, Coord pos) override;
+  void handle_click(KeyEvent, Coord) override { this->hidden = true; };
+  ~DiagnosticBox() {};
+};
+
+DiagnosticBox *init_diagnostic();
+
+#endif

include/lsp/lsp.h

@@ -2,19 +2,15 @@
 #define LSP_H
 
 #include "editor/editor.h"
+#include "main.h"
 #include "pch.h"
 #include "utils/utils.h"
+#include <cstdint>
+#include <sys/types.h>
 
-#define LSP_TIMEOUT 3000
-
-namespace lsp {
-extern std::mutex lsp_mutex;
-extern std::unordered_map<std::string, std::unique_ptr<LSPInstance>>
-    active_lsps;
-extern Queue<std::string> need_opening;
-extern std::unordered_set<std::string> opened;
-extern std::vector<Editor *> new_editors;
-} // namespace lsp
+#define LSP_TIMEOUT_START 3000
+#define LSP_TIMEOUT_QUIT_NORMAL 300
+#define LSP_TIMEOUT_QUIT_FORCE 80
 
 void lsp_worker();
 
@@ -48,6 +44,16 @@ struct LSPMessage {
   std::function<void(const LSPMessage &)> callback;
 };
 
+namespace lsp {
+extern std::mutex lsp_mutex;
+extern std::unordered_map<std::string, std::unique_ptr<LSPInstance>>
+    active_lsps;
+extern Queue<std::string> need_opening;
+extern std::unordered_set<std::string> opened;
+extern std::vector<Editor *> new_editors;
+extern Queue<std::unique_ptr<LSPMessage>> response_queue;
+} // namespace lsp
+
 struct LSPInstance {
   const LSP *lsp_info;
   std::string root_dir;
@@ -70,7 +76,6 @@ struct LSPInstance {
   Queue<json> inbox;
   Queue<json> outbox;
   std::unordered_map<uint32_t, std::unique_ptr<LSPMessage>> pending;
-  std::vector<std::unique_ptr<LSPMessage>> lsp_response_queue;
   std::vector<Editor *> editors;
 
   LSPInstance(std::string lsp_id) {
@@ -89,7 +94,15 @@ struct LSPInstance {
           {"capabilities", client_capabilities}}}};
     send_raw(initialize_message);
     pollfd pfd{stdout_fd, POLLIN, 0};
-    poll(&pfd, 1, LSP_TIMEOUT);
+    uint32_t waited = 0;
+    while (waited < LSP_TIMEOUT_START) {
+      poll(&pfd, 1, 50);
+      if (pfd.revents & POLLIN)
+        break;
+      if (!running)
+        return;
+      waited += 50;
+    }
     if (!(pfd.revents & POLLIN)) {
       exited = true;
       return;
@@ -179,34 +192,39 @@ struct LSPInstance {
     send_raw(initialized_message);
   }
   ~LSPInstance() {
+    uint32_t timeout =
+        running ? LSP_TIMEOUT_QUIT_NORMAL : LSP_TIMEOUT_QUIT_FORCE;
     for (auto &ed : editors)
       ed->lsp.store(nullptr);
     initialized = false;
     exited = true;
     if (pid == -1)
       return;
-    json shutdown = {{"id", ++last_id}, {"method", "shutdown"}};
+    json shutdown = {
+        {"jsonrpc", "2.0"}, {"id", ++last_id}, {"method", "shutdown"}};
     send_raw(shutdown);
     pollfd pfd{stdout_fd, POLLIN, 0};
-    poll(&pfd, 1, 500);
-    json exit_msg = {{"method", "exit"}};
-    send_raw(exit_msg);
-    int waited = 0;
-    while (waited < 100) {
-      int status;
-      pid_t res = waitpid(pid, &status, WNOHANG);
-      if (res == pid)
-        break;
-      std::this_thread::sleep_for(std::chrono::milliseconds(10));
-      waited += 10;
+    poll(&pfd, 1, timeout);
+    if (pfd.revents & POLLIN) {
+      json exit_msg = {{"jsonrpc", "2.0"}, {"method", "exit"}};
+      send_raw(exit_msg);
+      uint32_t waited = 0;
+      while (waited < timeout) {
+        int status;
+        pid_t res = waitpid(pid, &status, WNOHANG);
+        if (res == pid)
+          break;
+        std::this_thread::sleep_for(10ms);
+        waited += 10;
+      }
     }
+    close(stdin_fd);
+    close(stdout_fd);
     if (kill(pid, 0) == 0) {
       kill(pid, SIGKILL);
       waitpid(pid, nullptr, 0);
     }
     pid = -1;
-    close(stdin_fd);
-    close(stdout_fd);
   }
   bool init_process() {
     int in_pipe[2];
@@ -312,7 +330,7 @@ struct LSPInstance {
         if (it != pending.end()) {
           if (it->second->editor) {
             it->second->message = *msg;
-            lsp_response_queue.push_back(std::move(it->second));
+            lsp::response_queue.push(std::move(it->second));
           } else {
             auto message = *std::move(it->second);
             message.message = *msg;
@@ -335,7 +353,7 @@ struct LSPInstance {
         response->message = *msg;
         response->callback = editor_handle_wrapper;
         if (ed)
-          lsp_response_queue.push_back(std::move(response));
+          lsp::response_queue.push(std::move(response));
         else
           lsp_handle(*msg);
       }
@@ -344,11 +362,6 @@ struct LSPInstance {
   inline static void editor_handle_wrapper(const LSPMessage &message) {
     message.editor->lsp_handle(message.message);
   }
-  void callbacks() {
-    for (auto &message : lsp_response_queue)
-      message->callback(*message);
-    lsp_response_queue.clear();
-  }
   inline void send_raw(const json &msg) {
     std::string payload = msg.dump();
     std::string header =

include/pch.h

@@ -31,6 +31,7 @@ extern "C" {
 #include <filesystem>
 #include <fstream>
 #include <functional>
+#include <immintrin.h>
 #include <limits.h>
 #include <map>
 #include <mutex>

include/syntax/decl.h

@@ -6,7 +6,7 @@
 #include "pch.h"
 #include "syntax/trie.h"
 
-#define MAX_LINES_LOOKBEHIND 80
+#define MAX_LINES_LOOKAROUND 512
 
 struct Highlight {
   uint32_t fg{0xFFFFFF};

include/syntax/line_map.h

@@ -59,15 +59,8 @@ struct LineMap {
   }
 
   void apply_edit(uint32_t start, int64_t delta) {
-    if (delta < 0) {
-      int64_t count = -delta;
-      for (int64_t i = 0; i < count; i++) {
-        auto key = resolve_line(start + i);
-        if (!key)
-          continue;
-        lines.erase(*key);
-      }
-    }
+    if (delta < 0)
+      batch_remove(start, -delta);
     current_version++;
     edit_log.push_back({start, delta});
   }
@@ -79,6 +72,45 @@ private:
 
   uint32_t current_version;
 
+  void batch_remove(uint32_t line, uint32_t count) {
+    std::vector<uint32_t> lines_t;
+    lines_t.reserve(count);
+    for (uint32_t i = 0; i < count; i++)
+      lines_t.push_back(line + i);
+    for (int64_t v = current_version; v >= 0; v--) {
+      for (auto &l : lines_t) {
+        if (l == UINT32_MAX)
+          continue;
+        LineKey key = {l, (uint32_t)v};
+        if (lines.find(key) != lines.end()) {
+          lines.erase(key);
+          l = UINT32_MAX;
+        }
+      }
+      bool all_removed = true;
+      const auto &edit = edit_log[v];
+      for (auto &l : lines_t) {
+        if (l == UINT32_MAX)
+          continue;
+        all_removed = false;
+        if (edit.delta > 0) {
+          if (l >= edit.start_line) {
+            if (l < edit.start_line + edit.delta)
+              l = UINT32_MAX;
+            else
+              l -= edit.delta;
+          }
+        } else {
+          if (l >= edit.start_line)
+            l -= edit.delta;
+        }
+      }
+      if (all_removed)
+        break;
+    }
+    return;
+  }
+
   std::optional<LineKey> resolve_line(uint32_t line) {
     uint32_t current_line = line;
     for (int64_t v = current_version; v >= 0; v--) {

include/ui/diagnostics.h

@@ -1,19 +0,0 @@
-#ifndef UI_DIAGNOSTICS_H
-#define UI_DIAGNOSTICS_H
-
-#include "editor/decl.h"
-#include "io/sysio.h"
-#include "pch.h"
-#include "utils/utils.h"
-
-struct DiagnosticBox {
-  std::vector<VWarn> warnings;
-  std::vector<ScreenCell> cells;
-  Coord size;
-
-  void clear();
-  void render_first();
-  void render(Coord pos);
-};
-
-#endif

include/utils/utils.h

@@ -6,18 +6,19 @@
 template <typename T> struct Queue {
   void push(T val) {
     std::lock_guard<std::mutex> lock(m);
-    q.push(val);
+    q.push(std::move(val));
   }
   std::optional<T> front() {
+    std::lock_guard<std::mutex> lock(m);
     if (q.empty())
       return std::nullopt;
-    return q.front();
+    return std::move(q.front());
   }
   bool pop(T &val) {
     std::lock_guard<std::mutex> lock(m);
     if (q.empty())
       return false;
-    val = q.front();
+    val = std::move(q.front());
     q.pop();
     return true;
   }
@@ -137,6 +138,7 @@ std::string percent_encode(const std::string &s);
 std::string percent_decode(const std::string &s);
 uint32_t count_clusters(const char *line, size_t len, size_t from, size_t to);
 std::string trim(const std::string &s);
+std::string strip_newlines(const std::string &s);
 std::string substitute_fence(const std::string &documentation,
                              const std::string &lang);
 

include/windows/decl.h

@@ -113,16 +113,6 @@ extern std::vector<std::unique_ptr<Popup>> popups;
 extern std::vector<std::unique_ptr<TileBase>> floating_tiles;
 } // namespace layout
 
-inline void close(Popup *handle) {
-  std::erase_if(layout::popups,
-                [handle](const auto &p) { return p.get() == handle; });
-}
-
-inline void close(TileBase *handle) {
-  std::erase_if(layout::floating_tiles,
-                [handle](const auto &p) { return p.get() == handle; });
-}
-
 void render();
 void handle_click(KeyEvent event);
 

samples/ruby.rb

@@ -339,6 +339,5 @@ puts 'Ruby syntax highlighting test complete.'
 
 __END__
 
-
 Anything here should be ignored >><<
 {{{}}}[[[]]](((000)))

src/editor/adjustment.cc

@@ -2,72 +2,27 @@
 
 void Editor::ensure_cursor() {
   if (this->cursor < this->scroll) {
-    this->cursor.row = this->scroll.row;
-    this->cursor.col = this->scroll.col;
+    this->cursor = this->scroll;
     this->cursor_preffered = UINT32_MAX;
     return;
   }
   uint32_t numlen =
       EXTRA_META + static_cast<int>(std::log10(this->root->line_count + 1));
   uint32_t render_width = this->size.col - numlen;
-  uint32_t visual_rows = 0;
-  uint32_t line_index = this->scroll.row;
-  bool first_visual_line = true;
-  LineIterator *it = begin_l_iter(this->root, line_index);
-  if (!it)
-    return;
-  Coord last_visible = this->scroll;
-  while (true) {
-    if (visual_rows >= this->size.row)
-      break;
-    uint32_t line_len;
-    char *line = next_line(it, &line_len);
-    if (!line)
-      break;
-    if (line_len > 0 && line[line_len - 1] == '\n')
-      line_len--;
-    uint32_t offset = first_visual_line ? this->scroll.col : 0;
-    first_visual_line = false;
-    while (offset < line_len || (line_len == 0 && offset == 0)) {
-      Coord current = {line_index, offset};
-      last_visible = current;
-      visual_rows++;
-      if (visual_rows >= this->size.row)
-        break;
-      uint32_t col = 0;
-      uint32_t advance = 0;
-      uint32_t left = line_len - offset;
-      while (left > 0 && col < render_width) {
-        uint32_t g =
-            grapheme_next_character_break_utf8(line + offset + advance, left);
-        int w = display_width(line + offset + advance, g);
-        if (col + w > render_width)
-          break;
-        advance += g;
-        left -= g;
-        col += w;
-      }
-      if (line_index == this->cursor.row) {
-        if (this->cursor.col >= offset &&
-            this->cursor.col <= offset + advance) {
-          free(it->buffer);
-          free(it);
-          return;
-        }
-      }
-      if (advance == 0)
-        break;
-      offset += advance;
-      if (line_len == 0)
-        break;
-    }
-    line_index++;
+  uint32_t render_height = 0;
+  VisualIterator vis(root, scroll, render_width);
+  std::pair<Coord, Coord> vline;
+  Coord last_visible = scroll;
+  while ((vline = vis.next()).first.row != UINT32_MAX &&
+         render_height < this->size.row) {
+    render_height++;
+    last_visible = vline.first;
+    if (cursor >= vline.first && cursor < vline.second)
+      return;
   }
-  this->cursor.row = last_visible.row;
-  this->cursor.col = last_visible.col;
+  cursor = last_visible;
+  cursor_preffered = UINT32_MAX;
   this->cursor_preffered = UINT32_MAX;
-  free(it->buffer);
-  free(it);
 }
 
 void Editor::ensure_scroll() {

src/editor/edit.cc

@@ -44,10 +44,10 @@ void Editor::edit_erase(Coord pos, int64_t len) {
     }
     uint32_t start_row = point.row;
     uint32_t end_row = pos.row;
-    this->apply_hook_deletion(start_row + 1, end_row);
     this->root = erase(this->root, start, byte_pos - start);
     if (this->parser)
       this->parser->edit(start_row, end_row - start_row, 0);
+    this->hooks.edit(start_row, end_row - start_row, 0);
     if (do_lsp) {
       auto lsp = this->lsp.load();
       if (lsp->incremental_sync) {
@@ -113,10 +113,10 @@ void Editor::edit_erase(Coord pos, int64_t len) {
     }
     uint32_t start_row = pos.row;
     uint32_t end_row = point.row;
-    this->apply_hook_deletion(start_row + 1, end_row);
     this->root = erase(this->root, byte_pos, end - byte_pos);
     if (this->parser)
       this->parser->edit(start_row, end_row - start_row, 0);
+    this->hooks.edit(start_row, end_row - start_row, 0);
     if (do_lsp) {
       auto lsp = this->lsp.load();
       if (lsp->incremental_sync) {
@@ -169,9 +169,9 @@ void Editor::edit_insert(Coord pos, char *data, uint32_t len) {
   for (uint32_t i = 0; i < len; i++)
     if (data[i] == '\n')
       rows++;
-  this->apply_hook_insertion(pos.row, rows);
   if (this->parser)
     this->parser->edit(pos.row, 0, rows);
+  this->hooks.edit(pos.row, 0, rows);
   auto lsp = this->lsp.load();
   if (lsp) {
     if (lsp->incremental_sync) {
@@ -234,6 +234,7 @@ void Editor::edit_replace(Coord start, Coord end, const char *text,
       rows++;
   if (this->parser)
     this->parser->edit(start.row, end.row - start.row, rows);
+  this->hooks.edit(start.row, end.row - start.row, rows);
   auto lsp = this->lsp.load();
   if (lsp) {
     if (lsp->incremental_sync) {

src/editor/events.cc

@@ -96,7 +96,7 @@ void Editor::handle_event(KeyEvent event) {
         this->jumper_set = false;
         break;
       case 'N':
-        this->clear_hooks_at_line(this->cursor.row);
+        this->hooks.clear_line(this->cursor.row);
         break;
       case 's':
       case 'v':
@@ -218,18 +218,11 @@ void Editor::handle_event(KeyEvent event) {
     if (event.key_type == KEY_CHAR && event.len == 1 &&
         (event.c[0] >= '!' && event.c[0] <= '~')) {
       if (this->jumper_set) {
-        for (uint8_t i = 0; i < 94; i++)
-          if (this->hooks[i] == this->cursor.row + 1) {
-            this->hooks[i] = 0;
-            break;
-          }
-        this->hooks[event.c[0] - '!'] = this->cursor.row + 1;
+        this->hooks.set(event.c[0], this->cursor.row);
       } else {
-        uint32_t line = this->hooks[event.c[0] - '!'] - 1;
-        if (line > 0) {
+        uint32_t line;
+        if (this->hooks.get(event.c[0], &line))
           this->cursor = {line, 0};
-          this->cursor_preffered = UINT32_MAX;
-        }
       }
     }
     mode = NORMAL;

src/editor/helpers.cc

@@ -312,14 +312,6 @@ void Editor::delete_next_word() {
     this->edit_erase(this->cursor, next_col_cluster);
 }
 
-void Editor::clear_hooks_at_line(uint32_t line) {
-  for (uint8_t i = 0; i < 94; i++)
-    if (this->hooks[i] == line + 1) {
-      this->hooks[i] = 0;
-      break;
-    }
-}
-
 void Editor::cursor_prev_word() {
   uint32_t prev_col;
   word_boundaries(this->cursor, &prev_col, nullptr, nullptr, nullptr);

src/editor/renderer.cc

@@ -11,14 +11,7 @@ void Editor::render(std::vector<ScreenCell> &buffer, Coord size, Coord pos) {
       EXTRA_META + static_cast<int>(std::log10(this->root->line_count + 1));
   uint32_t render_width = size.col - numlen;
   uint32_t render_x = pos.col + numlen + 1;
-  std::vector<std::pair<uint32_t, char>> v;
-  for (size_t i = 0; i < 94; ++i)
-    if (this->hooks[i] != 0)
-      v.push_back({this->hooks[i], '!' + i});
-  std::sort(v.begin(), v.end());
-  auto hook_it = v.begin();
-  while (hook_it != v.end() && hook_it->first <= this->scroll.row)
-    ++hook_it;
+  this->hooks.start_iter(this->scroll.row);
   auto warn_it = this->warnings.begin();
   while (warn_it != this->warnings.end() && warn_it->line < this->scroll.row)
     ++warn_it;
@@ -140,14 +133,9 @@ void Editor::render(std::vector<ScreenCell> &buffer, Coord size, Coord pos) {
     while (current_byte_offset < line_len && rendered_rows < this->size.row) {
       uint32_t color = this->cursor.row == line_index ? 0x222222 : 0;
       if (current_byte_offset == 0 || rendered_rows == 0) {
-        const char *hook = "";
-        char h[2] = {0, 0};
-        if (hook_it != v.end() && hook_it->first == line_index + 1) {
-          h[0] = hook_it->second;
-          hook = h;
-          hook_it++;
-        }
-        update(pos.row + rendered_rows, pos.col, hook, 0xAAAAAA, 0, 0, 0, 1);
+        char hook = this->hooks.next(line_index);
+        update(pos.row + rendered_rows, pos.col, {hook, 0}, 0xAAAAAA, 0, 0, 0,
+               1);
         char buf[16];
         int len = snprintf(buf, sizeof(buf), "%*u ", numlen, line_index + 1);
         uint32_t num_color =
@@ -337,14 +325,8 @@ void Editor::render(std::vector<ScreenCell> &buffer, Coord size, Coord pos) {
     if (line_len == 0 ||
         (current_byte_offset >= line_len && rendered_rows == 0)) {
       uint32_t color = this->cursor.row == line_index ? 0x222222 : 0;
-      const char *hook = "";
-      char h[2] = {0, 0};
-      if (hook_it != v.end() && hook_it->first == line_index + 1) {
-        h[0] = hook_it->second;
-        hook = h;
-        hook_it++;
-      }
-      update(pos.row + rendered_rows, pos.col, hook, 0xAAAAAA, 0, 0, 0, 1);
+      char hook = this->hooks.next(line_index);
+      update(pos.row + rendered_rows, pos.col, {hook, 0}, 0xAAAAAA, 0, 0, 0, 1);
       char buf[16];
       int len = snprintf(buf, sizeof(buf), "%*u ", numlen, line_index + 1);
       uint32_t num_color = this->cursor.row == line_index ? 0xFFFFFF : 0x555555;
@@ -467,14 +449,10 @@ void Editor::render(std::vector<ScreenCell> &buffer, Coord size, Coord pos) {
       break;
     }
     set_cursor(cursor.row, cursor.col, type, true);
-    // if (this->completion.active && !this->completion.box.hidden)
-    //   this->completion.box.render(cursor);
-    // else if (this->hover_active)
-    //   this->hover.render(cursor);
-    // else if (this->diagnostics_active)
-    //   this->diagnostics.render(cursor);
     if (!this->hover_popup->hidden)
       this->hover_popup->pos = cursor;
+    if (!this->diagnostic_popup->hidden)
+      this->diagnostic_popup->pos = cursor;
   }
   free(it->buffer);
   free(it);

src/editor/worker.cc

@@ -1,36 +1,33 @@
 #include "editor/editor.h"
 
-// void hover_diagnostic(Editor *editor) {
-//   static uint32_t last_line = UINT32_MAX;
-//   if (last_line == editor->cursor.row && !editor->warnings_dirty)
-//     return;
-//   VWarn dummy;
-//   dummy.line = editor->cursor.row;
-//   editor->warnings_dirty = false;
-//   last_line = editor->cursor.row;
-//   auto first =
-//       std::lower_bound(editor->warnings.begin(), editor->warnings.end(),
-//       dummy);
-//   auto last =
-//       std::upper_bound(editor->warnings.begin(), editor->warnings.end(),
-//       dummy);
-//   std::vector<VWarn> warnings_at_line(first, last);
-//   if (warnings_at_line.size() == 0) {
-//     editor->diagnostics_active = false;
-//     return;
-//   }
-//   editor->diagnostics.clear();
-//   editor->diagnostics.warnings.swap(warnings_at_line);
-//   editor->diagnostics.render_first();
-//   editor->diagnostics_active = true;
-// }
+void hover_diagnostic(Editor *editor) {
+  static uint32_t last_line = UINT32_MAX;
+  if (last_line == editor->cursor.row && !editor->warnings_dirty)
+    return;
+  VWarn dummy;
+  dummy.line = editor->cursor.row;
+  editor->warnings_dirty = false;
+  last_line = editor->cursor.row;
+  auto first =
+      std::lower_bound(editor->warnings.begin(), editor->warnings.end(), dummy);
+  auto last =
+      std::upper_bound(editor->warnings.begin(), editor->warnings.end(), dummy);
+  std::vector<VWarn> warnings_at_line(first, last);
+  if (warnings_at_line.size() == 0) {
+    editor->diagnostic_popup->hidden = true;
+    return;
+  }
+  editor->diagnostic_popup->clear();
+  editor->diagnostic_popup->warnings.swap(warnings_at_line);
+  editor->diagnostic_popup->hidden = false;
+}
 
 void Editor::work() {
   if (!this->root)
     return;
   if (this->parser)
     this->parser->work();
-  // hover_diagnostic(this);
+  hover_diagnostic(this);
   // if (this->completion.active && this->completion.hover_dirty) {
   //   this->completion.hover.render_first();
   //   this->completion.hover_dirty = false;

src/extentions/diagnostics.cc

@@ -0,0 +1,187 @@
+#include "extentions/diagnostics.h"
+#include <cstdint>
+
+DiagnosticBox *init_diagnostic() {
+  auto diagnostic = std::make_unique<DiagnosticBox>();
+  diagnostic->pos = {0, 0};
+  diagnostic->size = {1, 1};
+  diagnostic->hidden = true;
+  DiagnosticBox *ptr = diagnostic.get();
+  layout::popups.push_back(std::move(diagnostic));
+  return ptr;
+}
+
+void DiagnosticBox::render(std::vector<ScreenCell> &buffer, Coord n_size,
+                           Coord n_pos) {
+  pos = n_pos;
+  size = n_size;
+  int32_t start_row = (int32_t)pos.row - (int32_t)size.row;
+  if (start_row < 0)
+    start_row = pos.row + 1;
+  int32_t start_col = pos.col;
+  Coord screen_size = {io::rows, io::cols};
+  if (start_col + size.col > screen_size.col) {
+    start_col = screen_size.col - size.col;
+    if (start_col < 0)
+      start_col = 0;
+  }
+  pos.col = start_col;
+  pos.row = start_row;
+  if (warnings.empty())
+    return;
+  uint32_t longest_line = 8 + warnings[0].source.length();
+  for (auto &warn : warnings) {
+    uint32_t longest = 0;
+    uint32_t cur = 0;
+    for (char ch : warn.text_full)
+      if (ch == '\n') {
+        longest = MAX(longest, cur);
+        cur = 0;
+      } else {
+        if (ch == '\t')
+          cur += 3;
+        ++cur;
+      }
+    longest = MAX(longest, cur);
+    longest_line = MAX(longest_line, longest + 7);
+    longest_line = MAX(longest_line, (uint32_t)warn.code.length() + 4);
+    for (auto &see_also : warn.see_also)
+      longest_line = MAX(longest_line, (uint32_t)see_also.length() + 4);
+  }
+  uint32_t content_width = MIN(longest_line, 150u);
+  size.col = content_width + 2;
+  auto set = [&](uint32_t r, uint32_t c, std::string text, uint32_t fg,
+                 uint32_t bg, uint8_t flags, uint32_t width) {
+    if (r < 0 || r >= size.row || c < 0 || c >= size.col)
+      return;
+    r += pos.row;
+    c += pos.col;
+    if (r < 0 || r >= screen_size.row || c < 0 || c >= screen_size.col)
+      return;
+    ScreenCell &cell = buffer[r * screen_size.col + c];
+    cell.utf8 = std::move(text);
+    cell.width = width;
+    cell.fg = fg;
+    cell.bg = bg;
+    cell.flags = flags;
+    cell.ul_color = 0;
+  };
+  uint32_t base_bg = 0;
+  uint32_t border_fg = 0x82AAFF;
+  uint32_t r = 0;
+  if (warnings[0].source != "") {
+    std::string src_txt = "Source: ";
+    uint32_t i = 0;
+    for (; i < 8 && i < content_width; i++)
+      set(1, i + 1, (char[2]){src_txt[i], 0}, 0x3EAAFF, base_bg, 0, 1);
+    for (; i < 8 + warnings[0].source.length() && i < content_width; i++)
+      set(1, i + 1, (char[2]){warnings[0].source[i - 8], 0}, 0xffffff, base_bg,
+          0, 1);
+    while (i < content_width)
+      set(1, ++i, " ", 0xffffff, base_bg, 0, 1);
+    r++;
+  }
+  int idx = 1;
+  for (auto &warn : warnings) {
+    char buf[4];
+    std::snprintf(buf, sizeof(buf), "%2d", idx % 100);
+    std::string line_txt = std::string(buf) + ". ";
+    for (uint32_t i = 0; i < line_txt.length(); i++)
+      set(r + 1, i + 1, (char[2]){line_txt[i], 0}, 0xffffff, base_bg, 0, 1);
+    if (r >= 23)
+      break;
+    const char *err_sym = "";
+    uint32_t c_sym = 0xAAAAAA;
+    switch (warn.type) {
+    case 1:
+      err_sym = "";
+      c_sym = 0xFF0000;
+      break;
+    case 2:
+      err_sym = "";
+      c_sym = 0xFFFF00;
+      break;
+    case 3:
+      err_sym = "";
+      c_sym = 0xFF00FF;
+      break;
+    case 4:
+      err_sym = "";
+      c_sym = 0xAAAAAA;
+      break;
+    }
+    std::string text = warn.text_full + " " + err_sym;
+    uint32_t i = 0;
+    while (i < text.length() && r < 23) {
+      uint32_t c = 4;
+      while (c < content_width && i < text.length()) {
+        if (text[i] == '\n') {
+          while (i < text.length() && text[i] == '\n')
+            i++;
+          break;
+        }
+        uint32_t cluster_len = grapheme_next_character_break_utf8(
+            text.c_str() + i, text.length() - i);
+        std::string cluster = text.substr(i, cluster_len);
+        int width = display_width(cluster.c_str(), cluster_len);
+        if (c + width > content_width)
+          break;
+        set(r + 1, c + 1, cluster.c_str(), c_sym, base_bg, 0, width);
+        c += width;
+        i += cluster_len;
+        for (int w = 1; w < width; w++)
+          set(r + 1, c - w + 1, "\x1b", c_sym, base_bg, 0, 0);
+      }
+      while (c < content_width)
+        set(r + 1, ++c, " ", 0xffffff, base_bg, 0, 1);
+      r++;
+    }
+    if (r >= 23)
+      break;
+    if (warn.code != "") {
+      uint32_t i = 0;
+      for (; i < 5 && i < content_width; i++)
+        set(r + 1, i, " ", 0x81cdc6, base_bg, 0, 1);
+      for (; i < warn.code.length() + 5 && i < content_width; i++)
+        set(r + 1, i, (char[2]){warn.code[i - 5], 0}, 0x81cdc6, base_bg, 0, 1);
+      while (i <= content_width)
+        set(r + 1, i++, " ", 0x81cdc6, base_bg, 0, 1);
+      r++;
+    }
+    if (r >= 23)
+      break;
+    for (std::string &see_also : warn.see_also) {
+      uint32_t fg = 0xB55EFF;
+      uint8_t colon_count = 0;
+      uint32_t i = 0;
+      for (; i < 5 && i < content_width; i++)
+        set(r + 1, i, " ", 0x81cdc6, base_bg, 0, 1);
+      for (; i < see_also.length() + 5 && i < content_width; i++) {
+        set(r + 1, i, (char[2]){see_also[i - 5], 0}, fg, base_bg, 0, 1);
+        if (see_also[i] == ':')
+          colon_count++;
+        if (colon_count == 2)
+          fg = 0xFFFFFF;
+      }
+      while (i <= content_width)
+        set(r + 1, i++, " ", fg, base_bg, 0, 1);
+      r++;
+      if (r >= 23)
+        break;
+    };
+    idx++;
+  }
+  size.row = 2 + r;
+  set(0, 0, "┌", border_fg, base_bg, 0, 1);
+  for (uint32_t i = 1; i < size.col - 1; i++)
+    set(0, i, "─", border_fg, base_bg, 0, 1);
+  set(0, size.col - 1, "┐", border_fg, base_bg, 0, 1);
+  for (uint32_t r = 1; r < size.row - 1; r++) {
+    set(r, 0, "│", border_fg, base_bg, 0, 1);
+    set(r, size.col - 1, "│", border_fg, base_bg, 0, 1);
+  }
+  set(size.row - 1, 0, "└", border_fg, base_bg, 0, 1);
+  for (uint32_t i = 1; i < size.col - 1; i++)
+    set(size.row - 1, i, "─", border_fg, base_bg, 0, 1);
+  set(size.row - 1, size.col - 1, "┘", border_fg, base_bg, 0, 1);
+}

src/extentions/hover.cc

@@ -36,7 +36,7 @@ void HoverBox::render(std::vector<ScreenCell> &buffer, Coord n_size,
   int32_t start_row = (int32_t)pos.row - (int32_t)size.row;
   if (start_row < 0)
     start_row = pos.row + 1;
-  int32_t start_col = pos.col; // pos here is the cursor pos
+  int32_t start_col = pos.col;
   Coord screen_size = {io::rows, io::cols};
   if (start_col + size.col > screen_size.col) {
     start_col = screen_size.col - size.col;

src/lsp/worker.cc

@@ -6,6 +6,7 @@ std::unordered_map<std::string, std::unique_ptr<LSPInstance>> active_lsps;
 std::unordered_set<std::string> opened;
 Queue<std::string> need_opening;
 std::vector<Editor *> new_editors;
+Queue<std::unique_ptr<LSPMessage>> response_queue;
 } // namespace lsp
 
 void lsp_worker() {

src/main.cc

@@ -8,9 +8,8 @@
 #include "windows/decl.h"
 
 std::atomic<bool> running{true};
-Queue<KeyEvent> event_queue;
-fs::path pwd;
 std::atomic<uint8_t> mode = NORMAL;
+fs::path pwd;
 
 namespace ui {
 Bar bar;
@@ -18,7 +17,7 @@ Bar bar;
 
 void background_lsp() {
   while (running)
-    throttle(8ms, lsp_worker);
+    throttle(16ms, lsp_worker);
 }
 
 int main(int argc, char *argv[]) {
@@ -44,34 +43,44 @@ int main(int argc, char *argv[]) {
   std::thread lsp_thread(background_lsp);
 
   while (running) {
-    KeyEvent event = throttle(1ms, read_key);
-    if (event.key_type != KEY_NONE) {
-      if (event.key_type == KEY_CHAR && event.len == 1 &&
-          event.c[0] == CTRL('q')) {
-        free(event.c);
-        running = false;
-        break;
-      }
-      if (mode != RUNNER) {
-        if (event.key_type == KEY_MOUSE) {
-          handle_click(event);
-        } else {
-          layout::focused_window->handle_event(event);
-        }
-      } else {
-        ui::bar.handle_event(event);
-      }
-      if ((event.key_type == KEY_CHAR || event.key_type == KEY_PASTE) &&
-          event.c)
-        free(event.c);
-    }
-    for (auto &lsp_inst : lsp::active_lsps)
-      lsp_inst.second->callbacks();
     layout::focused_window->work();
-    throttle(4ms, render);
-    throttle(4ms, io_render);
+    render();
+    throttle(16ms, io_render);
+    KeyEvent event;
+    while (running && lsp::response_queue.empty() &&
+           (event = read_key()).key_type == KEY_NONE)
+      std::this_thread::sleep_for(1ms);
+    uint8_t consumed = 0;
+    do {
+      if (event.key_type != KEY_NONE) {
+        if (event.key_type == KEY_CHAR && event.len == 1 &&
+            event.c[0] == CTRL('q')) {
+          free(event.c);
+          running = false;
+          goto quit;
+        }
+        if (mode != RUNNER) {
+          if (event.key_type == KEY_MOUSE) {
+            handle_click(event);
+          } else {
+            layout::focused_window->handle_event(event);
+          }
+        } else {
+          ui::bar.handle_event(event);
+        }
+        if ((event.key_type == KEY_CHAR || event.key_type == KEY_PASTE) &&
+            event.c)
+          free(event.c);
+      }
+    } while (++consumed < 32 && (event = read_key()).key_type != KEY_NONE);
+    std::unique_ptr<LSPMessage> msg;
+    while (lsp::response_queue.pop(msg)) {
+      msg->callback(*msg);
+    };
   }
 
+quit:
+
   if (lsp_thread.joinable())
     lsp_thread.join();