Building my own pastebin in Docusaurus

February 15, 2025 · 8 min read

Senior Front End Engineer @ Wealthy

Today, I embarked on a journey to solve a common problem: to quickly share small code snippets, interesting online findings, or just a simple personal todo list between me and my friends or between my different devices. Essentially a personal pastebin, but one that doesnt need any server or database, and can be operable through my docusaurus static site so as to not have to rely on other third party apps like whatsapp, slack or discord. Inspired by tools like Topaz paste which uses LZMA-JS to compress and decompress textual data on client side and base64 encoding to convert the binary data into URL safe ASCII characters. One additional feature I wanted on my site was the ability to render the decoded text as markdown to make it rich text - with syntax highlighting for code snippets, and able to render headings, todos, mermaid diagrams, etc like on github.

What I Learned

Compression Algorithms Compression reduces the size of data by removing redundancy. For text-based data, algorithms like LZMA, gzip, Brotli, and Z-standard are highly effective. Here’s a quick comparison:

LZMA: High compression ratio but slower.
gzip: Balanced compression and speed.
Brotli: Modern and optimized for web content.
Z-standard: Modern and optimized for web content.

Why Use Z-Standard and Base64 Encoding?

Why Zstd? Z-Standard (Zstd) is a fast compression algorithm known for its efficiency and speed. It allows us to compress text while maintaining high decompression speeds. This is useful for web applications where users might input large amounts of text, such as a Pastebin clone or an online note-sharing tool.
Why Base64 Encoding? After compression, the output is typically a binary buffer, which isn’t URL-friendly. Base64 encoding ensures that the compressed data can be safely stored in a URL parameter without issues. However, it increases data size by ~33%. Compression helps offset this overhead.
URL Length Limits Browsers impose URL length limits (e.g., ~2MB in Chrome). Compression is essential to fit more data into shorter URLs.

Implementation Steps

Setting Up the UI

I created a new page in my Docusaurus site (/paste) with a textarea for input, and a "Save" button.

src/pages/paste.jsx
import React, { useState } from 'react';

export default function Pastebin() {
  const [inputText, setInputText] = useState('');
  const [generatedUrl, setGeneratedUrl] = useState('');

  return (
    <div>
      <textarea value={inputText} onChange={(e) => setInputText(e.target.value)} />
      <button onClick={handleSave}>Save</button>
      {generatedUrl && <a href={generatedUrl}>Share this URL</a>}
    </div>
  );
}

Install the Zstd Library

To use Zstd in a JavaScript project, we need the @oneidentity/zstd-js package. Install it using npm:

npm install @oneidentity/zstd-js

This package provides JavaScript bindings for Zstd, allowing us to compress and decompress text directly in the browser.

Initializing Zstd in React

Since Zstd uses async webassembly , it requires asynchronous initialization. We initialize it inside a useEffect hook:

import { useState, useEffect } from "react";
import { ZstdInit } from "@oneidentity/zstd-js";

const Paste = () => {
  const [content, setContent] = useState("");
  const [zstdModule, setZstdModule] = useState(null);
  const [isZstdReady, setIsZstdReady] = useState(false);

  useEffect(() => {
    async function initZstd() {
      try {
        const module = await ZstdInit();
        setZstdModule(module);
        setIsZstdReady(true);
        console.log("Zstd Module loaded");
      } catch (error) {
        console.error("Zstd initialization failed", error);
      }
    }
    initZstd();
  }, []);
};

Compressing and Encoding User Input

Now that Zstd is initialized, let's compress user input when they press Enter. We will:

Convert the input text to a binary format using TextEncoder.
Compress it using ZstdSimple.compress.
Convert the compressed data to a Base64 string for safe URL storage.

  const handleSave = () => {
    if (!isZstdReady) return;

    try {
      const inputBuffer = new TextEncoder().encode(content);
      let encodedContent;

      if (inputBuffer.length < MIN_INPUT_LENGTH) {
        encodedContent = "P:" + uint8ArrayToBase64(inputBuffer);
      } else {
        const compressed = zstdModule.ZstdSimple.compress(inputBuffer, 5);
        encodedContent = "C:" + uint8ArrayToBase64(compressed);
      }

      const url = new URL(window.location.href);
      url.searchParams.set("content", encodedContent);
      window.history.replaceState(null, "", url.toString());
      toast.success("Content saved!");
    } catch (error) {
      console.error("Processing failed:", error);
      toast.error("Failed to save content");
    }
  };

Here, ZstdSimple.compress uses compression level 5, which balances speed and compression ratio.

warning

If the text is less than 100 bytes in length, then z-standard compression cannot be applied on it. To differentiate between such data while decoding, we add some arbitrary P: or C: at the start of the uncompressed and compressed data respectively. This way, during decoding and decompressing, we safely handle cases where data wasn't even compressed in the first place.

Decoding and Decompressing on Page Load When users open a URL with compressed data, we need to:

Retrieve the Base64-encoded compressed content from the URL.
Decode it back to a binary format.
Decompress it using ZstdSimple.decompress.
Convert it back to readable text using TextDecoder.

  useEffect(() => {
    async function initZstd() {
      try {
        const module = await ZstdInit();
        setZstdModule(module);
        setIsZstdReady(true);

        const url = new URL(window.location.href);
        const encodedContent = url.searchParams.get("content");
        if (encodedContent) {
          try {
            if (encodedContent.startsWith("C:")) {
              const base64Data = encodedContent.slice(2);
              const compressedBuffer = base64ToUint8Array(base64Data);
              const decompressed =
                module.ZstdSimple.decompress(compressedBuffer);
              setContent(new TextDecoder().decode(decompressed));
            } else if (encodedContent.startsWith("P:")) {
              const base64Data = encodedContent.slice(2);
              const bytes = base64ToUint8Array(base64Data);
              setContent(new TextDecoder().decode(bytes));
            }
          } catch (error) {
            console.error("Decompression failed:", error);
            toast.error("Failed to load content");
          }
        }
      } catch (error) {
        console.error("Zstd initialization failed:", error);
        toast.error("Failed to initialize Zstd");
      }
    }
    initZstd();
  }, []);

info

base64ToUint8Array and uint8ArrayToBase64 are two helper functions we have defined separately. Lets define them below.

Writing the Base64 Operations

/utils/base64-helpers.ts

// Updated to produce URL-safe Base64 without padding
const uint8ArrayToBase64 = (u8Arr) => {
  const CHUNK_SIZE = 0x8000;
  let result = "";
  for (let i = 0, len = u8Arr.length; i < len; i += CHUNK_SIZE) {
    const chunk = u8Arr.subarray(i, i + CHUNK_SIZE);
    result += String.fromCharCode.apply(null, chunk);
  }
  const base64 = btoa(result)
    .replace(/\+/g, "-")
    .replace(/\//g, "_")
    .replace(/=+$/, "");
  return base64;
};

// Updated to handle URL-safe Base64 with padding
const base64ToUint8Array = (base64) => {
  let padded = base64.replace(/-/g, "+").replace(/_/g, "/");
  const padLength = (4 - (padded.length % 4)) % 4;
  padded += "=".repeat(padLength);
  const binaryString = atob(padded);
  const bytes = new Uint8Array(binaryString.length);
  for (let i = 0, len = binaryString.length; i < len; i++) {
    bytes[i] = binaryString.charCodeAt(i);
  }
  return bytes;
};

export { uint8ArrayToBase64, base64ToUint8Array };

Challenges and Solutions

Text less than 100 bytes in length cannot be compressed. So it had to be handled differently.
URL Length Limits: Compression helped reduce data size significantly, making it fit within browser limits. However, for sharing urls over applications like slack, discord, pushbullet, whatsapp, etc, it always exceeded the character limit and had to be shared as a file.
Ensuring generated link is encoded properly and is URL safe.
Decoding while taking care whether the data was originally compressed or not while opening the data containing URL.

Final Thoughts and Improvement Ideas

Building this tool was a great learning experience. I now have a lightweight, self-hosted solution for sharing text and files between devices. Here’s what I learned:

Compression is essential for reducing data size and fitting it into URLs. I need to play around with different compression levels, and modern compression algorithms like Brotli, etc to find a balance between speed and efficiency.
Base64 encoding makes binary data URL-safe but adds overhead. There are a few other encoding options that can be explored to increase the encoding efficiency upwards from the 75% that base64 provides.
I would like to improve on this by adding syntax highlighting to the code snippets I share with others, and render markdown content like todos, codeblocks, tables, etc properly. I'll cover this in another blogpost.
Need to provide Keyboard shortcuts for common actions like copying the generated URL, pasting text into the textarea from user's clipboard, copying the content, Entering vim-like insert mode to show the editor and update the text, and a similar shortcut to resume preview or normal mode.
Use a URL shortening public api to create short links that can be shared over other platforms like normal urls.
Security: Base64-encoded URLs are readable by anyone. For sensitive data, I’ll add encryption in the future.
Sharing Via QR code: This can be a very useful addition for quickly opening on a mobile device.
Maybe add support for versioning of documents? However versioning data will have to be saved in the encoded URL, which user can use to maybe see most recent edit history. Saving too big a version history will not be feasible due to URL size limits. Perhaps using local storage to maintain a longer history on client side could be feasible, albeit difficult to share with others. Needs more exploring.
Consider adding support for file sharing, although images and videos are usually already compressed and don't get too much benefit from compression, and their size is too large for base64 urls.

What I Learned​

Why Use Z-Standard and Base64 Encoding?​

Implementation Steps​

Challenges and Solutions​

Final Thoughts and Improvement Ideas​

What I Learned

Why Use Z-Standard and Base64 Encoding?

Implementation Steps

Challenges and Solutions

Final Thoughts and Improvement Ideas