1. Introduction

Digital images travel as binary. Their container formats (JPEG, PNG, WebP, AVIF) assume a transport that preserves bytes. Many real channels do not. A chat message, a social post, a database text column, a URL, or a printed page all degrade or forbid binary payloads. On those channels an image survives only as text: as a Base64 blob (verbose, opaque, stripped by some sanitizers), an ASCII art rendering (low fidelity), or a link (deferred to a binary fetch).

imgoji takes a different position. Every mainstream operating system ships a font containing hundreds of color pictographs: emoji. The codec references those glyphs by codepoint and composites them to approximate a source image. The encoded artifact is a plain string of emoji. It is short enough to paste into a message, survives any text-safe transport, and renders back to an image in any browser without a bundled asset, because the browser already has the glyphs. We use 'imgoji' to refer to both the codec and the resulting emoji string; the 'representation' is the mathematical model (quadtree + blend); the 'string grammar' is the concrete serialization.

The representation is lossy and low-rate. It does not compete with JPEG at matched byte budgets on fidelity. It competes on a property JPEG cannot offer: the bitstream itself is human-legible text that composes an image directly from a universal preinstalled glyph set. The contribution is the channel, not the compression ratio.

Where imgoji sits among common image representations.

This paper makes the codec's design explicit and reports measured behavior. We contribute: (i) a convergent blend model over a position-implicit quadtree, with a provable prefix property; (ii) cost-complexity (BFOS) pruning for the quadtree, benchmarked against greedy subdivision; (iii) a compact, self-delimiting string grammar whose shorter prefixes remain valid images; (iv) a browser-resident semantic layer that uses CLIP and OWL-ViT to place object-correct base glyphs, gated by a greedy error filter; and (v) an evaluation including negative results: a controlled re-test of the matcher metric under a four-family judge ensemble, and an analysis of an inherent palette gap.

2. Background and related work

Progressive image coding. The prefix property imgoji relies on is the text analogue of embedded wavelet coders. Said and Pearlman's SPIHT^[10] and the embedded zerotree family produce bitstreams whose prefixes are successively finer reconstructions. Shapiro's EZW^[6] established the principle that tree-structured significance can be encoded in transmission order. imgoji adopts breadth-first quadtree traversal so that prefix lengths 1, 5, 21, 85 mark complete grid levels (1×1, 2×2, 4×4, 8×8), giving a level-aligned progression.

Rate-distortion optimization. Selecting where to spend bits is a Lagrangian problem. Sullivan and Wiegand formalized rate-distortion optimization (RDO) for video coding^[12]: minimize D + λR over coding decisions (D is distortion, R is rate). imgoji applies the same principle to quadtree pruning via the BFOS cost-complexity algorithm of Breiman et al.^[2], which produces a sequence of nested subtrees optimal at each complexity. Classical JPEG^[13] and its block-DCT successors spend RDO effort per transform coefficient; imgoji spends it per quadtree node.

Text and ASCII art. Text-based image rendering predates the web, from line-printer art to sixel and ANSI renderers. These map luminance to character density. imgoji differs by using color pictographs that carry their own chrominance, and by treating the string as an encoded stream with a decoder, rather than as a literal glyph-for-pixel map like traditional ASCII art.

Emoji semantics. Eisner et al. learned emoji embeddings (emoji2vec)^[5] from description text, establishing that emoji carry machine-usable semantic content. imgoji uses CLIP^[9] to rank the keyword labels associated with palette glyphs against an image, and OWL-ViT^[7] to localize them. Detections are projected into the codec's coordinate space. The vision-language models run client-side via Transformers.js^[1] (WebGPU or WASM), so no image leaves the device.

Color difference. We report quality in CIEDE2000 (ΔE00)^[11] as an independent judge. It is always computed in CIELAB, regardless of the matcher metric. This follows the methodology of constructing an external evaluator that the encoder cannot optimize directly.

Notation used throughout.

3. Representation

3.1 Model

A scene is an ordered list of placed glyph subjects, rendered back-to-front. A subject is one or more grapheme clusters laid out as a text run. The canvas is the unit square; transforms are frame-relative, so a scene is resolution- and font-independent. Each subject carries a translate (center origin, ±0.5 to the edges), a rotation, a scale (frame coverage), an opacity, and an HSV hue rotation. Order is the only z-index; later subjects paint over earlier ones. A formal grammar appears in the project's format specification; we summarize the codec-specific serialization here.

3.2 Quadtree and blending

The codec is the novel core. It approximates a raster image as a quadtree of blended glyphs. The first glyph approximates the whole image. The next four approximate the quadrants, the following sixteen each quadrant's sub-quadrants, and so on, to a depth capped at eight (one pixel per cell on a 256×256 working canvas). The traversal is breadth-first.

Each cell beyond the root blends its chosen emoji E over the parent's accumulated render A for that region:

A′ = (1−α)A + αE(1)

Intuitively, each cell adds a fraction α of its chosen emoji's color on top of what its ancestors have already painted. The blend uses a fixed ratio (α = 0.7 by default). The root takes α = 1. Each cell picks the emoji minimizing residual against the source over its region.

The decoder is ordinary source-over alpha compositing; the reconstruction therefore renders to SVG with no custom decode code (the browser composites each glyph as a <text> element at opacity α).

The blend operator is fundamental to the codec. Because alpha blending is a convex combination at every level, the color at a pixel is a weighted chain from the root to the finest covering cell. The reconstruction therefore lies inside the convex hull of the glyph palette. A pure replace operator lacks this property: once one glyph covers a pixel the output is confined to that glyph's mean color, and the approximation plateaus. Empirically, blending shrinks the residual at every depth on each image we tested (Table 1).

4. Convergence and the prefix property

Table 1 records reconstruction RMS (0-255 RGB, no seed, α=0.7) on the reference image (NASA Earthrise, 1000×1000) as depth grows (plotted in Figure 2). The approximation improves at every depth from 1 onward. The depth-0 value (145) exceeds the mean-color baseline (142): the single best whole-image emoji is slightly worse than a flat mean fill. This is a real cost of placing a glyph at the root that subsequent refinement must recover.

Table 1: Earthrise reconstruction RMS vs. depth (no seed, α=0.7, RGB; lower is better).

† depth 6 with the 16 solid color-anchor glyphs added to the palette (§5).

Theorem 1 (Prefix validity). Any prefix of an imgoji stream decodes to a complete image: every pixel has a defined color drawn only from glyphs emitted in the prefix.

Proof. The stream is a breadth-first walk of the quadtree, so a node's glyph is emitted before any of its descendants. The decoder paints each emitted glyph, blended over its region's accumulated render, as it reads it; the root is emitted first, so every pixel is initialized. Cut the stream at an arbitrary point. For any cell, either its glyph was emitted — its region then carries the blended contribution — or it was not, in which case the cell lies in an incompletely subdivided subtree whose nearest emitted ancestor was painted, and the region inherits that ancestor's render. No pixel is left undefined, and no emitted glyph references data beyond the cut. The run-length and leaf-run encodings preserve this: an RLE count is read as leading digits, so a run cut mid-way decodes its leading subset, and a leaf-run simply ends early. •

The property is what lets a semantic anchor (a positioned sprite at the root) read clearly before the quadtree refines color over it, and what lets a viewer render a fraction of the stream to show progressive build-up.

5. Glyph palette

The palette is generated from Unicode emoji-data.txt 15.1.0^[3] across the classic pictograph ranges, with skin-tone modifiers and component codepoints excluded. It is pinned to 15.1.0 rather than the latest version so it excludes bleeding-edge emoji that current system fonts may not ship and would render as tofu (missing-glyph boxes). The list contains 1,076 codepoints.

Solid color-anchor glyphs (circles and squares, including black and white: U+1F7E0-1F7EB, U+2B1B/2B1C, U+26AA/26AB) are included deliberately. Because blending is a convex combination, reachable colors are bounded by the convex hull of the palette. The classic pictograph ranges lacked a near-black glyph, so pure black was unreachable. Adding sixteen anchors extends the hull to the extremes. On Earthrise this drops depth-6 RMS from 53 to 51 (Table 1). The gain is modest because the textured earth and moon also bound the residual, and anchors do not help there. For matching the palette is deduplicated by mean color and texture, collapsing 1,076 codepoints to roughly 525 representatives.

6. Quadtree structure: cost-complexity pruning

The quadtree must decide where to subdivide. A fixed depth wastes tokens on smooth regions and under-resolves detail. imgoji grows the full quadtree once and prunes by cost-complexity, the BFOS algorithm^[2]. At each internal node the pruner compares keeping the subtree against collapsing to a leaf under the Lagrangian D + λR:

keep subtree iff D_leaf + λR_leaf > D_sub + λR_sub(2)

where D is area-scaled distortion and R is the token rate of the subtree versus a single leaf. The threshold slider maps to λ = 10^threshold/25, so sweeping it walks the BFOS pruning sequence. Because pruning is cheap relative to growth, the grown quadtree is cached per source image and encode parameters; retuning (a new threshold or byte target) re-prunes the cached quadtree in roughly 250 ms after a first growth of about 1.9 s for the 5,461-node quadtree.

Table 2 compares BFOS against greedy threshold subdivision (subdivide when a cell's residual exceeds a threshold) at matched deflated-byte budgets on a portrait. BFOS wins at every operating point, and the gap widens at low budgets where greedy subdivision mis-allocates scarce tokens.

Table 2: BFOS vs. greedy threshold (ΔE00 at matched byte budgets, portrait; lower is better).

A second structural decision avoids upscaling. Each cell is matched and painted at its native resolution R = min(C, size), where C is the comparison-resolution cap (default 32), rather than a fixed comparison size. Deep cells (87% of cells in a busy photo sit at R ≤ 8) then do 16 to 64 pixels of SSD instead of 1,024, and the encoder's dominant cost, canvas pixel readback, shrinks with them. Across a five-image test this change is 1.4 to 7× faster with equal or lower RMS, a strict Pareto win on three of five images.

7. String grammar

The codec's native serialization is a position-implicit breadth-first string. One invariant carries the grammar: lowercase ASCII letters are operators; uppercase hex digits are data. A cell is either a list of positioned DSL sprites (a glyph followed by one or more transform ops) or a single full-size glyph plus an optional leaf marker. For example, a plain cell is one glyph (🖼️), while a sprite cell is a glyph followed by transform ops (e.g. 🖼️s2x8, scaled and translated). Run-length encoding compresses maximal runs of identical graphemes to the grapheme followed by a hex count; a run of one is just the grapheme. Counts, op values, and the body share the [0-9A-F] alphabet but never collide, because hex following a lowercase op-letter is an op value, hex following a glyph in a run is a count, and the quadtree body itself contains no lowercase letters.

Values are variable-width. More hex digits mean a finer value, not a larger range, and a shorter prefix of the digits is a valid coarser approximation. The prefix property therefore propagates into the transform values themselves. The translate op is the only signed quantity. It is encoded no-zero sign-magnitude, so the range stays ±0.5 at every width and width controls granularity alone.

8. Transmission

The wire format stacks compression stages. RLE collapses uniform runs; deflate-raw^[4] (available in browsers as CompressionStream) stacks on top. For URL transport the fragment carries the payload: #s for raw text, #z for base64url of deflate-raw. The fragment is never sent to the server, so a static host's URL-length ceiling does not apply. Deflate yields roughly a seven-fold reduction over raw percent-encoded emoji, because base64 encodes six bits per ASCII character whereas percent-encoding uses twelve characters per 32-bit emoji codepoint. On a balanced encode (876 glyphs) the raw percent-encoded form is about 9,000 URL characters; the compressed fragment is about 1,300.

9. Semantic base layers

A color-only codec cannot place a face where a face is, or water where water is; it places the best-matching glyph by color, which at coarse resolution is often a generic anchor. imgoji adds an optional semantic layer that runs two foundation models in the browser. CLIP^[9] ranks the palette's keyword labels against the whole image; OWL-ViT^[7] localizes the top labels as bounding boxes. Boxes are computed on the 256×256 source canvas directly, so they land in raster space with no aspect transform.

The pipeline is optional (enabled via a checkbox); when enabled, it downloads hundreds of megabytes of model weights from a model registry). Detections become candidate base-layer sprites: each glyph with a translate (box center), a scale (box coverage), and a rotation. Selection runs in four stages: a confidence floor (thr = 0.05), an area cap (drop boxes covering more than half the image, which are typically whole-image mis-detections), deduplication by emoji, and a cap of sixteen candidates. The candidates render largest-first so the biggest sprite paints at the back.

9.1 Greedy error filtering

Not every detected object helps the reconstruction. A sprite is kept only if it lowers regional error. The filter renders each candidate onto the background-anchor base, measures mean ΔE00 in the sprite's footprint before and after painting, and reverts (snapshot and restore) any sprite that does not reduce the error. On a beach photograph this drops seven candidates to four: the retained glyphs (water, leg, swimwear, person) lower ΔE00 in their regions; the dropped glyphs (feet, beach-with-umbrella, swimmer) did not, because their pictographs do not match the local pixels.

9.2 Rotation and hue search

Each surviving candidate is tested over all sixteen rotations of the DSL rotation op (r0 through rF, 0° to 337.5° in 22.5° steps). The rotation minimizing ΔE00 in the footprint is kept. The snapshot region is padded by about a fifth of the cell side per edge to cover the rotated bounding extent, so a rotated glyph's stray corner pixels do not corrupt the next measurement. A rotation of zero is omitted from the output string, keeping it compact. The same sweep is then applied to the hue op (h0 through hF, the same 22.5° steps), holding the chosen rotation; the hue that further lowers ΔE00 in the footprint is kept, and h0 is omitted. Hue lets a glyph match colors away from its natural palette. On the beach image the retained sprites settled at 157°, 315°, 292°, and 0° respectively; the error filter plus rotation search chose the orientation of each glyph against measured distortion, not against detection confidence alone.

9.3 Parts-based person detection

People are the common case and the failure case of object detection on emoji. A single whole-body box from OWL-ViT would fill a tall region with one face glyph. The layer instead injects a parts query (head, eyewear, torso, legs, feet) and suppresses CLIP demographic face labels, so a person renders as a vertical strip of part glyphs (head, swimwear, leg, feet) over the background, filling squarish regions with squarish glyphs. The parts table self-selects: the detector scores the actual garment or footwear over absent ones.

10. Evaluation

We evaluate on a six-image corpus spanning natural photos, line art, and a depth map. The corpus is small by design: the encoder is deterministic, so the aim is to characterize behavior across content types (smooth, textured, line art) rather than to estimate a population mean, which would call for a statistical sample. Quality is mean ΔE00 (CIEDE2000) over the reconstruction versus the source, always in CIELAB. Table 3 reports an auto-configured run that sets the detail gate, bilateral filter, and threshold from a single one-pass analysis of the source, targeting a 500-byte deflated budget. All images hit the budget.

Table 3: Auto-configured encodes at a 500-byte deflated target.

The line figure reconstructs nearly losslessly (ΔE00 1.5) because its flat regions resolve to exact solid-anchor cells. Natural photos land between 7 and 16 ΔE00, bounded by detail the palette cannot represent. The byte cost is one to two orders of magnitude below a JPEG at its floor: at 234 deflated bytes the parrot encode is already below where JPEG can reach (JPEG hits its floor near 784 bytes at 24×24 and quality zero). Figure 3 traces the full rate-distortion curves across a wider budget sweep. Each curve's knee (open ring) is the point of diminishing returns; across the corpus it lands near 500 deflated bytes, a content-adaptive default for the encoder's byte budget.

11. Negative results and limits

Matcher color space. We tested replacing the matcher's CIE76 (CIELAB Euclidean) distance with OKLab^[8] on the theory that its perceptual uniformity would help. The original A/B judged both matchers with ΔE00 (CIEDE2000), which is itself CIELAB-derived. That judge shares CIELAB's coordinate transform with the CIE76 matcher. A CIELAB matcher is therefore structurally favored, and the test could not distinguish perceptual accuracy from home-field advantage. We re-ran the experiment with a four-family ensemble judge: ΔE00 (CIELAB-derived), OKLab-Euclidean (the OKLab matcher's target), Jzazbz-Euclidean^[14] (PQ/LMS), and CAM16-UCS-Euclidean^[15] (a color appearance model). Each matcher encodes at matched deflated-byte budgets, so the only variable is the matcher color space. The results, over three images at two budgets: CIE76 wins on ΔE00, Jzazbz, and CAM16-UCS in all six runs; OKLab wins on OKLab-Euclidean in five of six. OKLab winning its own metric is trivial (it optimizes it), as is CIE76 winning ΔE00. The decisive evidence comes from the two judges that neither matcher optimizes: Jzazbz and CAM16-UCS both rank the CIE76 matcher ahead. So the original finding was not a CIELAB-judge artifact; independent perceptual spaces also prefer CIE76, and OKLab's uniformity does not yield better reconstructions in the large-difference emoji regime. We retained CIE76. Caveat: all four spaces share the XYZ/LMS linear front end, so a residual family-agreement effect cannot be excluded without a judge outside that family or a human study; the Jzazbz margins are small. The choice of matcher metric is also second-order to the palette gap below.

Perceptual pruning. Growing the quadtree with ΔE00 distortion, so that BFOS allocated tokens to minimize perceptual error, was marginally better only at the extreme low-token end (by 0.1 to 0.4 ΔE00). It was clearly worse everywhere else, with a plateauing curve. The error is palette-limited, not allocation-limited: spending tokens on high-error regions cannot help when no glyph matches them.

The palette gap. To diagnose where quality is lost, we bucketed per-pixel ΔE00 by the original pixel's chroma, hue, and lightness. Error concentrates in mid-chroma cool hues (green, cyan, blue, purple): the palette carries 4 to 19 glyphs in those hues against 159 warm-hue glyphs (red, orange, yellow). Warm mid-chroma (skin, wood, food) is well covered. No rebalancing of a palette under 512 glyphs shifts coverage into the gap, because the glyphs to put there do not exist in the emoji corpus. The gap is inherent, not a tuning failure. Expanding to the full undeduplicated 1,076-codepoint palette yields about 4 to 5% ΔE00 improvement at 1.7× encode time, confirming the ceiling.

System-font dependence. A string encoded on one operating system decodes to different pixels on another (Apple, Google, Noto). This is the format's premise, not a defect: the affordance it exploits is that every device ships the glyph set, so referencing glyphs by codepoint needs zero asset distribution. Encode and decode stay self-consistent within a single browser session.

12. Discussion

imgoji is bounded by its palette, not its algorithms. The structural machinery (quadtree blending, BFOS pruning, native-resolution matching, prefix-valid grammar) is effective and reaches the palette's reachable gamut. Beyond that gamut, no metric or allocation choice recovers quality, because the glyphs are absent. The practical levers are therefore palette extension (more glyphs, or toward a wider hue corpus) and the semantic layer, which trades fidelity for object correctness at coarse resolution where color alone is ambiguous.

The representation is best read as a text format, not a codec competing on rate-distortion against binary image standards. Its uses are the channels binary cannot use: inline images in plain-text fields, progressive thumbnails in a URL fragment, copy-pastable image fragments in chat, and resolution-independent glyph art that renders from the viewer's own font.

13. Conclusion

An image can be a string of emoji. We introduced a convergent blend model that makes the string's reconstruction improve with length, and a breadth-first traversal that makes every prefix a valid image. We applied cost-complexity pruning to pick subdivision points optimally, and added a semantic layer that places object-correct glyphs, validated by measured error. The codec is bounded by its palette, and we reported where and why it cannot be improved by metric choice alone. The artifact is text: short, transmissible, and rendered from a glyph set the viewer already has.