When we talk about measuring similarity in AI, the conversation almost always centers on embeddings—high-dimensional vectors that capture semantic relationships between words, images, or concepts. But what if we step outside the bounds of strict semantic accuracy and instead ask: What does it mean for two things to “harmonize”?
Imagine a system where the goal isn’t to find the “correct” or most statistically similar match, but to surface novel insights and relationships through simulated synesthesia—mapping data to sensory experiences like color, sound, or texture. This shift in framing, from correctness to harmony, opens up a radically different and more creative approach to similarity.
From Semantic Proximity to Sensory Harmony
Let’s reframe the problem: Instead of measuring how close two nodes are in meaning, what if we measure how well their assigned sensory representations—colors, chords, textures—resonate with each other? The aim becomes to reveal non-obvious connections, not just reinforce what we already know.
Embracing Subjectivity & Aesthetics
- Beyond Objective Truth: This system doesn’t try to perfectly map existing semantic relationships. Instead, it creates new relationships based on aesthetic or perceptual frameworks. That’s a feature, not a bug.
- Experiential Output: “Correctness” is less about statistical accuracy and more about whether the resulting sensory combinations feel “right,” “interesting,” or “insightful” to a human observer.
- Human Interpretation: The real value emerges when a human (or AI) observes these harmonious or dissonant relationships and interprets what they might mean about the underlying data.
Redefining “Likeness”
- Aesthetic Qualities Over Meaning: Two nodes might be semantically distant but assigned colors that are complementary, or chords that sound pleasing together. This suggests a new kind of “likeness”—one rooted in harmony rather than proximity.
- Contextual Harmony: In color and music theory, harmony isn’t always about closeness. Complementary colors, for example, are harmonious precisely because they are far apart on the color wheel. This allows for more nuanced definitions of similarity than simple distance metrics.
Synesthesia as a Discovery Tool
- Sensory Bridging: By translating abstract data into sensory modalities (sight, sound, touch), we leverage our brains’ innate pattern-finding abilities in those domains.
- Novel Perspectives: Forcing data through a synesthetic lens can reveal connections invisible in text or numbers. Two unrelated concepts might consistently yield harmonious colors or chords, prompting us to ask why.
- Creative Inspiration: This approach is generative and creative, offering a springboard for new hypotheses, artistic expressions, or even functional design.
Challenges & Design Considerations
- Defining “Harmony” for AI: Humans have color and music theory, but how does an LLM learn or apply these rules to arbitrary data?
- Training Data: The model might need examples of how concepts map to harmonious colors or chords, or how emotional/thematic categories map to sensory qualities.
- Rule-Based vs. Learned: Is the AI generating sensory outputs based on learned associations, or following explicit rules (e.g., “joy” maps to a major chord and a bright color)?
- Interpretability: What does it mean when two nodes generate harmonious outputs? It’s not a direct semantic link, but an invitation to explore.
- Validation: Traditional metrics like accuracy don’t apply. Instead, we might ask:
- Do experts find the connections interesting or inspiring?
- Do these connections suggest new avenues for research or creativity?
- Do they lead to useful discoveries or deeper understanding?
- Designing the Mapping:
- Color Space: Should we use CIELAB for perceptual uniformity, or HSL/HSV for intuitive emotional mapping?
- Musical Scale/Key: Which scales or chord progressions best represent the desired harmony?
- Multi-Modal Integration: If assigning both colors and chords, how do these modalities relate? Do bright colors correspond to major chords, dark to minor?
Analogy: From Science to Art
This approach is less like a scientific measurement and more like an artistic process. A composer doesn’t pick notes at random—they choose combinations that evoke emotion or tell a story. Here, the AI “composes” a sensory representation of the data, and the human interprets the resulting “music” or “painting” to find hidden patterns.
Conclusion
Rethinking similarity through the lens of harmony and synesthesia transforms the concept from a rigid measurement tool into a powerful exploratory and creative instrument. By leaning into subjectivity, aesthetics, and intuition, we can surface relationships that might be invisible to purely logical or quantitative analysis. The success of this approach depends on thoughtful design of the synesthetic mappings—and on our willingness to interpret the resulting sensory “symphony” with curiosity and imagination.
If embeddings are the map, perhaps harmony is the art that helps us see the landscape in a new light.