I Turned My Browser into a Visual Synthesizer
I had a problem…
I wanted to make a hook for a video and I was looking for a cheap live visual performance tool… something interactive and fun. Everything on the market was either too expensive or just didn’t offer the kind of visual processing I was looking for.
I didn’t want a filter. I wanted an instrument.
I decided to just build the thing myself. I used Google Gemini and Claude Opus 4.6 as my two helpers. They developed the architecture and implementation strategies. I had them look at each other’s responses and pick them apart before a solid plan emerged. This was the biggest change in my vibe-coding workflow from my older projects. It is a strategy I now highly recommend.
This started out as a quick weekend project but eventually snowballed into bad-ASCII, a live visual synthesizer running entirely in the browser.
Treating UI Like a System
When designing the UI, I leaned heavily on my experience with music production software. In DAWs, you have massive, tactile XY pads to control multiple parameters at once (like cutoff and resonance).
I brought that exact concept into bad-ASCII. Instead of burying settings in endless dropdowns, the core interaction relies on large XY performance pads. Moving your cursor across a pad might manipulate the density of the grid on the X-axis while pushing the threshold of a distortion effect on the Y-axis. It feels physical. It encourages you to “play” the visuals.
An instrument is dead if it doesn’t react. To make bad-ASCII feel alive, it takes two primary inputs: your webcam (vision) and your microphone (pulse).
To push the vision aspect further, I integrated MediaPipe, an ML vision framework. Instead of just applying effects to the whole screen, bad-ASCII uses selfie segmentation to map the human silhouette in real-time. The AI severs the person from the background, allowing the engine to treat the human shape as a distinct, malleable material.
Under the Hood: The Mathematics of Making Things Look Cool
The fun part of building a visual synth is figuring out how to completely destroy an image mathematically while keeping the browser running at 60 frames per second.
The app processes the video feed through several distinct “engines.” Here are a few of my favorites and the math that makes them work:
1. Velocity Physics (Making the Grid Bounce)
I wanted the typography to feel like a physical object that reacts to sound. To do this, I gave every cell in the character grid its own inertia using Hooke’s Law (spring physics). The engine reads real-time FFT (Fast Fourier Transform) data from the microphone. When a heavy bass kick hits, the math injects a standing wave impulse into the grid’s X/Y coordinates. The spring force then pulls the characters back to their home positions. The result? The entire screen physically ripples and bounces to the beat.
2. Warp Fields (Melting Reality)
Instead of just applying a blur, I wanted the video to melt. I built an engine that uses 2D Simplex/Value noise algorithms to displace the sampling coordinates before the video is drawn to the canvas. By tying the scale and turbulence of this noise field to the audio energy, the reality of the webcam feed constantly swims, breathes, and warps in time with the music.
3. Reaction-Diffusion (Digital Glitch)
I implemented Gray-Scott reaction-diffusion equations, which mathematically simulate biological growth patterns (like the spots on a cheetah or the branching of coral). But here is the trick: I use the ML silhouette from MediaPipe to seed the simulation. If you stand still, mathematical coral and writhing worm patterns naturally grow out of the edges of your body and consume the background.
Multiplayer Mode
As the app grew, I hit a practical wall: if you are using this to VJ or perform, you want the visuals full-screen on a projector. But if the screen is projected, how do you tweak the controls? Oh, and what if all you want to do is have some fun with friends?
To solve this, I made the instrument collaborative.
I decoupled the heavy WebGL/Canvas rendering pipeline from the React control surface. Using WebSockets and Cloudflare Durable Objects, I built a room-based signaling server.
Now, the “Host” machine can project the video feed and run the heavy ML models. Meanwhile, anyone can enter a 4-letter room code on their phone to join the session as a “Client.” The state of the instrument is synced across all devices using Zustand. You can be standing in the crowd, tweaking the XY pads on your iPhone, and watching the projected visuals warp and glitch on stage in real-time.
Go Break It
Right now, bad-ASCII is still a toy and an experimental playground. It’s a space for me to test out wild visual ideas and push the limits of what the browser can handle.
But the best way to understand an instrument is to play it.
I’d love for you to try it out.
Turn on your camera, put on a track with a heavy bassline, and start messing with the XY pads. Try opening it on your laptop, joining the room on your phone, and controlling it remotely.
If you are a developer, a generative artist, or an AI enthusiast, I’d love to hear your feedback. What features should I implement next? What would make this a better performance tool? Let me know in the comments.
Hey, I’m Aalap. I’m a creative technologist and I love building interactive audio-visual tools for fun and profit. Here are some more articles you might enjoy:
