How Music Lenses works
Dense music embeddings give us a useful “general semantic similarity”—whatever that means for music—but they don’t give us much say in which attributes matter. If it’s the vocalist I’m interested in, I might want similar vocal character even when the mood or instrumentation is quite different. Music Lenses is a way to change that comparison without changing the underlying song embeddings.
Music Lenses is an experiment in doing just that with the existing MuQ-MuLan embeddings for Music4All. It follows Jasper Lu’s prompted-subspace approach, with one extra option: building the lens from concepts recovered through sparse inversion rather than using the text embeddings directly.
What makes a vocal lens?
Start with descriptions that vary along the thing you care about: breathy singing, belting, spoken word, a choir, no vocals, and so on. These are examples of a category, rather than a single query saying “more breathy, please.” The aim is to find directions that distinguish its members.
We normalize and center those concept vectors, then run PCA: the leading eigenvectors of their covariance give us an orthogonal basis for the lens. In the demo we keep enough directions to explain at least 80% of the member variance, with a minimum of two. We can then project every song onto that basis and compare the normalized projections by cosine similarity.
The figure below shows the geometry in three dimensions. Track A differs from the seed along a direction outside the vocal plane, so B starts out closer. Drop that perpendicular component and A becomes the closer match. Try Project, then Normalize, or drag the slider.
This is a small, constructed example to make the projection visible; the actual embeddings have 512 dimensions, and the retained subspace usually has more than two. We center the concept vectors to find the basis, but project audio directly as Bx, without subtracting the prompt mean. Neither the recording nor its original embedding changes.
Where sparse inversion comes in
A shared audio–text space doesn’t mean the two modalities occupy the same distribution. The text directions that look like a good description of a concept needn’t be the ones that best capture its audio-side structure. That’s the problem we explored in Steering dense music retrieval with open-vocabulary concept discovery.
The Text basis uses the original text vectors. With Sparse inversion, Adam instead finds a sparse code through an SAE trained on audio embeddings, balancing reconstruction of the text concept against an empirical audio-distribution prior. We decode and normalize those concept vectors, then use the same PCA procedure. The prior encourages plausible audio-side directions; it isn’t a guarantee of perfect disentanglement. Inversion only changes how we obtain the basis, not the song embeddings.
Trying it on music
The public demo has nine fixed lenses and both basis methods, with all 109,269 tracks available as starting songs. Changing the lens changes the comparison: vocal character, mood, instrumentation, speed, and the others can each bring different neighbours into view. The Prompts panel shows the descriptions used, including controls such as “no vocals.” Controls help define the space; they aren’t hard filters.
The map is a separate, approximate 2D visualization. Its layout moves with the lens, while the larger, linked dots are the ten exact cosine neighbours in the selected search space. Colours and group names come from captions and metadata, so they’re useful navigation hints rather than verified annotations. Zoom in for waveforms of the original 30-second excerpts; click one to listen, and use the star to make it the starting song.
Music similarity is entangled and hard to evaluate. Mood, speed, energy and instrumentation often move together, and a vocal lens won’t remove every other musical difference. Comparing text and inversion is fun, but examples aren’t an evaluation on their own: we’re collecting lens-specific listening judgments to get beyond “this one sounds better to me.”
A few implementation details
The basis uses the original Adam inversion with the L0-20 SAE checkpoint, prior-mean initialization, 500 steps, learning rate 0.001 and prior regularization 0.0001. Eigenvalues select the retained dimensionality; we don’t whiten or weight the projected coordinates by those eigenvalues.
Fixed lenses and their projections are packaged ahead of time. A browser worker searches the full projected corpus by cosine; the full-space top ten is precomputed to avoid downloading the raw corpus. Custom lenses remain available in the local research app.
For the map, t-SNE is fitted to 5,000 fixed tracks and the rest are interpolated from eight sample neighbours. Up to eight caption/metadata profiles are shown per category, including unspecified cases. Instrument profiles can be combinations; supplied language tags can be wrong. These display groups never determine the basis or the nearest-neighbour ranking.
Audio streams from the existing service as original Music4All excerpts. The static page does not run an encoder or inversion model. The waveform responds to the playing audio; the frozen envelopes come from those same excerpts.
For the original image examples and a fuller discussion, see Jasper’s post and Concept Lenses demo.
Back to the music ↗