publications

Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision–language embedding models such as CLIP struggle with binding: they recognize individual concepts but fail to represent which concepts form which objects. Although CLIP behaves like a bag-of-concepts model in cross-modal retrieval, object information is recoverable from its image and text embeddings separately. We study this tension through the binding function, which maps concepts to scene embeddings. We find that scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover object information. However, CLIP’s binding function is high-complexity, which likely prevents the image and text encoders from learning a shared binding mechanism that generalizes to unseen concept combinations. We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer models trained from scratch, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions characterized by multiplicative interactions between concepts, enabling systematic generalization.

Large-scale pre-trained vision-language models like CLIP demonstrate remarkable zero-shot performance across diverse tasks. However, fine-tuning these models to improve downstream performance often degrades robustness against distribution shifts. Recent approaches have attempted to mitigate this trade-off, but often rely on computationally expensive text-guidance. We propose a novel method for robust fine-tuning, SAE-FT, which operates only on the model’s visual representations. SAE-FT regularizes changes to these representations by penalizing the addition and removal of semantically meaningful features identified by a Sparse Autoencoder trained on the pre-trained model. This constraint prevents catastrophic forgetting and makes the fine-tuning process interpretable, enabling direct analysis of semantic changes. SAE-FT is both mechanistically transparent and computationally efficient, matching or exceeding state-of-the-art performance on ImageNet and its associated distribution shift benchmarks.

MEME is a benchmark for evaluating LLM-based agents operating in persistent environments. It defines six tasks across multi-entity and evolving axes, including three previously unscored: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Across six memory systems and three paradigms over 100 controlled episodes, all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% average accuracy) despite strong static retrieval. Prompt optimization and stronger LLMs fail to close the gap; only file-based agents with Claude Opus 4.7 partially succeed, but at  70x baseline cost.

When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding.

We formalize three desiderata for compositional generalization under standard training and show they impose necessary geometric constraints: representations must decompose linearly into per-concept components, and these components must be orthogonal across concepts. We then evaluate these predictions across modern vision models and find that the degree of this structure correlates with compositional generalization on unseen combinations.

We analyze CLIP’s behavior across modalities, revealing bag-of-words behavior in cross-modal tasks but more structured behavior in uni-modal tasks.

Analysis of multi-object learning dynamics in diffusion models and conditions for coherent multi-object generation.

Analysis of ranking properties in visual embedding spaces and their implications for retrieval and compositional reasoning tasks.

We investigate when diffusion-based classifiers can successfully understand compositional concepts, revealing specific conditions required for compositional reasoning.

Large-scale empirical study investigating whether scaling training data improves compositional generalization in vision models. We find that data scaling alone is insufficient without specific architectural inductive biases.

We demonstrate that classifiers built from intermediate layer features can achieve superior out-of-distribution generalization compared to final layer classifiers.

We demonstrate that linear structure in feature spaces is crucial for compositional generalization, going beyond simple decodability requirements.

Novel efficient algorithms for cross-validating Echo State Networks, significantly reducing computational cost while maintaining accuracy.

Novel approach to training recurrent neural networks that synchronizes learning with task structure.

Application of localized CNNs to improve wind forecasting accuracy by incorporating spatial patterns in meteorological data.

Security analysis and cryptographic evaluation of the KAP protocol using enhanced matrix power function.

Efficient algorithms for performing cross-validation on Echo State Networks with reduced computational complexity.