Zaid Khan

I'm a 1st-year PhD student at Mohit Bansal's group (MURGe Lab) at UNC Chapel Hill.

Most recently, I was a research intern at Ai2 working with Tanmay Gupta, Luca Weihs, and Ranjay Krishna. Before joining UNC, I was a student researcher in the Media Analytics Group at NEC Laboratories America under Manmohan Chandraker. I completed my BS+MS at Northeastern, where I worked with Raymond Fu.

 /   /   /   /   / 

Background

Before graduate school, I spent ~3 years as an early member of the engineering / data science organizations at two high growth startups: Roadie (acquired by UPS for $500m) and Intelligent Flying Machines / OneTrack.AI as software engineer, where I led efforts to scale data infrastructure to match growth, and worked on a range of challenging problems, including embedded deep learning, fault-tolerant distributed systems, realtime adaptive pricing, and data pipelines.

Outside of research, I lift weights, read (here's my goodreads profile), watch mixed martial arts, and sometimes wonder whether randomness is real.

Research

Current Work

• automatic skill-targeted data / environment generation: DataEnvGym frames data generation as an RL-style sequential decision-making problem. The goal is to build agents which can automate the process of identifying the weak skills of a model and generating training data to improve those weak skills. It builds on EnvGen, which generates training environments that help an agent learn skills the agent is weak at.
• LLM-driven exploration and planning: MutaGReP explores large code repositories to find realizable plans for complex, multi-step user requests. We use LLM-guided mutations and informed tree search to explore plan space, and use a symbol retrieval to keep the plans grounded in the codebase.

Prior Work

• Self-training / self-improvement
  • using reinforced self-training to improve program synthesis (Khan et al., CVPR 2024)
  • using unlabeled data to improve vision-language reasoning (Khan et al., CVPR 2023)
• Using uncertainty during reasoning and decision-making
  • using self-consistency to identify unreliable knowledge (Khan et al., CVPR 2024)
  • using uncertainty to decide when to expend more test-time compute (Khan et al., NeurIPS 2023)
• Vision-language representation learning
  • by aligning representations with minimal parameter updates (Khan et al., ICLR 2023)
  • by learning to reconstruct each modality (Khan et al., ECCV 2022)

Publications

Neural tree search for repo-level code-use planning. MutaGReP explores plan space through LLM guided mutations, while grounding the plan to functionality in the codebase using a symbol retriever.

Testing is a critical part of software engineering — what if we could automatically discover inputs which break your code? We show how to train SLMs (Qwen2.5-7B + Llama3.1-8B) to generate unit tests that break code and are useful for debugging.

* Equal contribution

We show how to identify unreliable responses from multimodal LLMs by examining the consistency of their responses over the neighborhood of a visual question, without requiring access to the model's internals.

We show how to selectively decompose complex questions into simpler sub-questions to improve zero-shot performance on challenging multimodal reasoning tasks.

Getting labels for a multimodal dataset can be expensive. We show how you can use unlabeled images to improve performance on data-scarce multimodal tasks.

We explore creating CLIP-like models by minimally updating already-trained vision and language models, finding that updating less than 7% of parameters can match full model training.

We demonstrate a very data-efficient way to align vision and language by learning to reconstruct each modality from the other.

Understanding the emotional content of social media posts is difficult for traditional sentiment analysis models. We show that language models do a good job of this if the post can be translated into a natural input space for them.

Are notions of algorithmic fairness based on racial categories meaningful? We study computer vision datasets that use racial categories, and empirically show that the racial categories encoded in each dataset are often highly inconsistent with each other and with human intuitions.

Adapted from Jon Barron's website..