research
Technical reports and research papers.
Technical Reports
What Does GPT-2 Look At?
Understanding the attention patterns of pretrained language models can help characterize their internal behavior. Inspired by prior analyses of BERT, we examine relative-position preferences, attention to token categories, attention entropy, and head similarity in GPT-2 Small. Across 1,000 Wikipedia segments, we identify five heads that assign more than half their attention to the current token and two that strongly prefer the preceding token. Many heads in intermediate and deeper layers attend heavily to an initial end-of-text (EOT) token, whereas non-boundary tokens assign almost no attention to natural paragraph boundaries. Entropy is relatively high in early layers, and trailing-EOT queries exhibit a distinct layer-dependent trend. Finally, a two-dimensional embedding reveals clusters associated with different attention patterns, with earlier-layer heads generally more dispersed than later-layer heads.
Read report (PDF)
Revisiting TitanFuzz: A Reproduction and Comparative Evaluation Against DeepREL
Testing deep learning frameworks has long been a challenging task. TitanFuzz introduces an LLM-based fuzzing approach that combines autoregressive code generation and code infilling to generate diverse test programs. In this study, we reproduce TitanFuzz and evaluate its effectiveness against DeepREL, an API-level fuzzer, on PyTorch. We select 30 APIs across six categories and conduct ten independent runs per API for each fuzzer, with a fixed time budget of 60 seconds per run. By independently replaying up to 50 generated candidates per run, we measure additional PyTorch Python source lines, previously unseen ATen operator names, valid test yield, and potential bugs. On average, TitanFuzz covers 22.75 additional Python lines and 17.54 new ATen operator names per API–run, compared with 3.44 and 13.21 for DeepREL. However, TitanFuzz produces a lower valid yield (50.69% versus 68.42%). These results highlight a trade-off between broader program exploration and test validity.
Read report (PDF)