School of Science Department of Mathematics 56 Zero-Cost Proxies for Neural Architecture Search Supervisor: YANG Can / MATH Student: LI Aaron Branson Cigres / MATH-STAT Course: UROP 1000, Summer This research begun with an aim to prove prior positive experimental results of our proposed modelindependent and data-only method for selecting embedding dimension of vision transformers. Attempts on this were not successful but instead led to experiments regarding the dynamics of embedding geometry during training. Results revealed formations of compact representation with “string” like geometry followed by a transition into distinct clusters. Grounded on this finding, we proposed and tested two methods for embedding dimension selection. One based on the notion of ambient dimension and another one based on visual characteristics of UMAP projection.
RkJQdWJsaXNoZXIy NDk5Njg=