<?xml version='1.0' encoding='UTF-8'?><?xml-stylesheet href='static/style.xsl' type='text/xsl'?><OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2026-11-22T10:57:57Z</responseDate><request verb="GetRecord" identifier="oai:ecommons.cornell.edu:1813/117106" metadataPrefix="dim">https://ecommons.cornell.edu/server/oai/request</request><GetRecord><record><header><identifier>oai:ecommons.cornell.edu:1813/117106</identifier><datestamp>2026-05-15T19:40:51Z</datestamp><setSpec>com_1813_35</setSpec><setSpec>col_1813_47</setSpec></header><metadata><dim:dim xmlns:dim="http://www.dspace.org/xmlns/dspace/dim" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:doc="http://www.lyncode.com/xoai" xsi:schemaLocation="http://www.dspace.org/xmlns/dspace/dim http://www.dspace.org/schema/dim.xsd">
   <dim:field mdschema="dc" element="contributor" qualifier="author">Gangavarapu, Tushaar</dim:field>
   <dim:field mdschema="dc" element="contributor" qualifier="chair" lang="en_US">Rush, Alexander</dim:field>
   <dim:field mdschema="dc" element="contributor" qualifier="committeeMember" lang="en_US">Bindel, David</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="accessioned">2025-06-30T20:37:05Z</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="available">2025-06-30T20:37:05Z</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="issued">2024-12</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="other">ProQuest Submission ID: 12319</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="other">ProQuest Publication ID: 31641441</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="uri">https://hdl.handle.net/1813/117106</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="doi">http://doi.org/10.7298/3nb8-x132</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="bibid">16921940</dim:field>
   <dim:field mdschema="dc" element="description" lang="en_US">165 pages</dim:field>
   <dim:field mdschema="dc" element="description" qualifier="abstract" lang="en_US">Transformers, powered by self-attention, have become foundational across modalities due to their scalability through parallelism. However, many real-world tasks—from byte-level language modeling to retrieval and reasoning—demand processing extended contexts, often spanning tens of thousands of tokens. The quadratic complexity of Transformers during training and linear KV-cache growth at inference makes them computationally prohibitive for such tasks, necessitating the exploration of more efficient architectures. Linear-time recurrent models and hybrid architectures, which combine linear recurrence with local attention, have emerged as promising alternatives to address these challenges. This thesis investigates two interconnected dimensions of long-context modeling: the development of scalable architectures and the interpretability of their underlying recurrence mechanisms. In the context of scalable architectures, we adapt the Mamba linear recurrent byte-level sequence modeling, tackling the computational challenges posed by extended sequence lengths. By introducing speculative decoding that combines subword drafting with byte-level verification, we demonstrate that token-free models can achieve subword-like inference speedups while maintaining robustness to noise and generalization over long sequences. On the interpretability front, we focus on hybrid models like RecurrentGemma, where recurrent hidden states serve as memory to bridge local attention windows. Using sparse autoencoders, we extract interpretable features from these recurrent states to explore their role in retaining long-range information. Through tasks such as long-context passkey retrieval, we reveal how these mechanisms facilitate extended context dependencies and offer insights into the memory of these fixed-memory models. Through these contributions, this thesis advances the understanding and development of architectures that combine scalability and interpretability for long-context modeling.</dim:field>
   <dim:field mdschema="dc" element="language" qualifier="iso">en</dim:field>
   <dim:field mdschema="dc" element="rights" lang="*">Attribution 4.0 International</dim:field>
   <dim:field mdschema="dc" element="rights" qualifier="uri" lang="*">https://creativecommons.org/licenses/by/4.0/</dim:field>
   <dim:field mdschema="dc" element="subject" lang="en_US">byte-level modeling</dim:field>
   <dim:field mdschema="dc" element="subject" lang="en_US">linear recurrence models</dim:field>
   <dim:field mdschema="dc" element="subject" lang="en_US">mechanistic interpretability</dim:field>
   <dim:field mdschema="dc" element="subject" lang="en_US">natural language processing</dim:field>
   <dim:field mdschema="dc" element="subject" lang="en_US">sparse autoencoders</dim:field>
   <dim:field mdschema="dc" element="title" lang="en_US">Linear recurrent models for robust and interpretable long-context modeling</dim:field>
   <dim:field mdschema="dc" element="type" lang="en_US">dissertation or thesis</dim:field>
   <dim:field mdschema="dc" element="relation" qualifier="localuri">https://newcatalog.library.cornell.edu/catalog/16921940</dim:field>
   <dim:field mdschema="dc" element="format" qualifier="mimetype">application/pdf</dim:field>
   <dim:field mdschema="thesis" element="degree" qualifier="discipline">Computer Science</dim:field>
   <dim:field mdschema="thesis" element="degree" qualifier="grantor">Cornell University</dim:field>
   <dim:field mdschema="thesis" element="degree" qualifier="level">Master of Science</dim:field>
   <dim:field mdschema="thesis" element="degree" qualifier="name">M.S., Computer Science</dim:field>
   <dim:field mdschema="dcterms" element="license">https://hdl.handle.net/1813/59810.2</dim:field>
   <dim:field mdschema="dspace" element="entity" qualifier="type">Publication</dim:field>
   <dim:field mdschema="cris" element="virtual" qualifier="collection" authority="https://cornell-ecommons.eks.prod.4science.cloud/handle/1813/47" confidence="600">Cornell Theses and Dissertations</dim:field>
   <dim:field mdschema="cris" element="virtual" qualifier="author">Gangavarapu, Tushaar</dim:field>
   <dim:field mdschema="cris" element="virtualsource" qualifier="collection">5893a6ea-7af3-41d7-abc6-04bcd26ab5df</dim:field>
   <dim:field mdschema="others" element="access-status">open.access</dim:field>
   <dim:field mdschema="others" element="access-status">open.access</dim:field>
   <dim:field mdschema="cerif" element="openaire" authority="" confidence="-1">&lt;Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="825c0533-dc6d-42bb-847c-5e5bedbedb26">
	&lt;Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843&lt;/Type>
	&lt;Language>en&lt;/Language>
   	&lt;Title>Linear recurrent models for robust and interpretable long-context modeling&lt;/Title>
   	&lt;PublishedIn>
    	&lt;Publication>
      	&lt;/Publication>
   	&lt;/PublishedIn>
   	&lt;PublicationDate>2024-12&lt;/PublicationDate>
   	&lt;DOI>http://doi.org/10.7298/3nb8-x132&lt;/DOI>
   	&lt;Authors>
      	&lt;Author>
        	&lt;DisplayName>Gangavarapu, Tushaar&lt;/DisplayName>
         	&lt;Affiliation>
         		&lt;OrgUnit>
         		&lt;/OrgUnit>
         	&lt;/Affiliation>
      	&lt;/Author>
	&lt;/Authors>
   	&lt;Editors>
	&lt;/Editors>
    &lt;Publishers>
        &lt;Publisher>
            &lt;OrgUnit />
        &lt;/Publisher>
    &lt;/Publishers>
    &lt;License>https://creativecommons.org/licenses/by/4.0/&lt;/License>
    &lt;Keyword>byte-level modeling&lt;/Keyword>
    &lt;Keyword>linear recurrence models&lt;/Keyword>
    &lt;Keyword>mechanistic interpretability&lt;/Keyword>
    &lt;Keyword>natural language processing&lt;/Keyword>
    &lt;Keyword>sparse autoencoders&lt;/Keyword>
   	&lt;Abstract>Transformers, powered by self-attention, have become foundational across modalities due to their scalability through parallelism. However, many real-world tasks—from byte-level language modeling to retrieval and reasoning—demand processing extended contexts, often spanning tens of thousands of tokens. The quadratic complexity of Transformers during training and linear KV-cache growth at inference makes them computationally prohibitive for such tasks, necessitating the exploration of more efficient architectures. Linear-time recurrent models and hybrid architectures, which combine linear recurrence with local attention, have emerged as promising alternatives to address these challenges. This thesis investigates two interconnected dimensions of long-context modeling: the development of scalable architectures and the interpretability of their underlying recurrence mechanisms. In the context of scalable architectures, we adapt the Mamba linear recurrent byte-level sequence modeling, tackling the computational challenges posed by extended sequence lengths. By introducing speculative decoding that combines subword drafting with byte-level verification, we demonstrate that token-free models can achieve subword-like inference speedups while maintaining robustness to noise and generalization over long sequences. On the interpretability front, we focus on hybrid models like RecurrentGemma, where recurrent hidden states serve as memory to bridge local attention windows. Using sparse autoencoders, we extract interpretable features from these recurrent states to explore their role in retaining long-range information. Through tasks such as long-context passkey retrieval, we reveal how these mechanisms facilitate extended context dependencies and offer insights into the memory of these fixed-memory models. Through these contributions, this thesis advances the understanding and development of architectures that combine scalability and interpretability for long-context modeling.&lt;/Abstract>
	&lt;Access xmlns="http://purl.org/coar/access_right" 
    >
    &lt;/Access>
&lt;/Publication>
</dim:field>
</dim:dim>
</metadata></record></GetRecord></OAI-PMH>