This paper relies on 2 prerequisite mathematical and architectural foundations.
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. We propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.
Introduced the Transformer architecture, replacing RNNs and LSTMs, laying the foundation for modern LLMs, Vision Transformers, and generative AI.
Replaced sequential recurrence with multi-head self-attention and positional encodings, enabling parallel training over massive sequence contexts.
Stacked encoder-decoder blocks featuring multi-head self-attention, layer normalization, residual connections, and point-wise feed-forward networks.
Achieved state-of-the-art BLEU scores on WMT 2014 translation while training in a fraction of the time of recurrent models.
Standard dot-product attention scales quadratically O(N^2) with sequence length N.
We propose Low-Rank Adaptation, or LoRA, which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
Recursive self-improvement (RSI) enables AI systems to turn experience and feedback into persistent changes that improve both their capabilities and the process of future improvement. We first use the Headroom-Closed Index (HCI) to reveal the problems of existing LLMs, then introduce the RSI concept and its development roadmap: from improvement-execution autonomy, improvement-strategy autonomy, experience-acquisition autonomy, and environment-adaptation autonomy, to recursive meta-improvement. Next we examine RSI across scenarios (e.g., scientific discovery, embodied intelligence, software engineering), highlighting their distinct requirements and development speeds. Drawing on diverse industry practices and preliminary empirical evidence, we connect RSI research with practical systems and identify key challenges to achieving genuine RSI.
