Andrej Karpathy Learning Series

Project Overview

A comprehensive series of projects following Andrej Karpathy's educational content, exploring deep learning fundamentals through hands-on implementation. This series covers both the makemore language modeling tutorials and GPT implementation from scratch, providing deep insights into neural networks, transformers, and language model architectures.

Key Topics: Neural Networks, Language Modeling, Transformers, Backpropagation, Attention Mechanisms, Character-level Language Models

Project Components

GPT Implementation

Building GPT (Generative Pre-trained Transformer) from scratch, implementing the complete architecture including attention mechanisms, positional encodings, and training loops.

Python Scripts

Core implementation files for the GPT architecture:

Py Scripts

Training Data

Text datasets used for training the language models:

Training Data

Generated Output

Text generated from the trained language models:

Generated Output

Trained Models

Saved models from training sessions:

Saved Models

Makemore Series

Following Karpathy's makemore tutorials to build character-level language models from scratch, progressing from simple models to more sophisticated architectures.

Notebooks

Makemore Part 1 Makemore Part 2 Makemore Part 3

Data

Names Dataset

Learning Outcomes

Technical Skills Developed

  • Neural network implementation from scratch
  • Backpropagation and gradient computation
  • Transformer architecture understanding
  • Attention mechanism implementation
  • Language model training and evaluation
  • PyTorch tensor operations and optimization

Key Concepts Mastered

  • Character-level language modeling
  • Embedding layers and positional encoding
  • Multi-head self-attention
  • Layer normalization and residual connections
  • Autoregressive text generation
  • Model training, validation, and sampling